Audit of Data Management

[ PDF version ]

Final Report
July 2020

Background

Context

The volume of data that the Canadian government produces, collects and uses has grown substantially. This growth has been largely attributed to recent advancements in digital technology. Organizations are, therefore, changing their business models, building new expertise and devising new ways of managing and unlocking their data to make better decisions, design better programs and deliver more effective services.

Considering the recent efforts to modernize the public service by making it more agile, equipped and inclusive, a forward-looking, open approach to data is believed to be an essential piece of public service renewal. To this end, the Clerk tasked a group of senior civil servants in January 2018 to come up with a data strategy for the Federal Public Service. This group produced a document known as the Report to the Clerk of the Privy Council: A data strategy roadmap for the Federal Public Service.

Subsequent to the publication of the above-mentioned report, all federal departments were tasked by the Clerk of the Privy Council to develop a data strategy customized to their needs by September 2019.

One of the Privy Council Office's (PCO) key functions is to serve Canada and Canadians by providing professional non-partisan advice and support to the Prime Minister, portfolio ministers, and Cabinet on matters of national and international importance. Credible data is needed to support evidence-based decision-making in a timely manner.

This audit was proposed as a result of consultations with senior management and in light of recommendations made in the above-mentioned report. As a project related to information management, this audit complements the 2015-2016 Audit of Recordkeeping Transformation.

This was a forward-looking audit that sought to assess PCO’s plans to develop and implement a data strategy while the work to develop such strategy was ongoing. The audit was intended to facilitate the strategy's development and implementation by identifying any potential risks that may impede the effective implementation of the strategy. It is anticipated that, after the data strategy is fully developed and implemented, there will be subsequent audits.

Whereas data may mean different things to different people, in this audit, based on aggregate project results and consistent with TB definition, data is defined as unorganized information that is commonly found in a structured format (file, database, statistics, etc.). But it can also be found in an unstructured format (Word documents, reports, etc.) that can be transformed into a structured format with technology (e.g., a word map or natural language processing).

Objective, scope & criteria

The audit examined data management-related and planning activities at PCO over the course of July-September 2019. The main objectives of the audit were to:

1. assess if PCO was on track to develop a data strategy; and,
2. identify any potential risks that might impede the implementation of the strategy when it is developed.

Using the Clerk’s report as a guideline, the audit was based on PCO meeting the following criteria (i.e. points of reference for assessing performance).

1. 1.0 Governance - PCO is on track to have data management governance mechanisms in place with defined accountabilities to effectively support data infrastructure planning, data collection, storage, and usage to meet the Department’s current and future needs.
2. 2.0 Policy, Procedure and Tools - PCO is on track to plan for, develop and implement a data strategy; and has in place or is in the process of putting in place policies and procedures related to the ethical and secure use of data to support evidence based decision-making.
3. 3.0 People and Capacity - PCO’s workforce has the knowledge and skills necessary to achieve data management outcomes in various modes including digital technologies and that PCO has processes and procedures in place to recruit, develop, train and retain skilled people that it needs to do data work in a digital environment.
4. 4.0 Environment and Digital Infrastructure - PCO has the right information technology environment and infrastructure to allow its skilled professionals to use technologies to support evidence-based decision-making.

Methodology

The approach and methodology of this audit engagement were risk-based and consistent with the International Standards for the Professional Practice of Internal Auditing (IIA) and with the Treasury Board Policy and Directive on Internal Audit. The audit conforms with the IIA’s Quality Assurance and Improvement Program.

Sufficient and appropriate audit procedures were conducted and evidence gathered to support the accuracy of the findings and conclusion in this report, and to provide an audit level of assurance. The findings and conclusion are based on a comparison of the conditions, as they existed at the time of the audit, against pre-established audit criteria that were agreed upon with management. These findings and conclusions are only applicable to the entity examined and for the scope and time period covered by the audit. The assessment was conducted in the following three phases:

• Planning Phase
  • Developed audit planning documents and assessment tools.
  • Conducted interviews and reviewed relevant documentation.
  • Performed walkthroughs and observations of some key critical processes or activities.
• Examination Phase
  • Identified, analyzed, evaluated, and recorded relevant information.
  • Provided briefings to management on the engagement progress.
• Reporting Phase
  • Identified, analyzed, evaluated, and recorded relevant information.
  • Provided briefings to management on the engagement progress.

Key findings

Governance

Criteria 1.0

PCO is on track to have data management governance mechanisms in place with defined accountabilities to effectively support data infrastructure planning, data collection, storage, and usage to meet the Department’s current and future needs.

Why is it important?

Establishing a governance structure at the right level will facilitate the provision of effective oversight for the development and implementation of the data strategy in the Department.

Key findings

• At the time of conducting this audit (June – September 2019), there was no clear direction on the composition and roles and responsibilities of the eventual governance structure for data management at PCO. However, those interviewed as part of this audit underlined the importance of establishing a governance structure at the right level as the first step to facilitate the development and implementation of a data strategy at PCO.
• A department-wide consultation carried out by the Working Group on Data Strategy has also reiterated the importance of this issue and has recommended that PCO establish by December 2019 appropriate governance instruments, including a task force to oversee and prioritize the implementation of the data strategy.
• While most interviewees suggested using the existing PCO governance committees instead of creating a new one, document review indicated that some other departments adopted a hybrid approach (i.e., amalgamated and repurposed two existing committees).
• The audit concluded that regardless of what governance model is adopted, it will be imperative for that body to have a data focused mandate.

Recommendation

The ADM, CSB should consider:

• Establishing/identifying a governance committee (e.g., at Assistant Deputy Minister level) to oversee the development and implementation of the data strategy.

Policy, procedures and tools

Criteria 2.0

PCO is on track to put in place policies and procedures related to the ethical and secure use of data.

Why is it important?

Having in place policies, procedures and tools contributes to ensuring that the fundamental processes of the data strategy are performed in a way consistent with the departmental and governmental requirements.

Key findings

• The audit conducted a comparative review of the departmental policies and directives related to Information Management and Information Technology. The results indicated that PCO policies, procedures and authorities in these domains are consistent with applicable TB policies and directives.
• The majority of the interviewees indicated that government policies and directives on Information Management, Information Technology, Access to Information and Privacy, Value and Ethics, Government Security, as well as the newly established Policy on Service and Digital (took effect on April 1, 2020) are adequate to ensure the ethical and secure collection, storage, sharing and use of data.
• Although the current Information Management policies and authorities cover the ethical and secure collection, storage and use of information, the rapid advent of new technologies might change the future landscape and needs.
• Therefore, as the Department embarks on implementing its data strategy, and as the use of innovative technologies, such as automation and artificial intelligence become more common in the collection, storage, sharing and analyses of data, it would be prudent to take a closer look at these policies and procedures to ensure that there are enough safeguards for the ethical and secure use of data.

Recommendation

The ADM, CSB, in collaboration with relevant ADM/Assistant Secretary colleagues, should consider:

• Reviewing existing policies and procedures to ensure that PCO’s data repositories have protective controls (e.g., access is granted on an as needed basis) and to iteratively apply detective controls (e.g., irregularities are identified promptly) where feasible to mitigate against risk of unauthorized access or use of data; and,
• Developing training and awareness campaigns to enhance PCO employees’ understanding and adherence to the norms of the ethical and secure collection, sharing and use of data.

People and capacity

Criteria 3.0

PCO’s workforce has the knowledge and skills necessary to achieve data management outcomes in various modes including digital technologies and that PCO has processes and procedures in place to recruit, develop, train and retain skilled people that it needs to do data work in a digital environment.

Why is it important?

A successful data strategy requires that the organization has the talent and capacity to manage, interpret, use and understand data.

Key findings

• The audit assessed to what extent PCO’s workforce has the knowledge and skills necessary to achieve data management outcomes and that PCO has processes and procedures in place to recruit, develop, train and retain skilled people that it needs to do data work in a digital environment.
• The common strand among interviewees was that the level of data literacy varies within PCO depending on particular job descriptions and specific functions.
• According to the interviewees, there are several groups and functions that have developed significant capacity in data literacy and data management. There are also a number of groups that have made certain attempts to improve data literacy and data skills within their functions. However, almost all these efforts have been described to be fragmented and lacked consistency and coordination.

Recommendation

The ADM, CSB, in collaboration with relevant ADM/Assistant Secretary colleagues, should consider:

• Conducting a training needs assessment and developing the appropriate training plans and programs for staff and management to upgrade their data literacy and data management skills; and,
• Initiatives to recruit and retain skilled personnel who are data literate to ensure that the Department has the talent and capacity needed to manage, understand, use and share data.

Environment and digital infrastructure

Criteria 4.0

PCO has the right information technology environment and infrastructure to allow its skilled professionals to use technologies to support evidence-based decision-making.

Why is it important?

The successful implementation of the data strategy requires the establishment of the appropriate information technology environment and infrastructure.

Key findings

• The audit assessed the extent to which PCO’s existing Information Technology infrastructure is conducive to facilitate data management and analytic practices, including horizontality and collaboration both within and outside the Department.
• Document reviews, including a review of a current draft of PCO’s Information Management/Information Technology Strategic Plan indicates that PCO has a comprehensive plan to upgrade and modernize its infrastructure in coming years.
• Despite all the improvements, including the deployment of applications such as GCdocs, the majority of interviewees indicated that PCO’s existing information technology architecture is designed primarily for working in silos, and as such, it does not effectively facilitate data sharing and collaboration.
  • For example, the absence of a government-wide interoperable secure network has been identified as one of the main barriers towards advancing data management objectives, data sharing and collaboration for those who primarily deal with confidential information.
• PCO has made significant investments in upgrading Information Technology infrastructure over the past years. However, most interviewees were of the view that continuous investments need to be made to modify, upgrade and modernize the current Information Technology infrastructure and to ensure that appropriate tools, applications and infrastructure are available to facilitate data management, data analysis, data sharing and collaboration.

Recommendation

The ADM, CSB should consider:

• Working with stakeholders such as Shared Services Canada and Treasury Board Secretariat, as well as PCO program managers to ensure that future investments in Information Technology infrastructure support the implementation of PCO’s data strategy.

Conclusion

Overall, our audit results indicate that there is enough evidence to provide a reasonable assurance that PCO is on track to create the necessary foundation for the development of a data strategy as outlined in the Clerk’s Report and aligned with the department’s lines of business.

The Department circulated its draft data strategy to key stakeholders in September 2019. The draft data strategy outlines short term (by March 2020), medium term (by March 2021) and long term (past March 2021) goals. The draft data strategy was described as an evergreen document, which will be modified periodically to reflect new opportunities and realities.

The audit has made a number of recommendations with the goal of supporting the successful implementation of the data strategy in the short term (See Annex A).

Future audits will assess the implementation of the data strategy and how it is helping the Department unlock the power of data to support its business processes and decision-making.

Annexes

Annex A – Recommendations

Annex B – Management response & action plan