Audit of Surveillance Activities

Executive Summary

Context

The Public Health Agency of Canada conducts public health surveillance in collaboration with federal, provincial, territorial, and international partners, as authorized by the Department of Health Act and the Public Health Agency of Canada Act^{Footnote 1}. Surveillance is the collection, analysis, interpretation, and dissemination of information to support decision-making to improve the health of Canadians^{Footnote 2}.

The Agency conducts surveillance through approximately 58 surveillance systems located with the Infectious Disease Prevention and Control Branch (IDPC), the Health Promotion and Chronic Disease Prevention Branch (HPCDP), and the Health Security Infrastructure Branch (HSIB). The Agency spends an estimated $63 million on these systems annually.

The Agency's surveillance activities depend on data provided by its partners; on the IM/IT capacity to receive, store, analyze, and publish data; and on epidemiological expertise.

Since 2007, the Agency has drafted three strategic plans to coordinate its surveillance activities. These plans launched initiatives to implement corporate models for data management and technology, for surveillance decision making, and for collaboration with its partners.

Previous audits of surveillance acknowledged this work and noted opportunities for improving coordination. The Agency must continue to improve its surveillance function in order to maintain its relevance within a dynamic public health environment^{Footnote 3}.

Audit Objective

To determine whether surveillance resources and activities are effectively managed to support the collection and analysis of quality data, the translation of evidence into surveillance products, and the timely dissemination of information that addresses users' needs.

Findings

Effective Processes

• At the branch and centre levels, formal and informal processes existed for the review of surveillance activities, such as the HPCDP Evidence Strategy consultation process;
• The Agency formally and informally collaborated with its partners, including provinces and territories, to mitigate risks related to the acquisition of data;
• Effective processes were in place for quality assurance of surveillance data;
• Branches and centres used a variety of mechanisms to assess the timeliness and relevance of their surveillance products;
• Branches identified high-level HR-related risks; and
• The Agency had a focal point for Sex and Gender-Based Analysis Plus (SGBA+) implementation and had reviewed SGBA+ application in peer-reviewed publications and in surveillance products.

Areas for Improvement

• Addressing gaps in the surveillance governance structure and corporate leadership, including the reinstatement of a corporate surveillance review process, to better support strategic planning and horizontal coordination of associated initiatives;
• Making necessary adjustments to complete the development of formal information-sharing agreements with provinces and territories, through further incorporation of project management best practices, adequate resourcing, and continuous senior management support;
• Adopting systematic and documented processes for assessing users' needs and the extent to which surveillance outputs meet those needs;
• Improving processes for prioritization and communication of information technology (IT) challenges;
• Developing strategies to address potential workforce competency challenges; and
• Further incorporating SGBA+ into surveillance activities to support the Agency's commitment to health equity.

Note: The scope of this audit predates PHAC's COVID-19 Response of 2020. Therefore, the audit findings and recommendations in this report do not specifically relate to PHAC's response to the pandemic. The Agency's response to COVID-19 will be reviewed in other internal and external audits.

Conclusion

We conclude that surveillance resources and activities were effectively managed to support the collection and analysis of quality data, the translation of evidence into surveillance products, and the timely dissemination of information that addresses users' needs. However, strengthened control processes are required to ensure that the Agency promotes a coherent and coordinated approach to surveillance, and effectively implements the horizontal initiatives described in its strategic surveillance plans

Strategic Direction

Context

In 2007, the Agency drafted a strategic plan to "guide the organization from a wide variety of existing surveillance systems (processes, people, technologies) into a coherent approach to surveillance"^{Footnote 4}. Subsequent plans reiterated the need for a shift "from a siloed approach to one that is more coherent and coordinated"^{Footnote 5}.

Prior to 2017, senior management committees and the Surveillance Integration Team (SIT) provided corporate oversight for surveillance. This model included a Chief Health Surveillance Officer (CHSO), supported by a dedicated team.

The CHSO position became vacant in 2017.

What did we expect to find?

We expected to find that the Agency had a governance structure in place to support the implementation of strategic direction and horizontal initiatives for surveillance. We also expected to find mechanisms in place to ensure corporate oversight of surveillance activities.

Findings

Key responsibilities formerly assigned to the CHSO were not redistributed, including for oversight of strategic surveillance plans, chairing the SIT, reporting on surveillance activities to the Executive Committee, providing technical expertise on surveillance priorities, and for championing a regular review of surveillance systems and activities for corporate decision making.

The Data, Partnerships, and Innovation Hub had assumed the CHSO's data management responsibilities.

The SIT had become less effective at making decisions due to unstable membership, a shift toward information sharing, and the absence of a CHSO embedded in a senior governance committee.

The Agency drafted its most recent surveillance strategic plan in 2016. The Executive Committee approved the plan in principle in 2016, but the plan never received formal approval, and no subsequent plan had been developed.

Significant initiatives identified in strategic surveillance plans were not developed, not implemented, or were discontinued. Examples include the Corporate Process for Public Health Surveillance Decision Making (CPSDM), and a Data Management Initiative.

Conclusion

A governance structure was in place for oversight of surveillance activities from April 2017 to March 2019. However, key leadership responsibilities were not redistributed following the elimination of the CHSO position. The lack of redistributed leadership responsibilities and the ineffectiveness of the SIT as a decision-making body reporting to senior management committees have impeded the implementation of measures articulated by the 2016 and 2013 strategic surveillance plans.

Corporate Decision Making

Context

A critical component of corporate oversight and decision making is the flow of adequate, timely, and relevant information to senior management. The World Health Organization has noted that, because new surveillance systems are often developed over time in response to shifting trends, existing systems may require periodic review to ensure the continued relevance and overall efficiency of surveillance efforts^{Footnote 6}.

What did we expect to find?

We expected to find standardized, corporate-level processes in place for the assessment of surveillance activities to inform strategic decision making, and for the coordination of surveillance management.

Findings

Branches and centres used internal processes that, to varying degrees, functioned as forums for the assessment of surveillance activities. These included project planning exercises, committee meetings, and periodic user satisfaction surveys. These processes primarily addressed operational or technical issues related to specific surveillance activities, and lacked the scope and standardization of a corporate assessment mechanism.

One branch recently undertook a comprehensive branch-wide consultative process, resulting in a forward-looking Evidence Strategy. The strategy focused on identifying future needs and on augmenting existing surveillance efforts to meet those needs, and was used as a decision-making tool for the reallocation of funds within one centre to address evidence gaps in priority areas. The Evidence Strategy did not identify or address areas where surveillance activities may no longer be a priority, or may be a low priority. As such, the process may not function as a corporate review and prioritization exercise designed to inform the creation or sunsetting of surveillance activities.

With the exception of these branch - and centre-level processes, branch management confirmed that there had not been a corporate-level assessment of surveillance activities since 2016.

Pursuant to the 2013-2016 strategic plan for surveillance, the Agency developed the CPSDM. The CPSDM facilitated periodic reviews of clusters of surveillance systems using clearly defined criteria and methodologies to assess their relevance and resource use. Through this process, the CHSO provided oversight for reviews presented to SIT and senior governance committees. However, the process was discontinued after 2015-16.

In 2016, at the request of the Agency's President, the Office of the Chief Science Officer conducted a review of surveillance systems and presented the results to senior management. No identifiable actions resulted from this assessment, and program managers stated that the results of the review were not communicated to program areas.

Conclusion

Although processes exist within branches and centres that may serve as assessment mechanisms for surveillance activities, they are often ad hoc, vary in scope and implementation, and are limited to the surveillance perspectives and priorities of individual branches or centres. The absence of a sustained and systematic corporate assessment process, such as the CPSDM, may be attributed to the gap in corporate surveillance leadership previously filled by a CHSO.

Data Acquisition

Context

The Agency has experienced ongoing data acquisition challenges. A 2016 review by the Office of the Chief Science Officer identified limitations to the representativeness of surveillance data, including gaps in coverage related to the consistency and completeness of data provided by the Agency's provincial and territorial partners^{Footnote 7}.

In 1999 and 2002, the Auditor General recommended that Health Canada address a relative lack of information sharing agreements with provinces and territories. The Agency made efforts to establish such agreements, but in 2008, the Auditor General expressed the view that progress in this area remained unsatisfactory. In 2009, work began on a Multilateral Information Sharing Agreement (MLISA), designed to establish standards for sharing public health information relating to infectious diseases and public health emergencies of international concern.

Technical annexes currently under development will provide MLISA with specificity regarding the content and format of information partners may choose to share. The Agency's 2016-2019 strategic surveillance plan presents both the Agency and a Public Health Network (PHN) steering committee as leads for annex development.

The Agency's methods of acquiring data include reliance on data sharing by provinces and territories, reliance on funding by external agencies, reporting by health practitioners, use of surveys, and laboratory testing.

What did we expect to find?

We expected to find that the Agency had effective processes for acquiring data to meet its surveillance needs, and that the Agency was managing risks associated with acquiring and assessing the quality of its surveillance data. Our assessment focused on areas of risk, as indicated by stakeholders and in prior audits and evaluations.

Findings

The Agency's contribution to the development of the technical annexes has lacked an adequate project management approach, including clearly defined accountabilities, responsibilities, deliverables, and timelines. We were unable to determine the Agency's process for estimating the resources required for development of the technical annexes. For several years prior to 2018, a half full-time employee (FTE) supported the Agency's MLISA lead. In late 2018, the Agency introduced more project management elements, including the formation of a team with a dedicated project coordinator.

The Agency's Corporate Risk Profile identifies MLISA as a current control for the risk that the Agency may lack access to timely and accurate data. We found that MLISA consists of statements of principles and administrative expectations. While MLISA may structure how information is shared, informal relationships of trust remain the de facto control to mitigate risks related to acquiring infectious disease surveillance data from other jurisdictions.

The Agency has embedded surveillance, laboratory, and public health officers within some provincial and territorial jurisdictions to facilitate data collection, analysis, and sharing, and to enhance provincial and territorial epidemiological capacity. Public health officers were recently instrumental in the collection of surveillance data related to the opioid crisis.

Contracts over $100 000 require cooperation with Public Services and Procurement Canada (PSPC). The Agency is addressing concerns about the timeliness of these processes, in part through the development of critical path documentation and ongoing collaboration with PSPC.

The Agency develops priorities for survey content negotiations with Statistics Canada through informal processes coordinated by the Data Coordination and Access Program within the Data, Partnerships, and Innovation Hub.

Surveillance systems had effective and well-documented processes in place to conduct data quality assessments and take appropriate actions, including Data Quality Assessment Reports and alternative data quality controls.

Conclusion

Risks posed by a lack of formal data sharing agreements persist. The pace of MLISA annex development presents a risk that this high-profile information sharing agreement will not be realized as anticipated. Additionally, we note that, as a control, MLISA may improve the consistency rather than the availability of information.

User Needs Assessments

Context

The Agency's credibility depends on its capacity to provide its stakeholders with timely surveillance information to support decision making that improves the health of Canadians^{Footnote 8}. In 2008, the Auditor General emphasized the importance of assessing users' needs and the extent to which they are met by surveillance products, noting that the Agency's users' needs "have not been systematically assessed and documented, nor has the extent to which the Agency's analyses and reports have met those needs" ^{Footnote 9}.

What did we expect to find?

We expected to find that management monitors the extent to which the characteristics and timeliness of surveillance outputs continue to meet user needs.

Findings

Sixty-nine percent of sampled surveillance systems had established timelines for the dissemination of surveillance products, with the majority of sampled systems meeting those timelines. Respondents cited lengthy internal and external approval processes, publishing delays, resource challenges, and operational priorities as reasons for not meeting timelines. External impediments included expert group and partner approval processes, data availability issues with providers, and privacy constraints.

Sixty-six percent of sampled surveillance systems had conducted at least one assessment of user satisfaction. For some systems, the most recently conducted assessment was in 2015. Processes for monitoring user satisfaction included feedback mechanisms through national advisory committees, the Public Health Network, and other networks, consultations with experts and communities of practice, periodic user satisfaction surveys, and web analytics.

Forty-four percent of sampled surveillance systems provided evidence that management actions were taken as a result of user satisfaction assessments. Mechanisms were in place for surveillance systems to report the extent to which key products were meeting timelines and users' needs.

In 2014, the Agency developed a Surveillance Knowledge Translation Standard (SKTS) to support the knowledge translation activities of PHAC's surveillance community. Agency-wide adoption of SKTS was intended to ensure that the Agency's public health surveillance information was provided to those who need it, in the formats and timeframes they needed, to support public health action to improve the health of Canadians. We found that there was no continued Agency-wide adoption of this standard of practice.

Conclusion

In general, management has been monitoring the extent to which surveillance products are meeting users' needs. However, adopting a more systematic and documented approach to monitoring could help to ensure that the Agency provides surveillance products to those who need them, with the characteristics and timeliness that meet users' needs.

IT Resources

Context

In 2013, the Agency's strategic surveillance plan called for the establishment of a corporate model for data management and technology. The Agency began development of a Data Management Initiative (DMI), which included components addressing data management, data acquisition, data security, and investment in the Agency's technical infrastructure. Not all components were completed, and the DMI was put on hold in 2018.

As part of a 2018 Government of Canada initiative, the Data, Partnerships, and Innovation Hub has begun consultations to develop a Data Strategy, which reflects a renewed commitment to a corporate approach to data management and its associated technical infrastructure.

What did we expect to find?

We expected to find effective processes for prioritizing IT challenges affecting the achievement of surveillance objectives and for communicating issues to those responsible for their resolution, including to senior management as appropriate.

Findings

Sixty-six percent of sampled surveillance systems experienced IT issues affecting achievement of their objectives. Surveillance staff attempted to communicate these issues through multiple channels.

Communication processes differed for the National Microbiology Laboratory (NML), which maintains an internal scientific informatics services division. Respondents for sampled NML systems reported not experiencing internal IT issues affecting objectives. NML management routinely discussed IT-related items.

In July 2018, the SIT began a report on surveillance IT issues, but subsequently suspended its meetings and did not escalate its findings to a Tier 2 committee.

The Chief Information Officer regularly presented an IT investment plan with input from branches to senior management. Planned IT expenditures were assigned prioritization and achievability scores according to Treasury Board and internal criteria. Some investments related to surveillance IT issues were included in the IT investment plan. However, Tier 1 and Tier 2 committee records provided little evidence of discussions or decisions related to surveillance-specific IT issues.

Conclusion

Communication of widespread IT surveillance issues via multiple channels has not generally resulted in their satisfactory resolution. The absence of a corporate process for prioritizing and communicating surveillance-related IT challenges, appropriately supported and overseen within the governance structure for surveillance, hinders the Agency's ability to effectively resolve them.

HR Resources

Context

The Agency's 2016-2019 strategic plan for surveillance includes action areas related to human resource capacity, and to innovation. The plan outlines the need for "appropriate competencies and expertise to enhance surveillance capacity", and the related objective of incorporating "innovative practice and technology" into surveillance activities^{Footnote 10}.

What did we expect to find?

We expected to find that branches had considered the workforce capacity and competencies required for surveillance activities, and had identified and addressed any related gaps. We also expected to find that specialists who perform activities outside of their typical duties, such as contracting or negotiating, had received adequate training and tools to perform these activities.

Findings

Branches had estimated annual staffing levels and identified HR-related gaps or risks at a high level, including the need for recruitment, prompted by an aging workforce and the need to retain and engage employees. Related mitigation actions included workplace wellness activities, student recruitment strategies, and employee engagement activities.

Branches identified the challenge of acquiring specialist competencies, such as mathematical modelling and big data analytics. We found little evidence that branches had developed comprehensive strategies for addressing their future competency needs, with the exception of one high-level branch document. The Surveillance Learning Framework, a non-mandatory tool to assess and develop surveillance-related competencies including those related to big data and to data science, was approved by the Surveillance Integration Team in 2018. However, this tool has not been made widely accessible to employees.

A survey of epidemiologists and biostatisticians found that specialists spent a median of 25% of their time on activities outside of their typical duties, such as contracting and negotiating. A majority of respondents who conducted these activities reported that they lacked the training and tools to perform them efficiently.

Conclusion

Branches have considered the adequacy of their HR capacity and have identified gaps at a high level. However, there is little evidence that branches have comprehensively identified their future competency needs for surveillance or developed strategies to address those needs. Furthermore, we conclude that epidemiologists and biostatisticians who perform activities outside of their typical duties may lack the training and tools or adequate support to perform them efficiently.

Sex and Gender-Based Analysis

Context

The policy of the Government of Canada's Health Portfolio is to use sex- and gender-based analysis plus (SGBA+) to develop, implement and evaluate the Health Portfolio's research, legislation, policies, programs, and services^{Footnote 11}. Specifically, the policy requires that all staff be responsible for using SGBA+ in their work, as appropriate.

What did we expect to find?

We expected to find that the Agency incorporates SGBA+ into surveillance activities, and provides the oversight, training, and tools necessary to ensure its implementation.

Findings

Senior management messaging has prioritized SGBA+, and Agency resources have been committed to ensure its ongoing implementation. A unit within HPCDP acts as a focal point for the Agency to support the implementation of SGBA+, including by facilitating online and in-person training, and by recording self-reported training results. The Agency has also developed SGBA+ support tools, such as the Toward Health Equity: A Guide to Sex and Gender-based Analysis in Agency Programs and Policies, to support staff in the mandatory application of SGBA+ to program and policy development.

The Agency's surveillance systems often rely on external providers for their data, and frequently incorporate disaggregated data by sex, age, and location. However, the Agency's analysis may be limited to the extent that data providers do not collect data elements related to SGBA+, particularly related to gender, and may also be limited by privacy considerations. Furthermore, the Agency's surveillance IT infrastructure may not be nimble enough to fully support implementation of SGBA+.

Through reviews conducted in 2014 and 2017, the Agency considered whether SGBA+ was included in scientific publications authored or published by the Agency. The publishing processes for the Agency's two journals (CCDR and HPCDP Journal) encourage authors to consider SGBA+ where appropriate.

A third review, published in March 2018, examined the application of SGBA+ in surveillance products, including reports and infographics^{Footnote 12}. The review found that most surveillance products included an element of SGBA+, but noted several areas for improvement and issued four related recommendations. These included recommendations to incorporate sex-based data disaggregation where appropriate, to collect data on additional indicators of social stratification, such as household income, and to expand options for sex-based data collection where appropriate.

Conclusion

The Agency provides the necessary oversight, training, and tools to ensure implementation of SGBA+, but additional opportunities exist to incorporate SGBA+ into surveillance data collection and analysis.

Appendix A - Scorecard

Audit of Surveillance

Criterion	Risk RatingFootnote *	Risk to Surveillance without Implementing Recommendation	Rec. #
Strategic Direction	4	The lack of redistributed leadership responsibilities and the associated ineffectiveness of the SIT as a decision-making body reporting to senior management committees have impeded the implementation of deliverables articulated by the 2016 and 2013 strategic surveillance plans. These impediments pose continued risks to the effective coordination of surveillance activities and the implementation of horizontal initiatives.	1
Corporate Decision-Making	3	The absence of a corporate assessment process for surveillance activities may hinder senior management's capacity to make informed strategic decisions or to demonstrate the continued relevance of surveillance activities. It may also impede the management of risks related to operating in a siloed environment.	2
Data Acquisition	3	The risks posed by a lack of formal data sharing agreements, as previously identified by the Auditor General and by internal audits, persist. The pace of MLISA annex development presents the risk that the value of this high-profile information sharing agreement will not be realized as anticipated.	3
User Needs and Surveillance Products	2	The absence of a systematic and documented approach to monitoring may impede the provision of surveillance products to those who need them, with the characteristics and timeliness that meet users' needs.	4
Surveillance IT Infrastructure and Human Resource	3	The absence of a corporate process for prioritizing and communicating IT challenges, appropriately supported and overseen within the governance structure for surveillance, hinders the Agency's ability to effectively resolve surveillance-related IT challenges.	5
	2	Branches may not identify their future competency needs for surveillance and develop strategies to address those needs. Epidemiologists and biostatisticians who perform activities outside of their typical duties may lack the training and tools or adequate support to perform them efficiently.	6
SGBA+	2	The Agency provides the necessary oversight, training, and tools to ensure the implementation of SGBA+, but additional opportunities exist to incorporate SGBA+ into surveillance data collection and analysis.	7
Footnote 1 Residual risk without implementing the recommendation. Return to footnote * referrer

1. Minimal Risk
2. Minor Risk
3. Moderate Risk
4. Significant Risk
5. Major Risk

Appendix B - About the Audit

1. Audit objective

The objective of the audit was to determine whether surveillance resources and activities are effectively managed to support the collection and analysis of quality data, the translation of evidence into surveillance products, and the timely dissemination of information that addresses users' needs.

2. Audit scope

The scope of this audit included surveillance-related activities within IDPC, HPCDP, and HSIB for the period April 1, 2017 to March 31, 2019. More specifically, the audit focused on governance mechanisms, output monitoring and reporting, data acquisition, and surveillance resources and infrastructure management.

Activities not in Scope

Data privacy and security were not examined in detail, due to a current audit of the Management of Privacy Practices that will determine whether personal information at HC and PHAC is effectively protected.

The audit did not examine processes and activities related to emergency preparedness or the management of outbreaks, due to the recent World Health Organization Joint External Evaluation (JEE) mission that assessed Canada's capacity to prevent, detect, and rapidly respond to public health threats.

The audit criteria are presented in Appendix C.

3. Audit approach

The audit was conducted at the Public Health Agency of Canada's headquarters. The principal audit procedures included, but were not limited to:

• walkthroughs and interviews with IDPC, HPCDP, and HSIB key officials responsible for surveillance activities;
• review of documentation, policies, standards, guidelines, and frameworks related to surveillance, as well as all relevant information gathered from previous audits to minimize duplication;
• detailed testing of related controls over surveillance activities; and
• analysis of findings from interviews, inquiry, document reviews, and detailed testing.

4. Statement of conformance

This audit was conducted in conformance with the International Standards for the Professional Practice of Internal Auditing and is supported by the results of the Office of Audit and Evaluation's Quality Assurance and Improvement Program.

Appendix C - Audit Criteria

The audit criteria were derived from the TBS Core Management Controls and the COSO Enterprise Risk Management Framework.

The following audit criteria were used to conduct the audit:

Audit of Surveillance

Audit Criteria

1. Adequate governance mechanisms are in place to provide strategic direction and ensure a coordinated approach to surveillance activities.
2. Surveillance resources (IT and HR) are effectively leveraged to support the achievement of surveillance objectives.
3. Surveillance outputs are regularly monitored by centres and branches to ensure timely release of information that addresses users' needs.
4. Periodic reviews of surveillance activities informs strategic corporate decisions.
5. Effective processes are in place to ensure the acquisition of quality data.
6. Surveillance activities consider sex- and gender-based analysis (SGBA+).