Evaluation of the Surveillance Function at the Public Health Agency of Canada

Prepared by
Evaluation Directorate
Public Health Agency of Canada/Health Canada
January 14, 2013

Table of contents

• 1. Introduction
  • 1.1 What is the surveillance function at the Public Health Agency of Canada?
  • 1.2 Scope of the evaluation
  • 1.3 Evaluation issues and questions
  • 1.4 Evaluation methodology
  • 1.5 Limitations
• 2. Background and context
  • 2.1 The Public Health Agency’s surveillance activities
  • 2.2 The Public Health Agency’s 2007 Surveillance Strategic Plan
  • 2.3 Resource allocation
  • 2.4 The environment for public health surveillance
  • 2.5 Governance and accountability of surveillance
  • 2.6 Partners and beneficiaries
• 3. Findings – relevance
  • 3.1 Continued need
  • 3.2 Alignment with the priorities of the Government of Canada and the Public Health Agency
  • 3.3 Alignment with federal roles and responsibilities
• 4. Findings - performance
  • 4.1 Effectiveness of setting surveillance priorities
  • 4.2 Achievement of the goals of the 2007 Surveillance Strategic Plan
  • 4.3 Efficiency and economy
• 5. Evaluation conclusions and recommendations
  • 5.1 Conclusions
  • 5.2 Recommendations
• Appendix A. Public Health Agency of Canada surveillance systems by branch and centre
• Appendix B. Description of the components of the conceptual model
• Appendix C. Surveillance governance model in effect during the evaluation
• Appendix D. Governance framework for the 2013-2016 Surveillance Strategic Plan
• Appendix E. Pan-Canadian Public Health Network governance structure
• Appendix F. Federal, provincial and territorial authorities for public health surveillance

Executive summary

This report presents the results of an evaluation of the surveillance function at the Public Health Agency of Canada covering the period from 2006-07 to 2011-12. This evaluation was undertaken by the Public Health Agency of Canada/Health Canada’s Evaluation Directorate.

Evaluation scope, purpose and methodology

The initial focus of the evaluation was the relevance of the Public Health Agency’s surveillance function. While the evaluation was underway, its scope was expanded to include some aspects of performance, using the Public Health Agency’s progress in achieving the goals and outcomes of the 2007 Surveillance Strategic Plan^{Footnote 1} as a proxy measure of performance. The evaluation also examined opportunities to improve the efficiency of the Public Health Agency’s surveillance function. The evaluation considered the context of the evolving nature of the function and recent transformation initiatives in the assessment of the performance of the surveillance function.

The relevance and performance of individual surveillance systems and surveillance programs are outside the scope of this evaluation.

Senior management requested this evaluation to support decision making regarding the Public Health Agency’s initiative to transform the public health surveillance function and the development of the 2013-2016 Surveillance Strategic Plan^{Footnote 2}. Senior management also required evaluative information to respond to:

• the 2009 Treasury Board of Canada Secretariat’s Policy on Evaluation^{Footnote 3} which requires comprehensive evaluation coverage of all direct program spending on a five-year cycle
• the 2008 May Report of the Auditor General of Canada^{Footnote 4} reviewing the surveillance of infectious diseases, which recommended that the Public Health Agency periodically evaluate its surveillance systems.

During the period of the evaluation, a number of changes occurred within the Public Health Agency, affecting some of the surveillance activities, governance structures and resource allocations. As a result, the evaluation was planned and executed as a developmental evaluation, which is defined as an evaluation that “adapts to the realities of complex non-linear dynamics rather than trying to impose order and certainty on a disorderly and uncertain world”^{Footnote 5}. The developmental aspect of the evaluation actively helped shape the ongoing surveillance initiative.

Information from multiple sources was used, including:

• a literature review
• a review of surveillance practices in England, the United States, Australia, Sweden and the European Union
• administrative file and document reviews
• key informant interviews with Public Health Agency staff and external stakeholders
• a survey of Public Health Agency staff working in surveillance.

The Public Health Agency’s surveillance function

Public health surveillance is one of six core public health functions. It involves:

• the tracking and forecasting of any health event or health determinant through the ongoing collection of data
• the integration, analysis and interpretation of data into surveillance products
• the dissemination of surveillance products to those who need to know in order to undertake necessary actions or responses.

Provinces and territories are important partners with and contributors of surveillance data to the Public Health Agency because of their responsibility for health services and as custodians for many administrative and clinical health data collections, e.g. hospitalizations, laboratory test results and drug prescriptions. Therefore, the Public Health Agency works closely with the provinces and territories to ensure a coordinated pan-Canadian approach to public health surveillance.

The evaluation refers to the Public Health Agency’s strategic outcome (the promotion of health, reduced health inequalities and the prevention and mitigation of disease and injury), and to the Program Alignment Architecture and surveillance governance structure that were in effect during the period covered by the evaluation. A new surveillance governance framework was introduced in 2012 and a new Public Health Agency Program Alignment Architecture, including a new strategic outcome for the Public Health Agency, is pending approval.

The Public Health Surveillance Sub-Activity (1.2.1) has been part of the Public Health Agency’s Program Alignment Architecture since 2010-11, and includes three Sub-Sub-Activities: Surveillance of Risk Factors and Determinants of Health (1.2.1.1), Surveillance of Chronic Disease (1.2.1.2) and Surveillance of Infectious Disease (1.2.1.3). In 2010-11, the Public Health Agency invested approximately $63 million in public health surveillance across these three Sub-Sub-Activities.

The Office of Public Health Practice is responsible for the coordination of the Public Health Surveillance Sub-Activity, but surveillance programs, surveillance systems and budgetary authority are spread across Public Health Agency programs. The Public Health Agency has approximately 50 surveillance systems and data holdings managed by its program areas.

During the period of this evaluation, the Public Health Agency’s surveillance activities were guided by the 2007 Surveillance Strategic Plan^{Footnote 6}. This plan has five broad areas of action:

• governance and organization
• integrated surveillance
• knowledge management
• partnerships and collaboration
• performance measurement, quality assurance and evaluation.

The evaluation was undertaken during a period of transition for the surveillance function. In 2011-12, the function was undergoing changes as a result of surveillance transformation and the development of the 2013-2016 Surveillance Strategic Plan^{Footnote 7}. As well, the Pan-Canadian Public Health Network was restructured in 2011, creating the National Surveillance Infrastructure Task Group under the Public Health Infrastructure Steering Committee. This task group, as well as the pre-existing Multi-lateral Information Sharing Agreement Task Group (maintained under the new structure) are both focussed on issues related to public health surveillance.

Evaluation conclusions and recommendations

The analysis of the information gathered for this evaluation resulted in findings and conclusions about the relevance and performance of the surveillance function (summarized in Table 1), which led to three key recommendations.

Conclusions

Relevance

There is a clear, continued need for public health surveillance due to emerging and re-emerging infections, chronic diseases, and other population health risk factors that continue to pose risks to the health of Canadians, with associated costs to Canadian society. Public health surveillance is needed to identify risk factors and determinants of health, to monitor these risks to the health of Canadians, and to inform policy and program decisions for upstream interventions and other prevention and control measures. By extension, an effective and responsive public health surveillance function requires that the surveillance infrastructure, which consists of the structures and processes required to enable the core public health function of surveillance, continues to be supported.

The Public Health Agency’s surveillance function aligns with the Government of Canada’s objectives and priorities by working towards the objective of healthy Canadians. Surveillance activities also enable the Public Health Agency to attain its strategic outcome: “Canada is able to promote health, reduce health inequalities, prevent and mitigate disease and injury”,^{Footnote 8} and support national priorities related to communicable and non-communicable diseases. The public health surveillance function also aligns with objective 2 of The Public Health Agency of Canada - Strategic Plan: 2007–2012, Information – Knowledge − Action^{Footnote 9}: “Ensure actions are supported by integrated information and knowledge functions”.

The Government of Canada and the Public Health Agency of Canada have a mandate and role in public health surveillance. Various acts, including the Department of Health Act^{Footnote 10} and the Public Health Agency of Canada Act^{Footnote 11}, confer authority on the federal government to carry out public health surveillance. Under the International Health Regulations^{Footnote 12}, Canada also has an obligation to carry out surveillance. Although a mandate exists, there is no commonly held understanding of the mandate and its associated vision, roles and responsibilities and priorities for public health surveillance, particularly among provincial and territorial partners.

The federal government has assumed a role in:

• monitoring inter-jurisdictional outbreaks
• providing the national evidence on disease and epidemiologic trends
• liaising with international counterparts
• supporting federal partners to fulfill their surveillance mandates
• building provincial and territorial surveillance and epidemiological capacity
• coordinating national surveillance activities.

These are appropriate roles for the Government of Canada and the Public Health Agency that align with the needs and expectations of the provinces, territories and other stakeholders.

While the public health surveillance function as a whole was not seen to duplicate the work of other federal organizations, the respective roles and responsibilities of the players are not clear. Program complementarity would be improved by closer collaboration among federal partners such as Statistics Canada, Health Canada and the Canadian Food Inspection Agency, as well as with organizations such as the Canadian Institute for Health Information.

Performance

Both the Pan-Canadian Public Health Network and the Public Health Agency have priority-setting processes. However, the Public Health Agency’s formal surveillance priority-setting process is not widely visible because priority setting occurs at the program or centre level.

Performance of the surveillance function was assessed in terms of the Public Health Agency’s progress in aligning its activities to achieve the goals of the 2007 Surveillance Strategic Plan^{Footnote 13}.

Overall, the Public Health Agency has made progress in implementing the initiatives associated with the goals of the 2007 Surveillance Strategic Plan. However, gaps remain in key areas of the public health surveillance infrastructure, including:

• information-sharing agreements
• coordinated and standardized data collection processes, procedures, tools and methods to improve the collection of comparable information
• robust national and interoperable systems
• the provision of timely, accurate and complete information to public health decision makers and Canadians.

Given the decentralized nature of surveillance within the Public Health Agency, analysis was complicated and the evaluation team was unable to assess the overall efficiency and economy of the surveillance function. However, key informants anticipate that greater efficiency and economy could be achieved through:

• standardization, coordination and integration
• data management
• setting surveillance and publication priorities.

Recommendations

The Public Health Agency should take a more strategic approach to public health surveillance including:

• assuming a lead role in developing a shared federal/provincial/territorial vision for the public health surveillance function in Canada
• establishing a formal mechanism and criteria to identify relative priorities for surveillance investments
• developing a more strategic approach to the approval and dissemination of surveillance products and information.

Policy framework for a federated model for public health surveillance in Canada

During the period covered by the evaluation, the Public Health Agency’s surveillance activities were guided by the 2007 Surveillance Strategic Plan^{Footnote 14}. However, the Public Health Agency and its provincial and territorial partners would benefit from an overarching policy framework for a ‘federated’ model for public health surveillance in Canada that sets out a shared vision and the essential elements of a national surveillance function.

This framework should cover both infectious and chronic diseases and include, but not be limited to, a description of:

• federal, provincial, territorial and local mandates
• roles and responsibilities for surveillance
• goals, objectives and operating principles
• surveillance priorities.

It is appropriate that the Public Health Agency take a lead role in developing this framework. However, the cooperation of the provinces and territories is essential to establishing an effective public health surveillance system. Therefore, to achieve a common understanding of a national system for public health surveillance in Canada, this framework should be developed collaboratively and disseminated widely.

Public Health Agency priority setting

Within the Public Health Agency, funding for surveillance is program- or centre-based and priority setting occurs at this level. The Public Health Agency could be more strategic about its surveillance investments by using a formal mechanism, including predefined criteria, to establish relative Public Health Agency-level surveillance priorities. These priorities should align with and support the achievement of the objectives of the Public Health Agency’s Strategic Plan^{Footnote 15}, as well as provincial and territorial and program priorities and information needs. Such a process would assist Public Health Agency staff in making decisions about resource allocation, and would ensure that surveillance investments are directed to areas of priority.

Knowledge management

The Public Health Agency’s role is to provide timely national analysis and surveillance products to public health decision makers and Canadians in a format that is useful to them. The Public Health Agency produces a wide range of knowledge products that are distributed in various formats and at various frequencies. However, several factors were found to affect the Public Health Agency’s ability to produce and disseminate surveillance products in a timely enough manner to assist in the detection of outbreaks or to inform policy and program decision making. These factors include timeliness and accuracy of information provided by the provinces and territories and other data providers, internal content approval processes and communications processes. To ensure that surveillance investments produce useful surveillance information that meets the needs of internal and external audiences, Public Health Agency programs should routinely assess user needs and satisfaction with their surveillance outputs and systematically collect data on the use of their surveillance products and information in policy and program interventions.

In addition, the Public Health Agency should:

• prioritize its surveillance publications and website surveillance content
• determine appropriate frequencies for dissemination based on user needs
• examine alternate approaches for timely approval, production and dissemination of surveillance products.

RECOMMENDATION 1

Take a strategic approach to surveillance that includes, but is not limited to:

1. working with partners and stakeholders to develop a policy framework for a ‘federated’ model for public health surveillance, and an approach for its implementation
2. establishing a formal mechanism and criteria for surveillance priority setting within the Public Health Agency that links surveillance priorities to public health, provincial, territorial and Public Health Agency strategic priorities
3. developing a surveillance knowledge management approach that identifies priority and appropriate surveillance products; examines approaches for their timely approval, production and dissemination; and reflects the needs of multiple audiences.

Integrated surveillance

The term ‘integrated surveillance’ must be defined and communicated. Despite a lack of common understanding, Public Health Agency surveillance managers and staff support integration, and envision integration as including:

• greater coordination and enhanced internal communication across the Public Health Agency’s divisions and centres involved in surveillance
• standardization and common approaches to surveillance within the Public Health Agency, including integration and pooling of data, priority-setting criteria and processes, and data analysis and reporting practices.

All levels of government recognize that greater standardization and harmonization will lead to the production of comparable data across the country. Without a harmonized approach to surveillance, systems are not likely to be interoperable. Areas of opportunity include, but are not limited to, working with surveillance partners on:

• data content, including standards and definitions
• data management processes, including secure transfer and storage of data and data access
• technological platforms and tools
• electronic health records.

Once developed, the Public Health Agency should continue to work with its partners to identify and resolve implementation issues.

RECOMMENDATION 2

The Public Health Agency should assume a leadership role in working with surveillance partners towards greater consistency of surveillance practices and processes, and where feasible, explore opportunities within the Public Health Agency for integration and linkages among its own surveillance systems and programs.

The Public Health Agency can further strengthen its surveillance function by improving its governance and performance measurement functions.

Governance

The Public Health Agency’s internal surveillance governance structure, in particular the Surveillance Integration Team, is effective as a forum for horizontal information exchange among programs and for issue management. The Surveillance Integration Team, comprising surveillance experts from across the Public Health Agency, works closely with the Surveillance Coordination Unit to develop a collaborative approach to surveillance. However, it could play a more effective role in defining surveillance priorities and promoting consistent implementation of surveillance initiatives across the Public Health Agency. While there is no requirement to significantly change the existing governance structure, consistent with the findings of the mid-term governance review, key informants noted that there are opportunities to strengthen the key components of the governance structure, including:

• strengthening the role of the Chief Health Surveillance Officer (formerly the Senior Surveillance Advisor) to provide authority for Public Health Agency-wide implementation of surveillance transformation and to ensure program accountability to routinely report progress on implementation of surveillance priorities
• providing the Surveillance Integration Team with a stronger advisory role
• ensuring that the Surveillance Coordination Unit is structured and adequately resourced to support the functions of the Chief Health Surveillance Officer, as well as to coordinate major horizontal initiatives.

Performance measurement

The 2008 May Report of the Auditor General of Canada^{Footnote 16} identified that the Public Health Agency needed to measure the performance of its surveillance systems. To address gaps in performance data, the Public Health Agency has developed a performance measurement framework for its surveillance systems. The Public Health Agency should finalize and implement the framework, and develop a plan to monitor economy and efficiency.

Generally, Public Health Agency programs already collect some data to measure the performance of their surveillance systems to guide the ongoing improvement and evaluation of these systems. By implementing the framework for its surveillance systems, programs would report using the core set of indicators which would allow for an assessment of the surveillance function. This assessment would determine if the Public Health Agency’s surveillance function is operating effectively, meeting the needs of stakeholders and providing quality data for public health decision making.

RECOMMENDATION 3

Strengthen the governance and accountability of the Public Health Agency’s surveillance function by:

1. providing the components of the governance structure with appropriate authority, accompanied by clearly defined accountabilities for programs
2. finalizing and implementing a Public Health Agency-wide performance measurement framework for its surveillance systems, and developing the necessary tools to monitor economy and efficiency.