Section 1: Evaluation of the surveillance function at the Public Health Agency of Canada – Introduction

1. Introduction

The purpose of this evaluation is to assess the relevance and performance of the Public Health Agency’s surveillance function. Senior management requested this evaluation to support decision making regarding the Public Health Agency’s initiative to transform the public health surveillance function and the development of the 2013-2016 Surveillance Strategic Plan^{Footnote 17}. Senior management also required evaluative information to respond to:

• the 2009 Treasury Board of Canada Secretariat’s Policy on Evaluation^{Footnote 18} which requires comprehensive evaluation coverage of all direct program spending on a five-year cycle
• the 2008 May Report of the Auditor General of Canada^{Footnote 19} reviewing the surveillance of infectious diseases, which recommended that the Public Health Agency periodically evaluate its surveillance systems.

The evaluation addresses the relevance of the Public Health Agency’s surveillance function, including the continued need for the surveillance function, its alignment with the priorities of the Government of Canada, the Public Health Agency and federal roles and responsibilities. The evaluation also addresses the effectiveness of setting priorities for surveillance, the Public Health Agency’s progress in achieving the goals of its 2007 Surveillance Strategic Plan^{Footnote 20} and opportunities for greater efficiency and economy. The evaluation covers the time period from 2006-07 to 2011-12.

1.1 What is the surveillance function at the Public Health Agency of Canada?

Public health surveillance is one of six core public health functions. The Public Health Agency’s Surveillance Strategic Plan is the key planning document for the Public Health Agency’s surveillance function. It defines surveillance as:

“the tracking and forecasting of any health event or health determinant through the on-going collection of data, the integration, analysis and interpretation of data into surveillance products, and the dissemination of those surveillance products to those who need to know in order to undertake necessary actions or responses. The overarching purpose of surveillance is to generate information and knowledge for public health action in the short and long term.”^{Footnote 21}

Public health surveillance encompasses the development and maintenance of surveillance systems that provide information on Canadian rates and trends, and early warning signals for the detection of urgent public health events. Public health surveillance is an essential component of effective public health practice because it provides evidence to inform governments and stakeholders (e.g. Canadian residents, public health professionals and experts and non-governmental organizations) when making policy and program decisions that promote and protect the health of Canadians. Surveillance supports the Minister of Health by informing action on federal priorities.

Surveillance is considered a cross-cutting activity within the Public Health Agency because surveillance-related activities are spread across all major program areas and are applied to a range of health issues including communicable diseases, non-communicable diseases and injuries, healthy child development and environmental health.

The Public Health Agency has approximately 50 surveillance systems and data holdings that are used to generate reports, bulletins and health advisories and to inform the development of programs and policies (Appendix A). To be classified as a surveillance system, a system must comprise the full cycle of activities described in the definition above. In addition to surveillance program areas, certain corporate groups play a significant role in supporting public health surveillance activities, for example the Communications and Information Management/Information Technology teams.

The Surveillance Framework and Evaluation Working Group (under the Surveillance Integration Team) created a conceptual model for the Public Health Agency’s public health surveillance function to show the complex interplay among the various components that contribute to the achievement of the expected outcomes (Figure 1).

The conceptual model shows that the public health surveillance function includes a number of foundational surveillance infrastructure elements, namely the following supporting processes and structures necessary for public health surveillance:

• Authorizing environment refers to the sources of legitimacy for the surveillance function.
• Operational capacity (core and enhanced elements) refers to the resources and capabilities associated with public health surveillance, and includes the core activities of surveillance (i.e. data collection, integration, analysis and interpretation, and dissemination).
• Enablers of operational capacity refers to training and professional development and to elements associated with public health information, including business requirements, hardware platforms, software, data standards, systems development, maintenance and support.
• Management, monitoring and evaluation refers to management capacity, governance mechanisms and accountability mechanisms that ensure that the function and individual surveillance systems are working as intended.
• Stakeholder engagement and collaboration refers to collaborative and coordination activities with internal and external stakeholders, as well as to engagement and consultation with internal and external stakeholders to develop, implement and maintain surveillance activities and systems.

The expected outcomes of the public health surveillance function are information and knowledge, which in turn inform public health policy and guide action to improve public health outcomes. The essence of the model’s theory is that decisions are informed by public health surveillance and will lead to actions that contribute to improved health outcomes among Canadians.

For a detailed description of the components of the conceptual model, refer to Appendix B.

Figure 1: Conceptual model of the Public Health Agency’s surveillance function^{Footnote 22}

Enlarge Figure 1

Source: Public Health Agency of Canada. Surveillance Framework and Evaluation Working Group. (2009). Towards a Surveillance Results Model for the Public Health Agency of Canada: Integrated Framework for Surveillance – A Conceptual Model.

1.2 Scope of the evaluation

The scope of this evaluation includes an examination of the core activities of surveillance: data collection; integration, analysis and interpretation of data into surveillance products; and the dissemination of those surveillance products to intended audiences. It also includes an examination of the infrastructure that supports and facilitates core surveillance activities.

During the time that this evaluation was conducted, the Public Health Agency had approximately 50 surveillance systems. An illustration of the Public Health Agency’s surveillance systems by branch and centre can be found in Appendix A. Evaluating the activities and performance of each of these surveillance systems was beyond the scope of this evaluation. Instead, individual systems will be examined as part of any upcoming evaluations of the specific programs they support.

1.3 Evaluation issues and questions

The evaluation issues and questions in Table 2 guided the development of the evaluation’s data collection instruments and the collection of data. All of these questions were covered by the evaluation. Key findings for each question were then synthesized into broader conclusions.

1.4 Evaluation methodology

The evaluation was designed and conducted by the Public Health Agency of Canada/Health Canada Evaluation Directorate, a group internal to the Public Health Agency but not involved in the program areas responsible for surveillance activities.

The Evaluation of the Surveillance Function at the Public Health Agency of Canada: Evaluation Methodology: Technical Report^{Footnote 23} contains a detailed description of the evaluation methodology.

1.4.1 Evaluation planning

An extensive evaluation scoping and planning exercise took place from April to December 2011. This included:

• a review of Public Health Agency plans, strategies and program-specific surveillance documents and reports
• a preliminary review of the published literature on surveillance
• a review of past and current assessments by the Auditor General
• a review of the conceptual model (logic model) for the surveillance function
• 23 semi-structured key informant interviews with four assistant deputy ministers/associate deputy ministers, 17 directors general, surveillance managers, and managers and staff of corporate service groups supporting surveillance activities, and two external stakeholders)
• the development of an evaluation framework.

1.4.2 Evaluation methods

The evaluation incorporated multiple lines of evidence, combining qualitative and quantitative information to ensure a balanced analysis of the relevance and performance of the surveillance function (see Table 3).

The data from each line of evidence were summarized under the appropriate evaluation issue and question, and then triangulated to substantiate the conclusions and findings.

1.4.3 Technical advisor

Due to the complexity of the surveillance function, the evaluation team engaged an independent advisor to provide subject matter expertise as required during all stages of the evaluation. The advisor has an academic background in surveillance and extensive professional experience in this field. The advisor reviewed the evaluation framework, key interview guides, the report on the surveillance function in other countries and the European Union, as well as the final evaluation report.

1.5 Limitations

This section describes the challenges encountered during the evaluation, and the limitations in the design and methods that may have implications for the validity and reliability of evaluation findings, conclusions and recommendations. This section also identifies the mitigation strategies put in place by the evaluation team to ensure that the evaluation findings can be used with confidence to guide program planning and decision making.

1.5.1 Expansion of scope to include performance

The evaluation was initially designed to address issues of relevance. Data collection was nearing completion when the scope was expanded to include performance to ensure that the Public Health Agency would meet the requirements of the Policy on Evaluation.^{Footnote 25}

Although the survey had been launched and most key informant interviews were complete, many key informants had identified gaps in the implementation of the goals outlined in the 2007 Surveillance Strategic Plan^{Footnote 26}, as well as other issues of performance. The evaluation team used this information to assess performance and further supplemented it with performance data from the reviews of documents and administrative files.

1.5.2 Lack of performance measurement data

A surveillance performance measurement framework was under development but had not been implemented at the time of the evaluation. As a consequence, comprehensive performance measurement data was unavailable at the function level for the period covered by the evaluation.

Despite the lack of performance data, the evaluation team assessed performance by examining the Public Health Agency’s progress in achieving the goals of the 2007 Surveillance Strategic Plan^{Footnote 27}. As well, performance data was collected through interviews and the document review. Findings from these multiple lines of evidence were triangulated to allow for the validation of findings on performance.

1.5.3 Lack of a clear and consistent definition of surveillance

Given the decentralized nature of surveillance within the Public Health Agency, analysis was complicated. There was a lack of a clear and consistent definition of surveillance and the allocation of resources to surveillance activities was not consistently recorded. Therefore the evaluation team was unable to access reliable resource allocation data for the period covered by the evaluation (2006-07 to 2011-12). Given the evolving definition of surveillance, capturing reliable data on the allocation of resources to surveillance activities remains an issue.

1.5.4 Ongoing changes to the surveillance function

During the period of the evaluation a number of changes occurred within the Public Health Agency. These changes affected some of the surveillance activities, governance structures and resource allocations. As a result, a major challenge for the evaluation was to ensure that the conclusions and recommendations remained relevant as the surveillance function evolved.

Given this changing environment, the evaluation was planned and executed as a developmental evaluation, which is defined as an evaluation that “adapts to the realities of complex non-linear dynamics rather than trying to impose order and certainty on a disorderly and uncertain world.”^{Footnote 28} The primary use of this approach is to support real-time feedback, learning, and changes in direction. The developmental aspect of the evaluation actively helped shape the ongoing surveillance initiative.

The challenge in keeping up with ongoing changes was also mitigated by regular update meetings with the program staff responsible for the surveillance transformation and strategic planning initiatives. As well, an evaluation team member attended the meetings of the surveillance Subject Matter Expert Reference Group as an observer.