Section 1: Evaluation of the surveillance function at the Public Health Agency of Canada – Introduction
The purpose of this evaluation is to assess the relevance and performance of the Public Health Agency’s surveillance function. Senior management requested this evaluation to support decision making regarding the Public Health Agency’s initiative to transform the public health surveillance function and the development of the 2013-2016 Surveillance Strategic PlanFootnote 17. Senior management also required evaluative information to respond to:
- the 2009 Treasury Board of Canada Secretariat’s Policy on EvaluationFootnote 18 which requires comprehensive evaluation coverage of all direct program spending on a five-year cycle
- the 2008 May Report of the Auditor General of CanadaFootnote 19 reviewing the surveillance of infectious diseases, which recommended that the Public Health Agency periodically evaluate its surveillance systems.
The evaluation addresses the relevance of the Public Health Agency’s surveillance function, including the continued need for the surveillance function, its alignment with the priorities of the Government of Canada, the Public Health Agency and federal roles and responsibilities. The evaluation also addresses the effectiveness of setting priorities for surveillance, the Public Health Agency’s progress in achieving the goals of its 2007 Surveillance Strategic PlanFootnote 20 and opportunities for greater efficiency and economy. The evaluation covers the time period from 2006-07 to 2011-12.
Public health surveillance is one of six core public health functions. The Public Health Agency’s Surveillance Strategic Plan is the key planning document for the Public Health Agency’s surveillance function. It defines surveillance as:
the tracking and forecasting of any health event or health determinant through the on-going collection of data, the integration, analysis and interpretation of data into surveillance products, and the dissemination of those surveillance products to those who need to know in order to undertake necessary actions or responses. The overarching purpose of surveillance is to generate information and knowledge for public health action in the short and long term.”Footnote 21
Public health surveillance encompasses the development and maintenance of surveillance systems that provide information on Canadian rates and trends, and early warning signals for the detection of urgent public health events. Public health surveillance is an essential component of effective public health practice because it provides evidence to inform governments and stakeholders (e.g. Canadian residents, public health professionals and experts and non-governmental organizations) when making policy and program decisions that promote and protect the health of Canadians. Surveillance supports the Minister of Health by informing action on federal priorities.
Surveillance is considered a cross-cutting activity within the Public Health Agency because surveillance-related activities are spread across all major program areas and are applied to a range of health issues including communicable diseases, non-communicable diseases and injuries, healthy child development and environmental health.
The Public Health Agency has approximately 50 surveillance systems and data holdings that are used to generate reports, bulletins and health advisories and to inform the development of programs and policies (Appendix A). To be classified as a surveillance system, a system must comprise the full cycle of activities described in the definition above. In addition to surveillance program areas, certain corporate groups play a significant role in supporting public health surveillance activities, for example the Communications and Information Management/Information Technology teams.
The Surveillance Framework and Evaluation Working Group (under the Surveillance Integration Team) created a conceptual model for the Public Health Agency’s public health surveillance function to show the complex interplay among the various components that contribute to the achievement of the expected outcomes (Figure 1).
The conceptual model shows that the public health surveillance function includes a number of foundational surveillance infrastructure elements, namely the following supporting processes and structures necessary for public health surveillance:
- Authorizing environment refers to the sources of legitimacy for the surveillance function.
- Operational capacity (core and enhanced elements) refers to the resources and capabilities associated with public health surveillance, and includes the core activities of surveillance (i.e. data collection, integration, analysis and interpretation, and dissemination).
- Enablers of operational capacity refers to training and professional development and to elements associated with public health information, including business requirements, hardware platforms, software, data standards, systems development, maintenance and support.
- Management, monitoring and evaluation refers to management capacity, governance mechanisms and accountability mechanisms that ensure that the function and individual surveillance systems are working as intended.
- Stakeholder engagement and collaboration refers to collaborative and coordination activities with internal and external stakeholders, as well as to engagement and consultation with internal and external stakeholders to develop, implement and maintain surveillance activities and systems.
The expected outcomes of the public health surveillance function are information and knowledge, which in turn inform public health policy and guide action to improve public health outcomes. The essence of the model’s theory is that decisions are informed by public health surveillance and will lead to actions that contribute to improved health outcomes among Canadians.
For a detailed description of the components of the conceptual model, refer to Appendix B.
Figure 1: Conceptual model of the Public Health Agency’s surveillance functionFootnote 22
Text Equivalent - Figure 1
The conceptual model draws together and illustrates the major components (shown as coloured element in the diagram which are Resource Investment, Authorizing Environment; Management; Monitoring & Evaluation; Operational Capacity Core & Enhanced Elements; Enablers of Operational Capacity; Stakeholder Engagement & Collaboration; External Influences; and Other Information) in the larger context of public health and, attempts to represent the dynamic nature in which surveillance activity ‘lives’ and evolves through the interplay of these components".
Source: Public Health Agency of Canada. Surveillance Framework and Evaluation Working Group. (2009). Towards a Surveillance Results Model for the Public Health Agency of Canada: Integrated Framework for Surveillance – A Conceptual Model.
The scope of this evaluation includes an examination of the core activities of surveillance: data collection; integration, analysis and interpretation of data into surveillance products; and the dissemination of those surveillance products to intended audiences. It also includes an examination of the infrastructure that supports and facilitates core surveillance activities.
During the time that this evaluation was conducted, the Public Health Agency had approximately 50 surveillance systems. An illustration of the Public Health Agency’s surveillance systems by branch and centre can be found in Appendix A. Evaluating the activities and performance of each of these surveillance systems was beyond the scope of this evaluation. Instead, individual systems will be examined as part of any upcoming evaluations of the specific programs they support.
The evaluation issues and questions in Table 2 guided the development of the evaluation’s data collection instruments and the collection of data. All of these questions were covered by the evaluation. Key findings for each question were then synthesized into broader conclusions.
The evaluation was designed and conducted by the Public Health Agency of Canada/Health Canada Evaluation Directorate, a group internal to the Public Health Agency but not involved in the program areas responsible for surveillance activities.
The Evaluation of the Surveillance Function at the Public Health Agency of Canada: Evaluation Methodology: Technical ReportFootnote 23 contains a detailed description of the evaluation methodology.
An extensive evaluation scoping and planning exercise took place from April to December 2011. This included:
- a review of Public Health Agency plans, strategies and program-specific surveillance documents and reports
- a preliminary review of the published literature on surveillance
- a review of past and current assessments by the Auditor General
- a review of the conceptual model (logic model) for the surveillance function
- 23 semi-structured key informant interviews with four assistant deputy ministers/associate deputy ministers, 17 directors general, surveillance managers, and managers and staff of corporate service groups supporting surveillance activities, and two external stakeholders)
- the development of an evaluation framework.
The evaluation incorporated multiple lines of evidence, combining qualitative and quantitative information to ensure a balanced analysis of the relevance and performance of the surveillance function (see Table 3).
The data from each line of evidence were summarized under the appropriate evaluation issue and question, and then triangulated to substantiate the conclusions and findings.
|Line of evidence||Sources|
Evaluators reviewed 15 peer-reviewed surveillance articles to explore the key issues and emerging developments in public health surveillance in Canada and internationally. The purpose of the review was to obtain information about the need for the Public Health Agency’s surveillance function. Articles were identified using a key word search of library databases.
Review of surveillance practices in England, the United States, Australia, Sweden, and the European Union
A consulting firm that reported to the evaluation manager reviewed surveillance practices to:
Administrative file and document review
Evaluation Services reviewed the available files and documents about surveillance activities at the Public Health Agency. This review addressed the evaluation issues related to relevance and performance.
Files and documents were initially identified by staff of the Surveillance Coordination Unit and through a search of relevant websites. Other program-specific documents were identified during interviews. Program- and system-specific data from the document review were analysed by evaluation issue and questions and findings rolled up to the function level.
The types of documents reviewed included:
Key informant interviews
The interviews addressed particular evaluation issues related to relevance, including roles and responsibilities, alignment with Government of Canada and Public Health Agency priorities and alignment with federal roles and responsibilities. The interviews also solicited input on areas for improvement.
Individuals with involvement or experience in public health surveillance, either within the Public Health Agency or externally, were targeted for the interviews. An interview guide was developed for each category of key informant.
The breakdown of interviewees by type of key informant is as follows:
A total of 83 one-hour (on average) interviews were conducted. These interviews were audio recorded. The majority were transcribed (verbatim) and uploaded to NVivo software for analysis. Where transcription was not possible due to time constraints, notes from the interviews were used for analysis.
Online survey of Public Health Agency managers and staff working in the area of surveillance
The online survey requested input about surveillance activities, emerging issues and concerns, the needs of partners and stakeholders, perceptions of the federal role, awareness of the Public Health Agency’s vision for surveillance, processes for setting surveillance priorities and performance measurement. The online survey targeted all Public Health Agency surveillance program staff, and managers who had not been interviewed.
Fluid Surveys software was used to deliver the survey and analyse the data.
The survey was sent to 180 potential respondents. 107 responses were obtained for a response rate of 59 per cent. The Evaluation of the Surveillance Function at the Public Health Agency of Canada: Evaluation Methodology: Technical ReportFootnote 24 contains the survey questionnaire and a detailed breakdown of survey results.
Due to the complexity of the surveillance function, the evaluation team engaged an independent advisor to provide subject matter expertise as required during all stages of the evaluation. The advisor has an academic background in surveillance and extensive professional experience in this field. The advisor reviewed the evaluation framework, key interview guides, the report on the surveillance function in other countries and the European Union, as well as the final evaluation report.
This section describes the challenges encountered during the evaluation, and the limitations in the design and methods that may have implications for the validity and reliability of evaluation findings, conclusions and recommendations. This section also identifies the mitigation strategies put in place by the evaluation team to ensure that the evaluation findings can be used with confidence to guide program planning and decision making.
The evaluation was initially designed to address issues of relevance. Data collection was nearing completion when the scope was expanded to include performance to ensure that the Public Health Agency would meet the requirements of the Policy on Evaluation.Footnote 25
Although the survey had been launched and most key informant interviews were complete, many key informants had identified gaps in the implementation of the goals outlined in the 2007 Surveillance Strategic PlanFootnote 26, as well as other issues of performance. The evaluation team used this information to assess performance and further supplemented it with performance data from the reviews of documents and administrative files.
A surveillance performance measurement framework was under development but had not been implemented at the time of the evaluation. As a consequence, comprehensive performance measurement data was unavailable at the function level for the period covered by the evaluation.
Despite the lack of performance data, the evaluation team assessed performance by examining the Public Health Agency’s progress in achieving the goals of the 2007 Surveillance Strategic PlanFootnote 27. As well, performance data was collected through interviews and the document review. Findings from these multiple lines of evidence were triangulated to allow for the validation of findings on performance.
Given the decentralized nature of surveillance within the Public Health Agency, analysis was complicated. There was a lack of a clear and consistent definition of surveillance and the allocation of resources to surveillance activities was not consistently recorded. Therefore the evaluation team was unable to access reliable resource allocation data for the period covered by the evaluation (2006-07 to 2011-12). Given the evolving definition of surveillance, capturing reliable data on the allocation of resources to surveillance activities remains an issue.
During the period of the evaluation a number of changes occurred within the Public Health Agency. These changes affected some of the surveillance activities, governance structures and resource allocations. As a result, a major challenge for the evaluation was to ensure that the conclusions and recommendations remained relevant as the surveillance function evolved.
Given this changing environment, the evaluation was planned and executed as a developmental evaluation, which is defined as an evaluation that “adapts to the realities of complex non-linear dynamics rather than trying to impose order and certainty on a disorderly and uncertain world.”Footnote 28 The primary use of this approach is to support real-time feedback, learning, and changes in direction. The developmental aspect of the evaluation actively helped shape the ongoing surveillance initiative.
The challenge in keeping up with ongoing changes was also mitigated by regular update meetings with the program staff responsible for the surveillance transformation and strategic planning initiatives. As well, an evaluation team member attended the meetings of the surveillance Subject Matter Expert Reference Group as an observer.
- Date modified: