Section 4: Evaluation of the surveillance function at the Public Health Agency of Canada – Findings performance
4. Findings - performance
This section examines the overall performance of the Public Health Agency’s surveillance function in terms of its effectiveness in setting surveillance priorities, and progress in achieving the goals and outcomes of the 2007 Surveillance Strategic Plan. In addition, this section provides observations about potential opportunities for greater efficiency and economy of the public health surveillance function.
4.1 Effectiveness of setting surveillance priorities
Finding 7. There are benefits to developing an Agency-level process for determining relative priorities for national surveillance investments.
The World Health Organization has noted the importance of priority setting for public health surveillance:
“…prioritization is part of the process to strengthen a national surveillance system for communicable diseases and can be used as an aid in making decisions for resource allocation. Once priorities are set, the adequacy of the existing surveillance system to cover the most important diseases needs to be reviewed and, if necessary, revised. Alternative methods of surveillance need to be considered and areas for improvement need to be identified.”Footnote 73
Although key informants from the Public Health Agency felt that the surveillance function is clearly a priority, they also felt that the Public Health Agency could improve the method it uses to determine relative priorities for surveillance investments.
The perceived benefits of setting clear surveillance priorities for the Public Health Agency include providing:
- a focus to consider “who is the best one to do it … is it the Public Health Agency or are there others who could do it?”
- a consistent message across the Public Health Agency about the surveillance activities that should receive attention, especially to assist corporate service groups (e.g. Information Management/ Information Technology and Communications teams) who often encounter competing demands for support.
4.1.1 Federal, provincial and territorial priority setting
Findings from the document review and from interviews with provincial and territorial key informants indicate that the Pan-Canadian Public Health Network structure is the mechanism that sets federal, provincial and territorial “consensus-based priorities to help governments focus and refine their public health investments and resources”Footnote 74. Annual reports for the Pan-Canadian Public Health Network indicate that broad priorities flow from the Deputy Ministers of Health and are then translated into priorities and action by the various Pan-Canadian Public Health Network committees and associated task groups. The Public Health Agency facilitates discussions with stakeholders through the Pan-Canadian Public Health Network to identify priorities and to facilitate coordinated action to address national priorities.
Three committees report directly to the Pan-Canadian Public Health Network Council. The committee primarily responsible for surveillance is the Public Health Infrastructure Steering Committee, which is supported by task groups. The task groups most relevant to surveillance are the National Surveillance Infrastructure Task Group and the Multi-lateral Information Sharing Agreement Task Group. (Refer to Appendix E for a description of the Pan-Canadian Public Health Network governance structure for surveillance-related activities).
In general, this governance structure is viewed as working well, although some provincial and territorial key informants did note that its priority-setting process is slow. The transition to the new Pan-Canadian Public Health Network structure in 2011 may have generated some confusion. For example, approximately one-half of provincial and territorial officials interviewed indicated that they were less clear about the surveillance decision-making process under the new structure.
4.1.2 Program-level priority setting
Priority setting practices vary among programs and can include stakeholder and advisory committee collaboration, strategic planning and priority setting exercises, and ongoing tracking, assessment and analysis.
The majority of Public Health Agency key informants and more than one-half (59 per cent) of Public Health Agency survey respondents indicated that their program area has mechanisms in place to identify and establish priorities for their public health surveillance activities. Furthermore, close to two-thirds of the survey respondents (64 per cent) indicated that their program area has mechanisms to assess whether surveillance priorities align with their stakeholder priorities. Despite practices being in place, almost one-half (47 per cent) of survey respondents indicated that there are gaps in priority setting.
4.1.3 Public Health Agency-level priority setting
Although there are priority-setting mechanisms in place at the program level, evidence from the key informant interviews indicate that there are opportunities to improve priority setting at the Public Health Agency level.
More specifically, key informant interviews revealed opportunities to:
- assess relative priorities for Public Health Agency surveillance investments
- ensure priorities are linked to public health, provincial, territorial and Public Health Agency strategic priorities
- ensure priorities continue to reflect national disease control priorities
- improve efficiency
- take advantage of new methods and techniques to strengthen surveillance.
Public Health Agency senior managers generally agreed that there is a need for a collaborative Public Health Agency-level process to establish “strategically determined” surveillance priorities. Some senior managers suggested that the Public Health Agency needs a consistent annual process to have a “collective discussion around priorities”, and that once identified, surveillance should focus on the agreed-upon priorities. As one senior manager noted, “PHAC has not historically focused its activities, which makes it problematic to clearly understand the Public Health Agency’s role. Trying to find a strategy to address everything can be overwhelming and quickly exceeds the available resources.”
Several senior managers pointed out that the priorities of public health surveillance should support the achievement of the strategic priorities of the Public Health Agency, “so the result is less of just a descriptive overview of trends and instead more connected to how we can effectively use surveillance to drive dialogue, debate, discussion and ultimately policy and program decisions that we can use to advance public health outcomes.” One senior manager expressed the risks of not having clear priorities and a well-articulated and understood Public Health Agency-wide surveillance strategy — “if something happened, a new disease emerged, to know what would be the best way for us to collect that information and data has not been established. You cannot set up your surveillance structure in the heat of the moment.”
Some respondents indicated that the existing surveillance governance structure is the appropriate mechanism for setting priorities; however, others noted that the process is unclear to surveillance staff. Public Health Agency key informants attributed the lack of clarity to:
- a lack of clear Public Health Agency-level priorities to use as a basis for surveillance priorities
- the Public Health Agency’s funding model, whereby funding for surveillance goes directly to programs and is not determined by any Public Health Agency-level prioritization exercise
- a lack of authority to prioritize activities across programs or to define the priorities of the organizations that support surveillance, such as communications as well as information management and information technology.
Respondents also noted that the Surveillance Integration Team had started to establish a mechanism for setting surveillance priorities, but it was never finalized.
Key informant interviews pointed to the need to better communicate surveillance priorities once they are set. Currently, the Public Health Agency’s surveillance priorities are not consistently well known within the Public Health Agency or by partners. For example, just over one-half of Public Health Agency survey respondents felt that the surveillance priorities are not known and are not shared within the Public Health Agency. Similarly, 10 of the 13 provincial and territorial interviewees reported that they were not aware of the Public Health Agency’s surveillance priorities.
Finding 8. The Public Health Agency of Canada has made good progress in strengthening the public health surveillance function, and in developing many of the foundational pieces for an integrated approach. However, an overall corporate model for surveillance management and practice is missing.
The Public Health Agency has made significant progress in strengthening the public health surveillance function. For example, the Public Health Agency has:
- reported that all 12 recommendations from the 2008 May Report of the Auditor General of Canada are either fully or substantially implementedFootnote 75
- developed many foundational pieces for a standardized approach and common practices for surveillance, including, for example, a data quality framework (2009)Footnote 76 and a Policy for the Collection, Use and Disclosure of Information Relating to Public Health (2009)Footnote 77, and is finalizing a privacy management framework.
The World Health Organization reports that, with respect to implementation of the International Health Regulations, Canada has the necessary formal and informal contacts and processes in place to efficiently support global early-detection and risk-assessment functions.Footnote 78
The Public Health Agency has done and continues to do significant work in collaborating with the provinces and territories. For example:
- Mechanisms and processes to promote alignment among federal, provincial and territorial surveillance roles and responsibilities are in place through the Pan-Canadian Public Health Network.
- The Public Health Agency and its provincial, territorial and international partners are able to work together and share information to address public health threats during periods of crisis, which was demonstrated during the recent outbreak events of listeriosis and H1N1.
- Significant advancements have taken place in the development of a federal/provincial/territorial Multi-lateral Information Sharing Agreement. The first “critical package” of the agreement that includes the main body (the legal text of the agreement), technical schedules for prioritized chapters (Influenza, Food and Waterborne) and non-technical schedules for governance and data management has been developed.
- The National Surveillance Infrastructure Task Group is developing a blueprint for an integrated national system for public health surveillance in Canada.
Although the Public Health Agency has made progress in developing the foundation for public health surveillance, including tools, strategies, policies and frameworks to promote standardization and consistency in surveillance practices, Public Health Agency-wide implementation of some of the initiatives has been slow. Insights into the slow progress are provided in the February 2012 Annual Status Report on Progress in Implementing May 2008 Auditor General Recommendations - Chapter 5: Surveillance of Infectious Diseases which notes, “While progress continues in all areas of PHAC’s response to the 2008 OAG report, the pace in achieving full implementation in some is being affected by the complexity, the steps required, and the engagement that is needed to move forward in a meaningful way.”Footnote 79
Some key informants stressed the need for a corporate approach to the Public Health Agency’s surveillance function, pointing to such issues as the lack of a strategic Public Health Agency-level priority-setting process, program-level decisions that do not necessarily align with Public Health Agency priorities, and a lack of program accountability for implementing surveillance initiatives.
4.2 Achievement of the goals of the 2007 Surveillance Strategic Plan
This section looks at progress made in achieving the five goals of the 2007 Surveillance Strategic Plan:
- governance and organization
- integrated surveillance
- knowledge management
- partnerships and collaboration
- performance measurement and quality assurance.
4.2.1 Governance and organization
Finding 9. The surveillance governance structure is effective as a forum for horizontal information exchange and the management of issues.
The 2007 Surveillance Strategic Plan goal for governance and organization is “to create, implement and maintain a responsive governance structure for surveillance within the Public Health Agency of Canada, accountable to the Chief Public Health Officer and with cross-organizational representation that ensures alignment with Public Health Agency priorities”.Footnote 80
The Public Health Agency’s internal surveillance governance structure, in particular the Surveillance Integration Team, provides a forum for horizontal information exchange among programs and between the Surveillance Coordination Unit and program surveillance experts. It also provides a forum for managing issues.
The Public Health Agency’s 2010-11 Departmental Performance Report notes:
“The Agency’s 2010–11 mid-term Surveillance Governance Review confirmed that the existing governance structure is working, and the key stakeholders involved in the review indicated that the Agency’s surveillance activities have greatly improved. It also concluded, however, that the Agency should improve coherence in its surveillance activities.”Footnote 81
Although the governance structure in effect during the period covered by this evaluation is generally working well, areas for improving the following positions and structures have been identified and have been addressed in the Public Health Agency of Canada Surveillance Strategic Plan 2013-2016Footnote 82.
- Chief Health Surveillance Officer: The Surveillance Governance Review Final Report noted that strong leadership or a champion is key to the success of horizontal initiatives. It also noted that “Such roles can be difficult as the leadership must work with and influence colleagues who have linear accountability for their organizations and functions. In many cases, such roles can be seen to be interfering in colleagues’ activities, and that is exactly what must be done.”Footnote 83 The review also noted that to build momentum and strong leadership, the position of Chief Health Surveillance Officer should be full-time, as originally envisaged by the 2007 Surveillance Strategic PlanFootnote 84. Until recently, the position was part-time and was held by the Director General of the Office of Public Health Practice. Currently the Chief Health Surveillance Officer is a full-time position.
- Surveillance Coordination Unit: The mid-term governance review proposed an enhanced role for the Surveillance Coordination Unit, from its support and secretariat role, to a role that supports programs and coordinates horizontal initiatives.
- Surveillance Integration Team: The mid-term governance review suggested a stronger advisory role for the Surveillance Integration Team and greater involvement by members’ respective organizations to support the coordination of surveillance. The Surveillance Integration Team could “benefit from senior management support of its activities, including expecting members to facilitate two-way communication between members’ organizations and the horizontal governance structure.” Public Health Agency key informants indicated that while the Surveillance Integration Team was successful as a forum for exchanging information, it was less successful at influencing surveillance priorities and ensuring that surveillance initiatives are implemented across the Public Health Agency. Some key informants suggested that the Surveillance Integration Team should be more strategic, have a stronger role in advising senior management on new surveillance initiatives and facilitate the priority setting process.
- Senior Management Committee: The mid-term governance review noted the need for a greater senior management engagement in surveillance and stated that “senior management needs to put its stamp on surveillance coordination and maintain an active interest.”Footnote 85 Similarly, some key informants suggested that the Public Health Agency’s senior management team should pay more attention to surveillance and support the participation of their staff in surveillance governance committees and horizontal initiatives, and acknowledge these activities as part of their role.
4.2.2 Integrated surveillance
Finding 10. Integrated surveillance as envisaged by the 2007 Surveillance Strategic Plan has not yet been realized.
The 2007 Surveillance Strategic Plan goal for integrated surveillance is “to develop, implement, and maintain an efficient network of surveillance systems, accessible to Public Health Agency partners and stakeholders where applicable, which strategically links people, processes, and technologies that is purposeful and used for public health action.”Footnote 86
The document review indicated that there have been calls for a national integrated system for health surveillance in Canada since 1999. Such a system would address the need for horizontal integration and federal/provincial/territorial standardization of surveillance activities. Among the reports calling for an integrated national surveillance system are: Proposal to Develop a Network for Health Surveillance in Canada − National Health Surveillance Network Working Group and Integration Design Team (1999)Footnote 87; End-to-End Health Surveillance Project (2002)Footnote 88 and Learning from SARS, Renewal of Public Health in Canada–National − Advisory Committee on SARS and Public Health, (2003).Footnote 89.
Significant work has been done to promote a more integrated national approach to surveillance:
- The Public Health Agency is working with the provinces and territories towards greater standardization, for example, in developing data standards and common case definitions. It is also putting formal agreements in place for the collection, use and dissemination of public health information.
- At the federal, provincial and territorial levels, key informants noted the opportunity for national integrated surveillance using electronic health records technology and the transmission of clinical data to public health. Adoption is quite high in some settings and geographical regions and public health must be engaged as these clinical systems are established to ensure that surveillance data are captured and can be extracted.
- The Public Health Agency supports the development of public health information management solutions such as Panorama. The Panorama system is intended to provide the public health software necessary for the management of communicable disease cases, outbreaks, immunization programs, public health materials and vaccines, notifications regarding critical events and emergencies and public health worker tasks. Panorama is expected to improve the collection and analysis of health information and to assist in the coordination of responses to infectious disease outbreaks. Panorama is available to all Canadian jurisdictions for implementation with their own healthcare information systems infrastructures. Three provinces and territories have decided to implement Panorama, while others are either planning for future implementation of Panorama while seeking necessary funding commitments, have decided not to implement Panorama, or are considering other options. The Public Health Agency is considering how best to support all jurisdictions in the implementation of their respective public health information management systems.
The Public Health Agency has also achieved greater harmonization of its internal surveillance activities by:
- Promoting common approaches to surveillance through the development of the foundation policies, frameworks and tools that contribute to greater consistency across surveillance programs (e.g. the Data Quality Framework, the Integrated Framework for Surveillance, work on a privacy management framework, the Policy for the Collection, Use and Disclosure of Information Relating to Public Health and a web-based tool for completing Privacy Impact Assessments).
However, as noted in finding 8 of this evaluation, the Public Health Agency has been only partially successful in implementing these initiatives throughout the organization. Public Health Agency surveillance managers and survey respondents agreed that common approaches could lead to greater efficiency by reducing duplication. As well, common approaches would improve the compatibility of data among all levels, allow for better data management, greater collaboration and coordination and would provide better quality public health evidence in the long term.
- Providing opportunities for internal communication across the divisions and centres of the Public Health Agency.
As discussed in section 4.2.1, the mid-term governance review and key informant interviews indicated that the Surveillance Integration Team is generally viewed as an effective forum for information exchange and issue management. The Surveillance Integration Team, as well as other internal surveillance communications forums, such as the surveillance community of practice and the Subject Matter Expert Reference Group, provides opportunities for horizontal communication and information exchange that can contribute to improved integration.
- Reducing disease silos by integrating organizationally, as for example, within the Centre for Chronic Disease Prevention and the Centre for Communicable Disease and Infection Control.
For example, the Centre for Chronic Disease Prevention has expanded its Canadian National Diabetes Surveillance System beyond diabetes to include mental illness, hypertension, heart disease and stroke, neurological and other chronic diseases, and has renamed it the Canadian Chronic Disease Surveillance System.
A significant challenge to integration is the current organization of surveillance systems. Today most surveillance activities are supported and managed by a variety of vertical disease silos and multiple databases, leading to the current state of approximately 50 surveillance systems and data sets that operate independently and with disparate business processes. The document review and key informant interviews indicated that this vertical organization hinders the ability to fully assess the impact of diseases and conditions and to analyze emerging events and ongoing trends, and causes inefficiency in data collection, analysis and management.
Another challenge is that, according to the majority of surveillance managers and staff interviewed in the Public Health Agency, a common understanding of the term “integrated surveillance” does not exist. Interpretations ranged from coordination of and coherence in surveillance activities across the Public Health Agency to a single surveillance system. Even with this lack of common understanding of what an integrated surveillance system is, key informants supported greater integration, particularly if it represents a move to common approaches, techniques, standardization of data collection and definitions and interoperability of systems. Note that no one supported the integration of individual surveillance systems into a single surveillance system.
4.2.3 Knowledge management
Finding 11. The Public Health Agency of Canada does not consistently provide timely and useful surveillance products.
The 2007 Surveillance Strategic Plan goal for knowledge management is “to establish and maintain a constant transfer of information and knowledge for public health decision making.”Footnote 90
Timeliness of surveillance products
The literature emphasizes that surveillance is useful only to the extent that it is able to provide timely and accurate evidence that can guide interventions and support effective health practice, planning and evaluation. What constitutes timeliness depends on the nature and urgency of the disease and the needs of public health decision makers.
Issues about the timeliness of reporting of surveillance information were identified in internal program documents, audit and evaluation reports, and reports on notifiable diseases and chronic diseases. For example, the Hepatitis C in Canada: 2005-2010 Surveillance ReportFootnote 91 states that current surveillance methods do not allow for timely reporting of results. The 2008 May Report of the Auditor General of CanadaFootnote 92 concluded that the Public Health Agency’s “informal mechanisms for detecting and monitoring infectious diseases needs to be strengthened”, and that “this coupled with gaps and delays in the support of data by provinces and territories means that the Agency cannot always systematically analyse and report on public health threats”.
Similar issues about the timeliness of reporting of surveillance information were also identified in key informant interviews. Most provincial and territorial officials, as well as Public Health Agency managers and staff, indicated that the Public Health Agency does not provide its surveillance information and products in a timely enough manner to support policy and program decision making and public health action. The implications of delays in releasing surveillance information were highlighted by one senior manager who said that “PHAC is not adapting well to emerging concerns because data collection and reporting is not timely. Surveillance is intended to be data that you collect, analyze, interpret and disseminate to people that need to know in a timely way” and waiting two years to have an annual report published is not surveillance.
Despite a general sense of lack of timeliness of the Public Health Agency’s surveillance products, most provincial and territorial key informants did provide examples of timely surveillance information, including reports on enteric and respiratory diseases (e.g. PulseNet, FluWatch) tuberculosis, human immunodeficiency virus reports and the web-based Infobase Data Cubes for chronic disease.
The analysis of best practices of other countries and the European Union indicated that they provide regular national reports, usually weekly, but in some instances in real time. These reports include data on all notifiable diseases (typically 50-70 diseases) and may also include data on infectious agents. All jurisdictions (except for the European Centre for Disease Prevention and Control) have implemented some national-level collection, integration, analysis and reporting for selected chronic diseases. In particular, all have national cancer registries or databases as well as a number of other disease-specific registries or databases.
According to Public Health Agency surveillance managers and staff, the factors that affect the ability of the Public Health Agency to release timely surveillance information include the following:
- Timeliness of data collection: The document review and key informant interviews indicate that the inability to collect timely, accurate and complete information is a long-standing issue. The issue was raised in the 1999, 2002 and 2008 Auditor General reports. The May 2008 Report of the Auditor General of Canada on the surveillance of infectious diseases identified delays in the receipt of information from provinces and territories in relation to an outbreak situation: “timeliness in responding to an outbreak depends on the interval between the time a person is first seen by a health care provider and the time a report on the event is produced. The Public Health Agency can directly control only the time from when it receives information or specimens to when it produces its reports; the first steps are largely beyond its scope.”Footnote 93 Similarly, internal reports from the national notifiable disease program (2011), chronic diseases (2011) and hepatitis C (2008) also identified issues of timeliness of receipt of information from data sources as a factor that prevents the Public Health Agency from producing timely data analysis and dissemination. Many Public Health Agency key informants also identified issues with the quality of data received from the provinces, territories and other data sources. Some of the provincial and territorial key informants acknowledged that the provinces and territories do not always immediately submit their data, which in turn affects the Public Health Agency’s ability to produce timely reports.
- Capacity for analysis and interpretation: Public Health Agency surveillance managers identified issues of capacity related to analysis and interpretation. Some surveillance managers observed that skilled resources are being used for data collection. They felt that these skilled resources should be engaged in analysis and interpretation and that junior-level staff are capable of basic trend analysis.
The surveillance staff survey asked respondents to rank where most resources are allocated (human and financial) in their organization among the four activities of data collection, data integration, analysis and interpretation and reporting and dissemination. Of the survey respondents, 22 per cent indicated that resources were equally distributed among these four activities. However, 20 per cent of respondents identified data collection as the activity where most of their organization’s resources are allocated.
Similarly, an internal report from the chronic disease area noted that employees spend far too much of their time negotiating for access to the data.”Footnote 94
- Content approval processes: Many Public Health Agency key informants indicated that internal processes for approval of content in surveillance products can be lengthy. This, compounded by time-consuming communications processes as described below, has a significant impact on the timelines for the release of surveillance information to public health decision makers. Some key informants, including senior managers, attributed the lengthy content approval process to the requirement for the Public Health Agency to be sensitive to broader policy implications of surveillance information. One senior manager suggested that the Public Health Agency should be more strategic and distinguish between the information that requires additional scrutiny and the information that is part of the Public Health Agency’s routine business, and consider opportunities to streamline the approval process for routine reports.
- Communications processes for production and release of surveillance products: Public Health Agency surveillance managers indicated that to comply with legal and policy requirements, all surveillance products posted to the Public Health Agency’s website must be in HTML format. Because HTML conversion involves a lengthy and costly process, the Communications and Public Affairs Directorate has introduced new processes that require programs to provide evidence that posting the content to the Public Health Agency’s web site is an effective means of distributing information to audiences. Consequently, some reports that have traditionally been posted are no longer being posted routinely on the Public Health Agency’s web site. Surveillance managers and staff indicate that to expedite the release of their surveillance material, they are seeking alternative ways to get the information to their target audience, including posting summary reports and indicating that the full report is available upon request.
Alternative approaches for timely dissemination of surveillance products were identified in interviews with senior managers, surveillance managers and staff and included the following:
- consider alternatives to the traditional approaches to collecting surveillance information, including sentinel or syndromic surveillance and better use of technology (e.g. satellite imagery, GPS)
- present information in a consistent format which allows for more timely dissemination and also ensures that information is branded and recognized as a Public Health Agency product
- consider options for appropriate formats for the release of information (e.g. short summary reports)
- consider partnerships for the development of knowledge products.
Usefulness of surveillance products
The extent to which the Public Health Agency’s surveillance products meet the needs of partners and stakeholders is not routinely assessed, although the document review and key informant interviews highlight the need to undertake such an assessment. For instance, the Auditor General recommended that to “ensure that its surveillance systems for HIV, the West Nile virus and the influenza virus are best meeting the needs of the users, the Public Health Agency of Canada should systematically assess and document the user needs.”Footnote 95 While this was done for the areas in question, the evaluation did not find evidence that this is a consistent practice across surveillance programs.
Some changes within the Public Health Agency could improve the usefulness of surveillance products. For example, approximately one-quarter of Public Health Agency survey respondents suggested that changes are required in the Public Health Agency’s role in knowledge development, translation and exchange, and approximately two-thirds of respondents indicated that reporting and dissemination need strengthening. A report prepared for the Centre for Chronic Disease Prevention examined the format of traditional chronic disease surveillance reports, the rationale for the frequency of the major surveillance reports, the target audience and their information needs. The report concluded that the Centre’s “surveillance programs should pay greater attention to dissemination of their findings to policy-makers, researchers, health professionals and the general public, ensuring that their reports are presented in appropriate formats and venues.”Footnote 96 An evaluation report for the C-EnterNet programFootnote 97 indicated that the program could benefit from better marketing and should identify key decision makers that benefit from the information produced.
To ensure that surveillance information produced by the Public Health Agency meets the needs of its target audience, the Public Health Agency should consider collecting evidence about:
- the use of Public Health Agency products in policy and program decision making
- changes in public health practice as a result of surveillance information
- end-user satisfaction with surveillance products.
4.2.4 Partnerships and collaboration
Finding 12. The Public Health Agency of Canada collaborates extensively with external partners and stakeholders in surveillance activities.
The 2007 Surveillance Strategic Plan goal for partnerships and collaboration is “to effectively manage and maintain internal/external partnerships and engage them in collaborative surveillance activities that support public health action.”Footnote 98
The review of internal program documents identified the diversity of formal and informal relationships and collaborative networks across multiple disciplines. Survey results confirmed the breadth and diversity of these relationships and the majority (89 per cent) of survey respondents indicated that their program area is involved in activities related to stakeholder engagement and collaboration. Survey respondents indicated working relationships with:
- provinces and territories (92 per cent)
- other Government of Canada departments (79 per cent)
- academia (75 per cent)
- non-governmental organizations (61 per cent)
- the World Health Organization (52 per cent)
- national health surveillance networks (50 per cent)
- health care centres (hospitals, etc.) (49 per cent)
- other countries (42 per cent)
- global health surveillance networks (38 per cent)
- the private sector (12 per cent).
The informal relationships were identified as very important by many of the Public Health Agency key informants, with some surveillance programs owing their success to informal relationships. For example, long-standing collaborative relationships between the Public Health Agency and provincial and territorial staff were identified as a significant factor for the success of the tuberculosis surveillance program.
Funding recipients and international key informants spoke positively about their relationships with the Public Health Agency. For example, most funding recipients expressed satisfaction with their relationship with the Public Health Agency, particularly with regard to building capacity. Officials from the Pan-American Health Organization and the World Health Organization indicated that Canada is viewed as a leader and an effective partner in public health surveillance within the international arena. The Public Health Agency is perceived as adapting quickly to new disease threats, being transparent, sharing information with international counterparts quickly and for leading by example, thereby making other countries aware of the intent of the International Health RegulationsFootnote 99.
Greater coordination and clarity of roles is needed, particularly in relation to Statistics Canada. For example, one senior manager at the Public Health Agency said that “our relationship with Statistics Canada is a really important one to all of us...” and that “there is just a natural collaboration so that we are inputting the public health implications of Statistics Canada data, but not developing separate reports on virtually the same kinds of subject areas”.
In addition to the Public Health Agency, several other federal departments and agencies have a role with respect to reporting on issues that affect the health of Canadians, including Statistics Canada, the Canadian Food Inspection Agency and Health Canada. As well, the Canadian Institute for Health Information, an independent, not-for-profit organization, produces health information with the objective of contributing to the improvement of the health of Canadians and the health care system. Some provincial and territorial key informants expressed confusion about the roles of the various organizations. For example, one key informant spoke of interacting with organizations including the Public Health Agency, Health Canada and Statistics Canada, and felt that “just trying to understand the alignment amongst and within that group is a mystery.” Some key informants expressed the need for a surveillance strategy among these organizations to clarify roles to prevent duplication of surveillance activities.
4.2.5 Performance measurement and quality assurance
Finding 13. The Public Health Agency of Canada has made some progress in developing an overarching performance measurement strategy for surveillance, but has not yet fully implemented it.
The 2007 Surveillance Strategic Plan goal for performance measurement and quality assurance is “to develop, implement, and maintain performance measurement, quality assurance and evaluation activities that are timely, relevant and used to improve surveillance within the Public Health Agency and ensure that it is constantly aligned with the Public Health Agency priorities”.Footnote 100
The document review indicated that the Public Health Agency’s Performance Measurement Framework has Public Health Agency-level performance indicators to meet annual reporting requirements. However, as public health surveillance has been a Program Sub-Activity within the Program Alignment Architecture only since the 2010-11 reporting period, the function is relatively new in terms of its associated performance measurement lifecycle.
At the program level, performance measurement strategies are not consistently in place for the ongoing measurement of the performance of surveillance systems. Only 25 per cent of Public Health Agency staff surveyed indicated that a performance measurement strategy was in place for their surveillance system, and more than one-third (38 per cent) of respondents did not know if one existed. Approximately 39 per cent of survey respondents indicated that performance data was being collected, although a performance measurement strategy did not exist for their system. Despite this, of the survey respondents who indicated that performance information is being collected, 57 per cent reported that performance information was being used to improve procedures, plan projects and programs and contribute to the Public Health Agency’s Performance Measurement Framework. Of note is that 36 per cent of survey respondents identified performance monitoring as a gap or an area that should be strengthened.
Generally, Public Health Agency programs routinely collect some data to measure the performance of their surveillance systems and programs and to guide ongoing improvement and evaluation. However, there is no consistency across programs as to the information being collected. Public Health Agency managers and staff generally reported that they are monitoring only program outputs (i.e. the number of requests for surveillance products, the number of presentations, publications, analyses, etc.). Some reported that they have attempted to use surveys to collect outcome data, such as user satisfaction with their surveillance products, but that the response rate has been low.
Key informant interviews indicated that discussions are taking place within the Public Health Agency about the development of a performance measurement framework that will provide a core set of performance indicators for individual surveillance systems. These performance indicators would help programs to determine if their systems are operating effectively, meeting the needs of stakeholders and providing quality data for public health decision making. This data will then be rolled up for an assessment of performance at the surveillance function level. A draft performance measurement framework was developed in 2010, but it was decided by the Surveillance Integration Team that more work was required to streamline the framework in order to have only a small number of solid indicators that would have relevance across all systems.Footnote 101 This work has led to a revised performance measurement framework and accompanying tools which were piloted in 2012. Based on the results of these pilots, the framework is being revised. The revised framework and an implementation plan are scheduled for presentation to the Surveillance Integration Team early in 2013. The performance measurement framework will then be phased in for the remaining surveillance systems.Footnote 102
The Public Health Agency has developed and implemented a data quality framework that standardizes the assessment and documentation of surveillance data strengths and limitations. The framework is being implemented in three phases: phases I and II have been completed and phase III is expected to be completed in 2012-13. Once implementation is complete, every surveillance program in the Public Health Agency is expected to use the framework to perform regular assessments of the quality of their databases.
The February 2012 status report on implementation of recommendations of the May 2008 Report of the Auditor General of CanadaFootnote 103 indicates that the Public Health Agency has made substantial progress in implementing the data quality framework. It also indicates that issues of deficiencies in data received from partners are being addressed through existing mechanisms, including the establishment of data standards, the Multi-lateral Information Sharing Agreement, as well as the Public Health Agency’s participation in the Canada Health Infoway Standards Collaborative. Program areas also work with their partners and stakeholders to address apparent data quality deficiencies by implementing recommendations from their data quality assessments. The results from the second cycle of quality assessment are not yet available, and therefore the evaluation was unable to ascertain whether program areas had addressed data quality issues identified in the first round of data quality assessments.
Feedback on the usefulness of the data quality framework was mixed. Some key informants indicated that assessments do help programs to identify areas for improvement, however, a few key informants expressed concern that the data quality framework is complex and “really difficult to use across the whole range of systems and users”.
4.3 Efficiency and economy
Finding 14. Opportunities for greater efficiency and economy exist.
Even though surveillance activities are a discrete Sub-Activity of the Program Alignment Architecture (2011-12), in practice they are decentralized and controlled at an operational level. This results in different methods used by various programs and centres in tracking and reporting of surveillance activities and their associated expenditures. Therefore, the evaluation team could not conclude that the Public Health Agency has minimized resources while maximizing outcomes with respect to surveillance information for decision making and public health action.
Based on the document review and an analysis of staff interviews, the following three areas of opportunity to improve the efficiency and economy of the surveillance function within the Public Health Agency were identified:
- standardization, coordination and integration
- data management
- setting surveillance and surveillance publication priorities.
4.3.1 Standardization, coordination and integration
Greater efficiency could be achieved by strengthening the infrastructure to better support surveillance activities. As discussed in section 4.2.2, since 2007, through work undertaken by the Surveillance Coordination Unit and the Surveillance Integration Team, the Public Health Agency has introduced policies, frameworks, business processes, procedures and tools that better support standardized Public Health Agency-level and national approaches to public health surveillance. The Public Health Agency should continue its efforts to standardize and coordinate surveillance activities, as a consistent and agreed-to approach provides opportunities to streamline the Public Health Agency’s surveillance activities, supports efforts to develop a national approach to surveillance and reduces duplication of effort and inefficiencies.
Efficiencies could also be gained by examining the Public Health Agency’s surveillance systems and data holdings, as well as related activities, to identify opportunities to reorient programs, coordinate resources and to increase interoperability between surveillance platforms. For example, the Centre for Chronic Disease Prevention and the Centre for Communicable Diseases and Infection Control have integrated surveillance within their organizations for efficiency.
4.3.2 Data management
Implementing a data management framework would create additional efficiencies. Records of discussion from meetings of the Surveillance Integration Team highlight issues regarding duplication of data requests, redundancies throughout the organization, high variability in the format and content of data and a lack of awareness across the organization of data holdings and data for which the content and meaning are not known. Treating surveillance data as a corporate asset and creating a framework that addresses all aspects of managing surveillance information assets, including acquiring, creating, housing, delivering, maintaining and retiring data, will enable standardization and coherence in how the Public Health Agency treats its data holdings. Reducing duplication would lower the cost of data, improve the management and control of increasing amounts of data and ensure that the Public Health Agency leverages data from existing investments.
4.3.3 Surveillance priorities
Developing and strengthening disease surveillance and response at the national level requires a substantial and long-term commitment of human, financial and material resources. As discussed in section 4.1, this investment warrants a formal corporate-level process to periodically review surveillance investments to ensure that surveillance activities continue to reflect national disease control priorities, that activities are still relevant and needed, and that there continues to be a federal role for these activities. Without a formal priority setting process, the Public Health Agency risks continued investment in surveillance systems that no longer address needs.
4.3.4 Surveillance publication priorities
Surveillance information is the main output of public health surveillance activities, and this information influences how public health activities are undertaken both in Canada and internationally. As discussed in section 4.2.3, the Public Health Agency publishes a wide range of surveillance information of varying types and timelines, at significant cost. It is unclear to what extent these publications target the right audience, meet user needs or are useful for public health decision making. This, coupled with the increased costs associated with legal and policy requirements for web posting, suggests that the Public Health Agency should examine and prioritize its surveillance products. The examination should clarify the target audience for various surveillance products and identify their needs, identify publications that no longer align with the Public Health Agency’s surveillance priorities and identify partnerships for publications. Efficiencies can be achieved by eliminating publications that no longer meet users’ needs or that no longer align with federal priorities.
Report a problem or mistake on this page
- Date modified: