ARCHIVED - Executive Summary
The 2005 federal budget allocated $300 million over five years and $74.4 million per year in ongoing funding to the Public Health Agency of Canada (PHAC) for the Integrated Strategy of Healthy Living and Chronic Disease (ISHLCD, or “the Strategy”). This funding ensured Canada had an integrated approach to addressing major chronic diseases through addressing risk factors, as well as through complementary disease-specific work.
As part of this comprehensive initiative, there are six Functional Components, including the Surveillance Functional Component. This formative evaluation focuses on the Surveillance Functional Component of the PHAC’s ISHLCD.
The Surveillance Functional Component consists of three action areas that guide activities and overall objectives: 1) Enhance the analysis, interpretation and use of existing data sources for surveillance; 2) Expand data sources for surveillance; and 3) Coordinate, plan and evaluate surveillance activities. The budget for the Surveillance Functional Component is $36.5 million over five years with $10.5 million ongoing.
The purpose of this evaluation is to support the Results-based Management and Accountability Framework (RMAF) requirement to conduct an evaluation of the ISHLCD Functional Components. The overall intent of this formative evaluation is to guide and inform stakeholders regarding “how far and how fast” program implementation of the Surveillance Functional Component has proceeded, and what early outcomes have been generated for the action areas.
Given the emphasis of the implementation of the ISHLCD since its inception in 2007 has been on enhancing external engagement and the ISHLCD Operational Matrix, this evaluation examined the concepts of engagement and integration. The following questions related to program relevance, design, delivery and success were addressed:
- Does the Surveillance Functional Component continue to be consistent with departmental and government-wide priorities, and does it realistically address an actual need?
- Are the most appropriate means being used to achieve objectives?
- Is the Surveillance Functional Component effective in generating outputs, given its resources, and without unintended consequences?
As an exploration of implementation, including the principle program concepts of engagement and integration, it was determined that a qualitative approach with key informants, balanced by a review of key documents, would best inform this formative evaluation. Thirty-five (35) internal and external key informants were interviewed. Selection criteria for interviewees included ensuring sufficient knowledge about the ISHLCD, as well as balance across chronic disease areas. Key documents, identified through a validation of key internal/external documents with stakeholders, were also reviewed as part of the evaluation.
As a qualitative assessment of program performance, data validity and reliability concerns were at least partially controlled for through:
- interviewing both internal and external respondents;
- triangulating data via document reviews as well as key informant interviews;
- developing consistent practices for interviewing and transcription; and
- using a qualitative analysis software package to help establish a coding audit trail to support potential replication of the findings.
For the Surveillance Functional Component, the evaluation questions address:
- relevance of the program when compared with need and government priorities;
- appropriateness of the program as a means toward achieving its objectives; and
- the program’s effectiveness in generating outputs, without unintended consequences, given the resources provided.
The Surveillance Functional Component has demonstrated a high level of consistency with federal requirements and joint Federal/Provincial/Territorial (F/P/T) priorities, directives and plans for enhancing chronic disease surveillance capacity. The findings from this evaluation illustrate PHAC has shown sensitivity and respect for the roles of other public health organizations in an evolving national policy context, where changes in government and the creation of a disease-specific agency outside of government presented challenges in implementing the original vision of the ISHLCD.
The Surveillance Division and their partners have faced the challenge of increases in simultaneous expectations for greater breadth and depth of surveillance information. The establishment of disease-specific initiatives within the broader context of healthy living and chronic disease prevention necessitates continued leadership by PHAC to monitor a growing number of partnerships and surveillance information requirements.
Design and Delivery
While the implementation of the Surveillance Functional Component has been delayed compared to the original schedule established with Treasury Board, this was largely due to internal and external constraints beyond the immediate control of program management. The fact that the Surveillance Functional Component may be able to accelerate program implementation over the next several fiscal years to be on schedule will represent a significant program accomplishment.
The systematic approach of documenting and tracking used to support the management of the Gs&Cs process may provide a model for determining and coordinating the management of the other outputs of the Surveillance Functional Component. Outside of the Gs&Cs program, most outputs are not being measured systematically (e.g., only half of those who respond to requests for information are contributing to the monitoring of these requests). The continued development and implementation of the surveillance indicator framework as a surveillance planning and reporting tool should be considered as a significant resource in the planning of a performance measurement system.
Progress towards accomplishing the outputs described in the Surveillance Logic Model has been reported via a number of mechanisms. In particular, a number of comprehensive chronic disease-specific epidemiologic reports have been generated as part of the Strategy. Similarly, activities are underway to improve analytic capacities for existing chronic disease data, although delays in implementation have had an impact on the realization of some outputs.
Among the lessons learned during the implementation of the Surveillance Functional Component is that enhanced communication with stakeholders to help manage their expectations for surveillance data, and to identify and communicate the many steps required to coordinate and deliver enhanced surveillance capacity, would clearly benefit the Strategy.
One challenge in measuring progress of the Surveillance Functional Component has been the issue of conceptually defining and measuring engagement. Given the need to develop and pilot a qualitative measure of engagement to support the assessment of the sole immediate outcome, this evaluation’s assessment of engagement is preliminary and intended to be supportive of future attempts to assess this concept. The results of this evaluation indicate engagement with stakeholders by the Surveillance Functional Component has been successful in increasing trust and the sharing of information.
While outside the scope of this evaluation, the fact that the operationalization of all ISHLCD Functional Components is based on the application of matrix management concepts (i.e. the ISHLCD Operational Matrix) necessitates that any future evaluation of ISHLCD should identify well known critical successes, limiting factors and any standard measures of matrix management functions. Respondents reported challenges to implementation of this matrix approach, identifying that the work has been “siloed”, “stove-piped”, or there was a poor connection between the ISHLCD matrix approach and the organizational hierarchy.
Among the intermediate outcomes described in the Logic Model, respondents identified progress in the areas of data quality and providing chronic disease surveillance information, as well as trends in chronic diseases, risk factors and disease outcomes. Internal respondents reported more positive perceptions than external respondents. This finding further identifies the need to communicate advances in chronic disease surveillance capacity with a broader stakeholder audience.
Finally, assessing longer-term outcomes, such as the impact of surveillance on decision-making, can be regarded as premature given the short period of time the Surveillance Functional Component has had to implement its work. Future work would also need to be informed by best practices identified by the Knowledge Development, Exchange & Dissemination (KDED) Component of the ISHLCD.
The recommendations from the Surveillance Functional Component are focused on building on the success of existing surveillance initiatives and ongoing collaboration with provincial/territorial (P/T) partners in three areas: programs, evaluation and overall ISHLCD.
- Increase stakeholder awareness of chronic disease surveillance and the role and priorities of PHAC in surveillance. Use the launch of surveillance products and services as opportunities to communicate.
- Work with internal staff and relevant stakeholders to clarify and communicate roles and responsibilities in the area of cancer surveillance, in particular between PHAC and the Canadian Partnership Against Cancer (CPAC).
- Improve communication with external stakeholders (including P/T governments, nongovernmental organizations (NGOs), potential partners and funding recipients) concerning progress made on the achievement of outputs related to the Surveillance Component. This will help address the current lack of awareness with regards to Component priorities and progress.
2. Adherence to Realignment Initiative on Gs&Cs
- While Gs&Cs management appears to be consistent with standard operating procedures and may excel in certain areas (such as the use of a reviewer comments matrix and a single Web page to provide applicant information and disseminate reports), there will be the need to ensure the Gs&Cs management adheres to the forthcoming findings and recommendations from the Realignment Initiative on Gs&Cs.
3. Knowledge Development and Exchange (KD&E)
- Define knowledge development and exchange within surveillance. Establish linkages across Functional Components and with corporate supports and resources for PHAC-wide KD&E activities.
4. Performance Monitoring System Development
- Establish a performance monitoring system, based on the PHAC Evaluation Framework for Surveillance Systems and the ISHLCD KD&E performance measurement framework, to establish a performance monitoring system to track activities, outputs and immediate outcomes for the Surveillance Functional Component.
5. Outcome Tool Development
- In order to gather data further down the results-chain of the Logic Model, the development of approaches and instruments to measure intermediate outcomes of the ISHLCD Surveillance Functional Component are required to facilitate decision-making and support the measurement requirements of future evaluation designs.
6. Evaluation Working Group
- Develop an evaluation working group, with representation from evaluation and program staff, to inform the development of the performance monitoring system and evaluation tools, as well as ensure system utility. Committee representation from PHAC regional offices will be sought as appropriate.
7. Engagement Tool Development
- Given the key importance of the concept of engagement for the overall Strategy (as a common immediate outcome for all ISHLCD Functional Components), it will be essential for the monitoring and evaluation of the Strategy to develop an approach and tools to systematically assess the immediate outcome of engagement across all Functional Components.
8. Strategy Performance Measurement
- There is a need for corporate level support to establish a consistent performance measurement framework and monitoring system across the Functional Components of the ISHLCD. This will allow the contributions of the Functional Components and the Strategy as a whole to be monitored.
9. Matrix Functioning
- Identify, assess and communicate internal and external ISHLCD coordination mechanisms reflecting best practices for matrix management and an integrated approach. For example, KD&E is critical for all Components, including the Surveillance Functional Component, and linkages should be strengthened between the Components and KD&E work within the Strategy.
Report a problem or mistake on this page
- Date modified: