Language selection

Government of Canada / Gouvernement du Canada

Search

Evaluation of the Canadian Public Health Service Program at the Public Health Agency of Canada

Prepared by
Evaluation Directorate
Public Health Agency of Canada and Health Canada

July 2013

Table of contents

List of Tables

Executive summary

This report presents the findings of an evaluation of the Canadian Public Health Service program at the Public Health Agency of Canada.

Evaluation purpose

The evaluation addresses requirements in the Treasury Board of Canada’s Policy on Evaluation (2009) for all programs to be evaluated on a five-year cycle, and supports program planning and decision making. This is the first evaluation of the Canadian Public Health Service program.

The evaluation examines the relevance of the program, including the continued need for the program and its alignment with federal government and Public Health Agency roles and priorities. It also addresses the performance of the Public Health Agency in achieving expected program outcomes and demonstrating efficiency and economy. The time period examined is between April 2006 and October 2012.

The scope of this evaluation does not include activities related to the Public Health Agency’s other field services programs, including the Canadian Field Epidemiology Program, the Laboratory Liaison Technical Officer program, the Office of Quarantine Services, and the HIV Field Surveillance program.

Description of Canadian Public Health Service program

The Public Health Agency developed the Canadian Public Health Service program in 2006 to help address Canada’s public health capacity needs. The Canadian Public Health Service program temporarily places trained federal public health employees (Public Health Officers) in public health organizations (placement sites) across Canada to respond to short-term routine and emerging public health needs at these sites. Placement sites may include provincial, territorial and local health authorities as well as public health-oriented non-governmental organizations. These placements are expected to contribute towards two intermediate outcomes: Public Health Officer placements increase the public health capacity of the site, and serve as an opportunity for career development of the Public Health Officers. As a result of their involvement at these placement sites, it is expected that Public Health Officers will gain valuable experience in a range of public health issues and settings, helping to equip them for potential mobilizations (federal response in the event of a pandemic or a health emergency) should the need arise. A sub-component of the program places graduate students into public health organizations in Northern communities to facilitate the completion of their research practicums, and to help contribute to the host organizations’ scoped capacity needs.

This evaluation examines both components of the Canadian Public Health Service program. The Agency allocated approximately $3.4 million for this program in 2012-13. Since program inception, the Public Health Agency has spent $13.2 million for the Canadian Public Health Service program. Currently the program employs 14 active Public Health Officers.

Evaluation findings, conclusions and recommendations

The synthesis and analysis of the information gathered for this evaluation resulted in findings and conclusions about both relevance and performance, which in turn led to three recommendations.

Summary of findings and conclusions

Relevance

The need to address Canada’s public health capacity still exists. Although empirical data is limited, there are recognized public health capacity gaps in a number of regions (northern territories, Atlantic provinces, rural/remote areas) and professions (epidemiologists, nurses, and policy analysts) that the Canadian Public Health Service is attempting to address.

Building Canada’s capacity in public health is an appropriate role for both the federal government and the Public Health Agency of Canada. In Canada, public health is a shared responsibility between federal, provincial and territorial and municipal governments, the private sector, non-government organizations, health professionals and the public. While provinces and territories have the primary responsibility for their respective human resource and capacity needs, the consequences of their capacity gaps can affect the public health system as a whole. Response to public health events may be national in scope and beyond the capacity of any particular province or territory to address by itself. As such, the Government of Canada may choose to play a supporting role in this situation. Subject to governmental policy considerations, the spending powers may be used to create non-regulatory programs in areas of provincial jurisdiction.

The mandate of the Public Health Agency of Canada is to support the Minister of Health in carrying out her powers, duties and functions in relation to public health. The Public Health Agency’s role in addressing Canada’s public health capacity needs generally aligns with and supports the Agency’s mission “to promote and protect the health of all Canadians, through leadership, partnership, innovation and action on public health”. In addition, the Public Health Agency’s role includes: intergovernmental collaboration on public health; the facilitation of national approaches to public health; emergency preparedness and response; and the prevention and control of chronic and infectious diseases.

The federal role is further defined in program authorities received to address Avian and Pandemic Influenza Preparedness. The Public Health Agency received funding to address public health capacity through activities to:

  • build a skilled and responsive federal public health workforce across Canada
  • provide the critical public health capacity required to prepare for and respond during a public health emergency.

The Canadian Public Health Service program does not appear to duplicate the role of other stakeholders. However, the program has the potential to overlap with the Canadian Field Epidemiology Program, in particular if the focus on training and development of Public Health Officers is prioritized over building site capacity.

Strengthening Canada’s public health capacity is a priority for the Government of Canada and the Public Health Agency of Canada and is encompassed within three of the Public Health Agency’s strategic directions for the next five years, as per corporate strategic documents: leadership on health promotion, disease prevention, and health protection; strengthened public health capacity and science leadership; and enhanced emergency preparedness and response.

Performance

The Canadian Public Health Service program has made progress towards its stated intermediate outcome of helping to address public health gaps at participating placement sites. Through Public Health Officer placements, sites gained an improved ability to address ongoing core public health work, an improved ability to undertake self-identified priority projects that they were previously unable to pursue, and an improved response to public health events. Similarly, student projects supported organizational capacity gaps by allowing sites to undertake discrete projects that may not otherwise have been accomplished in the desired timeframe.

Evaluative evidence also indicates that the Canadian Public Health Service program has made progress towards achieving its stated intermediate outcome of enhancing capacity of partner organizations to fulfill their public health needs. Site supervisors reported that Public Health Officers have contributed to their sites through the provision of technical skills such as epidemiological surveillance and analysis, the development of high-quality reports, the facilitation of networking, as well as the delivery of training to site staff. Public Health Officers have also facilitated knowledge transfer through their sharing of lessons learned from previous site placements.

Although the sites have benefitted from the placement of a Public Health Officer in their organization, it appears that this capacity might potentially be lost, in some instances, once the Public Health Officer has been rotated to another site unless deliberate efforts are made during their tenure to build capacity in more lasting ways.

Progress has also been made towards achieving the intermediate outcome related to the development of a cadre of Public Health Officers, available to function in a range of public health issues, including mobilizations. This in turn, supports the intentions laid out in the program authorities which described building a skilled and responsive federal public health workforce across Canada able to provide the critical public health capacity required to prepare for and respond during a public health emergency. Public Health Officers have worked on projects covering a spectrum of public health topics, have received group training to prepare for potential mobilizations, and in some cases, have had the opportunity to mobilize.

Similarly, the program has also made progress towards achieving its intermediate outcomes related to the development of Public Health Officers. While the program has not yet established official core competency training or related performance measurement materials, current and former Public Health Officers indicated that they have acquired transferable knowledge and skills as a result of their site placements. Furthermore, students have fulfilled educational requirements for graduate degrees in public health, and have gained work experience in public health in the North.

While the program has made progress towards the achievement of the intermediate outcomes currently stated in its draft logic model, the evaluation identified that significant challenges remain with respect to the program’s overarching design and delivery. The program’s intended results and associated program theory are unclear to many program stakeholders. As a result, the site selection and matching process appear to have resulted in placement sites being selected that do not represent those within the country with the greatest public health capacity needs, as indicated by senior provincial/territorial representatives during data collection. Additionally, the lack of flexibility in the duration of Public Health Officer placements and the requirement to rotate has negatively impacted program outcomes for placement sites and Public Health Officers. The requirement to rotate out of province and the challenges associated with this have also led to decreased Public Health Officer retention.

Finally, additional challenges remain, including the missed opportunity for substantial collaboration and knowledge exchange between the Canadian Public Health Service program staff and Public Health Agency staff working in similar content areas. This collaboration could lead to leveraging of shared knowledge and partnerships among program areas, resulting in a more efficient and effective delivery of program activities. Lastly, there lack of a performance measurement strategy to help the Office of Public Health Practice monitor progress and assist in programmatic decision making, and therefore performance information could have been better used to assess and adjust performance for the program as a whole.

Recommendations

Intended Outcomes

The Public Health Agency aims to enhance public health capacity in Canada through a number of field service programs. Each of these programs targets a specific component or element of the public health system and employs its own unique approach to building capacity. While work is currently underway within the Agency to define and describe what is meant by public health workforce and capacity, at the time of data collection, there did not appear to be a commonly accepted Agency-level understanding of this terminology. Therefore, the precise means by which the Canadian Public Health Service program compliments the Agency’s other efforts in this area is not clearly articulated.

Evaluation evidence indicates that the Canadian Public Health Service program objectives are not clear or well understood by all program stakeholders. This may be the result of a number of factors. Firstly, the authorities for this program covered Avian and Pandemic activities more broadly, and therefore limited details were included with respect to the specific intent and rationale for the Canadian Public Health Service. Secondly, there is a limited availability of detailed program documents (strategic and descriptive). Thirdly, the program does not have a management approved logic model summarizing the program theory. A logic model was provided for the purpose of the evaluation; however, it is incomplete as the two ultimate outcomes for this program were still under development at the time this evaluation was being conducted. Further, the logic model identifies general program outcomes (immediate, intermediate and ultimate) but does not clearly identify program inputs, activities or outputs. Additionally, the draft logic model does not contain clearly defined outcomes. For example, the ultimate outcome of “strengthening public health capacity” is interpreted differently by various program stakeholders.

This lack of clarity in the program theory has negatively impacted the delivery of the program and thus its ability to achieve the most appropriate results for the Agency within the context of the other field service programs.

Recommendation 1

Within the context of the Agency’s strategic directions and efforts towards building public health capacity, determine, articulate and communicate desired program results.

Design and Delivery

The current lack of clarity associated with the program’s intended outcomes has resulted in a program delivery model that lacks a clear, consistent and commonly understood strategic approach. This is best demonstrated by the site selection criteria that are used to assess placement site applications. These site selection criteria are not well defined and have undergone multiple iterations since program inception. In addition, they appear to be paradoxical, as the current program delivery model may not support the achievement of both intermediate outcomes. The site selection criteria also do not appear to reflect or support all the activities outlined in the original program mandate (e.g. surge capacity), and may also be missing links to current program priorities.

The site selection criteria have also partly contributed to sites being selected that do not appear to represent those within the country with the greatest perceived public health capacity needs. While Regional Coordinators have liaised with provincial/territorial and local stakeholders to identify their capacity needs/priorities via the solicitation of proposals, feedback from senior provincial/territorial representatives suggests that the program is still not reaching the provincial/territorial areas with the greatest capacity needs. The current placement of Public Health Officers has reflected the profession-related public health capacity needs (epidemiology, public health nurse, policy analyst); however, they do not reflect recognized regional disparities (Northern communities, Atlantic provinces, rural/remote areas). For example, the northern (all territories combined) and eastern region (all Atlantic provinces combined) of Canada, which both represent areas with recognized capacity needs, combined have only been the recipient of approximately 43% of the Public Health Officers placements. To date, Public Health Officer placements have not occurred in the province of Quebec due to reasons related to Quebec legislation. This means that the majority (57%) of Public Health Officer placements have occurred in the remaining five provinces (Ontario, British Colombia, Alberta, Saskatchewan and Manitoba), which according to data collected for this evaluation are not the areas with the greatest capacity needs. Furthermore, the majority of Public Health Officer placements to date have taken place in larger urban centres, as opposed to the priority rural and remote areas identified by key informants.

Similarly, the current duration of Public Health Officer placements and the requirement to rotate may not support the achievement of desired program outcomes. For example, the requirement to rotate Public Health Officers at the two year mark has resulted in incomplete project work at some placement sites, leaving these sites with ongoing capacity needs. Likewise, the requirement to rotate at the two year mark has also negatively impacted learning and development outcomes for Public Health Officers and slowed the formation of a sustainable cadre of Public Health Officers due to decreased Public Health Officer retention. Although there have been occasions where Public Health Officers have left the program to pursue academic or employment opportunities, the most common reason given (approximately 80% of former Public Health Officers) for their exiting the program is due to their displeasure with the requirements associated with rotation.

By articulating a clear strategic focus and developing accompanying operational processes, the program will be positioned to align its resources to more effectively achieve priority outcomes. This may partly be a result of the requirement for placement sites to have strong mentoring capacity to also facilitate Public Health Officer professional development.

Recommendation 2

Review and revise current operational practices, including approaches to site selection, rotation, and placement duration, to support achievement of desired results.

Performance Measurement

The Canadian Public Health Service program does not have a formal performance measurement strategy in place to monitor progress and assist in programmatic decision making. This is exacerbated by the lack of a detailed, management approved, logic model and associated narrative clearly describing program theory.

Limited performance data has been collected about program outcomes, and the data collected about activities and outputs appears to be inconsistent and unsystematic. Gaps in the performance data present challenges when attempting to measure and assess the program’s progress towards the achievement of its intended outcomes for Public Health Officers, students and placements sites. There is also no evidence that the limited performance information collected was ever used to inform program or policy decisions regarding the Canadian Public Health Service program.

There are promising activities currently in the works which hint at the development of a more comprehensive performance measurement culture within the Canadian Public Health Service program. For example, the program is working to develop a more substantive logic model and narrative which would clearly articulate the proposed theory of change. While this is a positive indication of a more effective performance measurement function, future work must be comprehensive in its approach (documenting outputs and outcomes of both Public Health Officer and student placements), grounded in a clear theory of change, systematically integrated into operations, and then actively used in management practices to assess and adjust performance.

Collaboration

Since the program’s inception in 2006, the Canadian Public Health Service program staff has initiated dialogue to share information and collaborate on related work with a number of other areas within the Office of Public Health Practice and other Public Health Agency colleagues.

There has been significant collaboration with the regional officers, including partnering to deliver a variety of training opportunities to Public Health Officers and other Public Health Agency staff working on related issues. The working relationship between Canadian Public Health Service staff in the National Capital Region office and the various Public Health Agency regional offices has been key to the effectiveness of the delivery of the program, and ongoing liaison between these offices has been a critical element in the design and delivery of this program.

Collaboration with other areas of the Agency has been minimal. For example, there is limited collaboration and knowledge exchange between Public Health Officers / students and Public Health Agency staff working in similar content areas, and Public Health Officers do not have a good understanding of opportunities to link with specific Public Health Agency issue experts to support them in their placement activities.

The impact of this gap on the Public Health Agency’s ability to achieve the objectives of the Canadian Public Health Service program in the most efficient manner possible is not known. It is anticipated that greater integration and collaboration would support greater leveraging of knowledge, and enhance the effectiveness of the program.

Recommendation 3

Maximize opportunities for enhanced program performance by:

  • a)Strengthening the performance measurement approach for the program to better inform programmatic
           decision-making.
  • b)Facilitating collaboration and knowledge exchange activities across the Public Health Agency of Canada.

Page details

Date modified: