Evaluation of the Public Health Agency of Canada's Public Health Workforce Development Activities 2010-2011 to 2014-2015

Prepared by
Office of Audit and Evaluation
Health Canada and the Public Health Agency of Canada

July 2016

List of Acronyms

AERO

All Events Response Operations

APHC

Applied Public Health Chairs

CBMI

Competency Based Management Initiative

CFEP

Canadian Field Epidemiology Program

CIHR

Canadian Institutes of Health Research

CPHI

Centre for Public Health Infrastructure

CPHS

Canadian Public Health Service

F/P/T

Federal/Provincial/Territorial

FSTR

Field Services Training and Response

HSIB

Health Security Infrastructure Branch

IHR

International Health Regulations

MPH

Masters of Public Health

PAA

Program Alignment Architecture

PHAC

Public Health Agency of Canada

PHO

Public Health Officer

PHSCBI

Public Health Scholarships and Capacity Building Initiative

PHWD

Public Health Workforce Development

P/T

Provincial/Territorial

SEPH

Skills Enhancement for Public Health

Table of Contents

• Executive Summary
• Management Response and Action Plan
• 1.0 Evaluation Purpose
• 2.0 Program Description
  • 2.1 Program Context
  • 2.2 Program Profile
  • 2.3 Program Narrative
  • 2.4 Program Alignment and Resources
• 3.0 Evaluation Description
  • 3.1 Evaluation Scope, Approach and Design
  • 3.2 Limitations and Mitigation Strategies
• 4.0 Findings
  • 4.1 Relevance: Issue #1 - Continued Need for the Program
  • 4.2 Relevance: Issue #2 - Alignment with Government Priorities
  • 4.3 Relevance: Issue #3 - Alignment with Federal Roles and Responsibilities
  • 4.4 Performance: Issue #4 - Achievement of Expected Outcomes (Effectiveness)
  • 4.5 Demonstration of Economy and Efficiency
• 5.0 Conclusions
  • 5.1 Relevance Conclusions
  • 5.2 Performance Conclusions
• 6.0 Recommendations
• Appendix 1 - Professional Experience Guidelines for Field Epidemiologists
• Appendix 2 - Logic Model
• Appendix 3 – Detailed Financial Breakdown by Program Area
• Appendix 4 – Evaluation Description
• Appendix 5 – Intensity and Reach of PHWD Activities
• Appendix 6 – Summary of PHWD Activities (inputs and outputs)

List of Tables

• Table 1: Planned Budget for all Program Components ($)
• Table 2: Limitations and Mitigation Strategies
• Table 3: Registrations in Self-Directed Skills Online Courses by Province / Territory
• Table 4: Number of Participants by Province Completing Facilitated Skills Online Courses: Top Five Provinces per Year, 2011-2014
• Table 5: Variance: Planned Spending vs Expenditures 2010-2011 and 2014-2015 ($)

Executive Summary

Evaluation Purpose and Scope

The purpose of the evaluation was to assess the relevance and performance of the Public Health Agency of Canada’s (PHAC) Public Health Workforce Development (PHWD) Activities. The scope of the evaluation covered the period from April 2010 to December 2015 and included most of the activities identified under Sub-Program 1.1.1 of the Program Alignment Architecture: Public Health Workforce, as well as the internal-facing Competency Based Management Initiative.

A number of other program and activity areas in PHAC also contribute to the development of the public health workforce, however these were out of scope for this evaluation, specifically: the Public Health Law and Ethics Program (which ended in 2012), the National Collaborating Centres for Public Health (recently evaluated in 2014-2015), other PHAC training efforts, the emergency response work of the Centre for Public Health Infrastructure, and the development of the All Events Response Operations (AERO) database (will be included in an evaluation of PHAC’s emergency preparedness and response activities scheduled for 2016-2017).

The evaluation was undertaken in fulfillment of the Financial Administration Act (for grants and contributions) and the Treasury Board of Canada’s Policy on Evaluation (2009). The evaluation was conducted by the Health Canada and the Public Health Agency of Canada’s Office of Audit and Evaluation in accordance with the Five-Year Evaluation Plan 2015-2016 to 2019-2020.

Program Description

Broad issues facing Canada’s public health workforce were brought to the forefront in seminal reports preceding and following the SARS crisis in 2003, highlighting that Canada’s public health system required critical attention to address its lack of capacity to fulfill essential public health functions and pointing to the need for action. Key changes to Canada’s public health landscape made in response included the establishment of PHAC in 2004, the federal/provincial/ territorial Pan-Canadian Public Health Network and Pan-Canadian Framework for Public Health Human Resources Planning, and a significant increase in the number of post-secondary Masters of Public Health programs.

Prior to, and during this time frame, PHAC contributed to Canada’s public health workforce through a variety of activities currently delivered by the Centre for Public Health Infrastructure (CPHI) in PHAC’s Health Security Infrastructure Branch. These include six distinct programs with varying target populations designed to develop and maintain Canada’s public health workforce (including that of PHAC) through efforts aimed at ensuring that the workforce has the competency and capability to respond to public health issues, including day-to-day and during public health events:

• Canadian Field Epidemiology Program (two-year training program in applied epidemiology);
• Canadian Public Health Service (temporary placement of trained federal public health employees across Canada);
• Field Services Training and Response Training Unit (delivery of training on applied epidemiology);
• Public Health Scholarships and Capacity Building Initiative (comprised of the Public Health Workforce Products and Tools Contributions program, the Applied Public Health Chairs program, and the Dr. James Rossiter MPH Practicum Awards program);
• Skills Enhancement for Public Health (the primary component being Skills Online); and
• Public Health Workforce Unit (lead on the implementation of the Core Competencies for Public Health in Canada Release 1.0).

CONCLUSIONS - RELEVANCE

Continued Need

PHAC’s PHWD activities continue to be relevant programs in Canada, as challenges affecting the public health system continue and may impede the ability of the public health sector to carry out its essential functions. PHAC’s current efforts provide support, specifically in response to public health workforce needs, related to insufficient numbers of competent personnel in all regions and knowledge and skill deficiencies among new and existing public health staff.

Shortages of workforce personnel and therefore capacity continue to exist across the country. While this is more pronounced in the P/Ts that lack a provincial public health agency, these shortages are noted in all P/Ts. In addition, according to key informants (i.e., public health experts, P/T representatives, and program staff) knowledge and skill deficiencies among new and existing public health personnel persist. Combined, these challenges consequently, potentially expose populations to increased public health risks.

In addition to these ongoing workforce needs, key informants expressed concerns about higher-level system or infrastructure needs (including inadequate federal/provincial/territorial coordination and collaboration on workforce and system issues, and insufficient workforce data to accurately identify gaps and enable human resources planning) that have persisted since they were first raised in post-SARS reports and which may limit the impact of ongoing efforts to build the public health workforce.

Alignment with Government Priorities

PHAC's efforts to develop Canada's public health workforce are broadly aligned with the Government of Canada’s priorities to address the health and safety of Canadians, and to develop a highly skilled workforce, as noted in Government of Canada strategic policy and planning documents, including the Speeches from the Throne.

Within the Public Health Agency, improving public health capacity has consistently been noted as a priority in many corporate planning and strategic documents (i.e., Reports on Plans and Priorities, Strategic Horizons Document, and the Corporate Risk Profile) over the past five years.

Alignment with Federal Roles and Responsibilities

It is appropriate for the federal government and PHAC to support the development of Canada’s public health workforce.

Existing legislative and/or program authorities speak to the federal role and provide PHAC with a mandate to develop the knowledge and skills of public health professionals through training and development activities, to support skill development of front line professionals through the production of tools and best practices, and to support public health training in Canadian universities (e.g., bursaries, scholarships). These documents also provide PHAC with a mandate to augment recruitment and development activities to increase the number of experienced public health professionals available for positions across Canada, and to support improved linkages between universities and local public health units.

Other players (e.g., P/Ts, Academic Institutions, Professional Associations, and National Collaborating Centres for Public Health) are also contributing towards Canada’s public health workforce needs through similar efforts. While minimal evidence of overlap or duplication with PHAC activities was observed, so too was there minimal evidence of coordination among the various players and their efforts. As a result, there is an opportunity to re-think the federal investment, in terms of level and scope, by considering a stronger PHAC leader/convener role. This is supported by public health stakeholders, who support PHAC’s current PHWD efforts, but also expressed a desire for the Agency to enhance its efforts as a leader and convenor.

CONCLUSIONS - PERFORMANCE

Achievement of Expected Outcomes (Effectiveness)

In general, PHAC’s PHWD activities are demonstrating progress towards achieving intended immediate and intermediate outcomes.

Through the provision of opportunities for, and delivery of, formal training, mentoring, experiential learning, and practicum placements, PHWD programs (i.e., Skills Online, CFEP, CPHS, APHC and Dr. James Rossiter MPH Awards) have contributed to the improved knowledge and skills of public health professionals and students. By supporting the synthesis of evidence-based knowledge (APHC) PHWD activities have also contributed to this outcome.

PHAC’s PHWD activities have resulted in greater access to Agency staff and, in some cases, tools for public health stakeholders. Through these programs (CFEP, CPHS, Dr. James Rossiter MPH Awards), PHAC staff and MPH students have been placed in public health organizations across the country to support of day to day and emergency response functions. Through these placements, PHAC staff and students have contributed to improved capacity in host organizations. While tools have been developed through these programs (i.e., Core Competencies for Public Health, Public Health Workforce Products and Tools program), the extent to which they have been made available to public health organizations has been inconsistent. Although some tools have been accessible to the public health workforce writ large, others appear to have remained solely with the funded organization. Impact and use data associated with these tools also varies, however, evidence of uptake and use of the Core Competencies for Public Health by public health organizations was collected.

PHAC’s Workforce Development programs have contributed to enhanced knowledge and skills of public health professionals and to improved capacity of public health organizations. However, the extent to which these efforts have resulted in the public health workforce having the abilities to execute their public health functions is unclear. Given that knowledge and skill deficiencies in the public health system are not well defined, the degree to which knowledge and skill improvements align with public health workforce needs is unclear. A similar statement can be made in regards to the extent that PHAC tools are addressing capacity gaps and helping to meet system needs.

Demonstration of Economy and Efficiency

PHAC’s PHWD activities are funded at approximately $8M annually and the funding is spread across six program areas.  As a result, PHAC is involved in a range of activities, however, the depth of involvement in any given activity is limited.  With the same funding allocation, an investment could be made in select key strategic priority areas, or as discussed in Section 4.3, PHAC may choose instead to shift its role or approach from one of “doing” to one of coordinating and/or leading.

Currently, examples of collaboration within the PHWD program areas, as well between the PHWD programs and other areas of PHAC, exist; however, there is a lack of a formalized approach to collaboration. Formal governance mechanisms are required to oversee PHAC’s PHWD activities, particularly related to the CBMI. Of note, the existing PHWD strategic plan document cites improved program coordination and governance as a goal.

In general, the PHWD programs have been successful in monitoring performance, and each program has tailored the collection of performance data to suit their needs. The degree to which performance data is systematically analyzed and used to inform program delivery was less clear, although examples of this were noted. Moving forward, improved consistency across the various PHWD programs to systematically review and use performance data is needed.

RECOMMENDATIONS

The findings from this evaluation of PHAC's Public Health Workforce Development activities have resulted in the following three recommendations.

Recommendation 1:

Re-assess the federal investment (in terms of level and scope), and determine the appropriate federal role (mix of activities) in building the public health workforce. Consideration should be given to the need to:

• Convene stakeholders and enhance efforts as a leader/convener;
• Streamline ‘like’ efforts underway within PHAC; and
• Maximize program impact and efficiency by investing in activities with sustainable impacts.

PHAC currently invests approximately $8M per year (not including the amount transferred to CIHR) towards the PHWD activities. Spread across six different programs (i.e., CFEP, CPHS, FSTR, PHWU, PHSCBI and SEPH), PHAC uses this funding to accomplish many objectives on a national scale. This allows for PHAC to be involved in a breadth of activities, at the expense of the depth to which it can be involved in any given one. However, there may be opportunities for streamlining investments, such as the two programs which fund MPH student practicums (i.e., CPHS, Dr. James Rossiter MPH Practicum Awards program). In addition, program impacts could be enhanced by prioritizing the sustainability of desired outcomes in program design (e.g. supporting the dissemination of successful products and tools). Other players (e.g., P/Ts, Academic Institutions, Professional Associations, and National Collaborating Centres for Public Health) are also contributing towards Canada’s public health workforce needs through similar efforts. These stakeholders have expressed a desire for PHAC to take on a stronger leadership or convener role in this area. In general, PHAC’s current efforts reflect more of a ‘doing’ role than a ‘leading’ or ‘convening’ role. It is timely for PHAC to re-assess current PHWD efforts with a mindset towards identifying the most appropriate federal role moving forward.

Recommendation 2:

Following the first recommendation, develop a strategic plan to articulate and communicate the federal role related to public health workforce development.

A strategic plan document was recently developed for PHAC’s PHWD programs in response to a 2013 internal audit. While this document provides an overarching vision for PHAC’s PHWD activities, it focuses primarily on internal efforts to build capacity, and does not clearly articulate how external facing programs will contribute to this vision. In light of the first recommendation (to determine the appropriate mix of activities moving forward), an update to this document will be necessary to clearly articulate the federal role with respect to PHWD moving forward.

Recommendation 3:

Establish formal governance to guide PHAC’s internal workforce development activities (i.e., CBMI), and formalize a roll out with clear timelines, objectives and expectations.

Recently, PHAC’s PHWD activities have been directed internally to support the Agency’s workforce. This includes the CBMI, which aims to support a talent management culture within PHAC. To date, the roll out of this initiative has been based on the identification of early adopters and volunteers from across PHAC, with no formal, senior management-approved, collaborative work plan, led by a formal governance structure, to guide the process. There is an opportunity to establish a formal governance structure to guide this initiative moving forward. With this governance in place, timelines, objectives and expectations related to the initiative could be clarified for all stakeholders.

Management Response and Action Plan

1.0 Evaluation Purpose

The purpose of the evaluation was to assess the relevance and performance of the Public Health Agency of Canada’s Public Health Workforce Development (PHWD) Activities for the period of April 2010 to December 2015.

The evaluation was undertaken in fulfillment of the Financial Administration Act (for grants and contributions) and the Treasury Board of Canada’s Policy on Evaluation (2009). The evaluation was conducted by the Health Canada and the Public Health Agency of Canada’s Office of Audit and Evaluation in accordance with the Five-Year Evaluation Plan 2015-2016 to 2019-2020.

2.0 Program Description

2.1 Program Context

Broad issues facing Canada’s public health workforce were brought to the forefront in the 1998 Krever Commission^{Footnote 1}, the 2002 Kirby Report^{Footnote 2}, and the seminal report issued by Dr. Naylor in response to the SARS crisis in 2003^{Footnote 3}. These reports highlighted that Canada’s public health system required critical attention to address its lack of capacity to fulfill essential public health functions, and pointed to the need for action.  Further details on the identified gaps are outlined in section 4.1.

In response, key changes to Canada’s public health landscape were made and have included:

• the establishment of the Public Health Agency of Canada (PHAC) and the position of the Chief Public Health Officer to serve as the federal lead on national public health issues (2004);
• the development of the landmark federal/provincial/territorial (F/P/T) Pan-Canadian Framework for Public Health Human Resources PlanningFootnote 4, which recognized the need for inter-jurisdictional collaboration and presented ‘building blocks’ that were seen as necessary to make progress (2005);
• the launch of the Pan-Canadian Public Health Network, a network of individuals across Canada from many sectors and levels of government, who work together to strengthen public health in Canada (2005). This network struck the Public Health Human Resources Task Group (disbanded in 2012), which piloted a workforce enumeration effort, drafted the first set of public health competencies, developed the Guidelines for MPH Programs in CanadaFootnote 5, and assessed the idea of creating a Canadian accreditation system for schools and programs of public health; and
• a significant increase in the number of post-secondary Masters of Public Health (MPH) programs, from five (prior to 2003) to 17 (as of 2015).^{Footnote a}

Prior to, and during this time frame, PHAC contributed to Canada’s public health workforce through a variety of activities. Currently, they are collectively known as PHAC’s PHWD activities, and are delivered by the Centre for Public Health Infrastructure (CPHI) in PHAC’s Health Security Infrastructure Branch. The first of these programs to be established was the Canadian Field Epidemiology Program (CFEP) in 1975. Skills Online was launched in 2002, and the Core Competencies for Public Health in Canada Release 1.0 (Core Competencies) were released by PHAC in 2007. The Public Health Scholarships and Capacity Building Initiative (PHSCBI) was formed in 2006, merging the former Public Health Professionals program and the Scholarships, Bursaries and Community Apprenticeships in Public Health program. The Canadian Public Health Service (CPHS) program was created in 2006, while the Competency Based Management Initiative (CBMI) was initiated in 2015. Each of these programs is described in more detail in section 2.2.

Of note, other PHAC efforts, outside the scope of this evaluation (i.e., they do not fall within Program Alignment Architecture 1.1.1: Public Health Workforce), were also made in support of Canada’s public health workforce during this time frame. For instance, following the release of the Core Competencies, PHAC supported the establishment of  a number of discipline-specific competency sets (e.g., Public Health Nursing Discipline Specific Competencies, Minimum Competencies for Medical Officers of Health, Health Promoter Competencies), all of which were informed by the Core Competencies. Each of these competency sets were developed by professional organizations and focussed attention on the need to incorporate competencies into undergraduate and graduate-level education programs. Additionally, PHAC launched, in 2005, the National Collaborating Centres for Public Health. Located regionally across Canada, the six National Collaborating Centres are designed to promote and support the use of knowledge and evidence in public health programs, policies and practices.

This evaluation represents the first time that PHAC’s PHWD activities under Program Alignment Architecture 1.1.1 have been evaluated as a whole. Separate evaluations were completed for PHSCBI in 2010-2011 and CPHS in 2013-2014.

2.2 Program Profile

PHAC’s PHWD activities (summarized below), which include six distinct programs with varying target populations but similar funding amounts, are designed to develop and maintain Canada’s public health workforce (including that of PHAC) through efforts aimed at ensuring that the workforce has the competency and capability to respond to public health issues, including day to day and during public health events.^{Footnote b}

Canadian Field Epidemiology Program

The CFEP contributes to Canada’s public health capacity through the training of public health professionals in applied epidemiology. This includes specialized techniques and competencies required to respond to diverse public health issues in real-life settings. Field epidemiologists in the program spend two years as full-time PHAC employees, although they work and learn at placement sites within a provincial, territorial, regional or municipal government department or public health agency. Placement sites provide the infrastructure, opportunities and mentoring needed to meet program requirements. During the two year placement, participants must complete nine professional experience guidelines that range from giving an oral presentation to a scientific audience to participating in a field investigation.^{Footnote 6} A list of all nine professional experience guidelines is included in Appendix 1. The CFEP supports Canada’s public health emergency response capacity, through a component of the program which involves the mobilization of field epidemiologists within Canada and around the world, to support public health organizations as they respond to urgent public health events.

Canadian Public Health Service Program

The CPHS program aims to help address Canada’s public health capacity needs through the temporary placement of trained federal public health employees (Public Health Officers) in public health organizations (placement sites) across Canada. Public Health Officers (PHOs) represent a variety of disciplines including epidemiologists, policy analysts and nurses. Placement sites may include provincial, territorial and local health authorities as well as public health-oriented non-governmental organizations.

Following the 2013-2014 evaluation, the CPHS program shifted its strategic focus and revised its site selection criteria to favour placement sites with known capacity needs (i.e., northern, rural and remote settings). PHO placements are two years in duration, during which time PHOs contribute to increased public health capacity of the placement site. As a result of this placement, PHOs are exposed to a range of public health issues and settings, helping to equip them for potential mobilizations (federal response to public health events) should the need arise. A sub-component of the program places MPH students into public health organizations in northern communities to facilitate the completion of their research practicums, help contribute to the host organizations’ capacity needs, and stimulate student interest in considering a career in northern Canada.

Field Services Training and Response (FSTR) Training Unit

The Training Unit situated within FSTR Division develops and delivers training to build the professional and technical competencies needed for applied epidemiology.  This specialised training helps prepare Field Epidemiologists, PHOs and other PHAC staff to respond to routine and emerging public health threats in Canada and abroad. The majority of the training is delivered in person to facilitate peer to peer exchange, knowledge synthesis and application of learning. Examples of training events include Epidemiology in Action, an annual three week course providing the foundations for applied public health investigations, and the Field Service Training Institute, a week-long event covering a range of applied public health training modules.

Public Health Scholarships and Capacity Building Initiative

Consisting of the Public Health Workforce Products and Tools Contributions program, the Applied Public Health Chairs program, and the Dr. James Rossiter MPH Practicum Awards program, the PHSCBI was launched with a view towards increasing PHAC’s readiness capacity in people, systems and tools. The Initiative, delivered through a combination of grants and contributions and in partnership with Canadian Institutes of Health Research (CIHR), supports three key objectives: to increase the number and skills of public health professionals; to enhance relationships between university programs in public health and public health organizations; and, to develop public health training products and tools.

• The Public Health Workforce Products and Tools Contributions Program supports the development of products and tools designed to improve the skills and capacity of public health practitioners. Through contribution funding ($75K -$250K), the program supports eligible public health organizations (e.g., professional associations, local public health units, and academic institutions) over a three-year period to develop these tools. It is expected that the products and tools developed will contribute towards advancing the implementation and uptake of core competencies in the workplace leading to enhanced competencies and greater capacity.
• The Applied Public Health Chairs (APHC) Program is delivered by CIHR’s Institute of Population and Public Health in partnership with PHAC and with contributions from other CIHR Institutes and partners. Through this program, public health faculty in academic institutions across Canada have received funding to support public health intervention research, capacity building and knowledge translation activities. There have been two rounds of funding, each lasting five years: the first in 2007-08 (up to $925K per award for five years, and one-time only supplement of $75K for knowledge translation) and the second in 2013-14 (up to $925K per award for five years). The APHC Program contributes to increasing the visibility of public health in academia through public health intervention research and knowledge development, as well as through the participation of the Chairs in knowledge exchange, partnership building, curriculum development and mentoring. In addition, the APHC program provides learning opportunities to new researchers and to the students who work with them, with the objective of further enhancing the knowledge and skills of future public health professionals.  Finally, the program aims to strengthen relationships and interactions between academia and public health practice, by requiring that all grant recipients work in collaboration with public health organizations as part of the applied research process.
• The Dr. James Rossiter MPH Practicum Awards Program provides existing MPH students seeking workplace internships with stipends and travel allowance in order to facilitate participation in public health workplace placements as part of their MPH curriculum requirement. Through participation in practicum placements, students are provided an opportunity to gain practical public health experience to enhance their skills, as well as opportunities to enter the public health profession. Conversely, host organization capacity stands to benefit through the additional student resource to address specific areas of interest for them. 

Skills Enhancement for Public Health (SEPH)

The Skills Enhancement for Public Health program aims to provide flexible learning options to develop knowledge, skills and competencies in the domain of public health. This program delivers on-line continuing education to front line public health professionals across Canada to help develop and maintain a competent and current public health workforce. The primary component of the program is Skills Online, a series of online courses covering a variety of public health topics that have been mapped to the Core Competencies for Public Health in Canada Release 1.0.^{Footnote 7} The program offers both facilitated and self-directed courses. Facilitated courses are offered at a cost of $75 to $175, while self-directed courses are free of charge. Facilitated courses are offered in the fall, winter and spring and take eight weeks to complete, and range in topics from basic epidemiological concepts to health literacy for public health professionals. The self-directed courses can be taken at any time. The self-directed courses currently offered are Core Competencies for Public Health in Canada Toolkit, Introduction to Public Health in Canada, Introduction to Literature Searching and Introduction to International Health Regulations.

Public Health Workforce Unit

The Public Health Workforce Unit has served as PHAC’s lead on the implementation of the Core Competencies for Public Health in Canada Release 1.0. The Core Competencies are defined as the essential knowledge, skills and attitudes necessary for the practice of public health. Since their release in 2007, the Public Health Workforce Unit has concentrated on facilitating uptake and integration of the Core Competencies in public health organizations across Canada, and has also supported the development of discipline-specific competency sets.

More recently, the Public Health Workforce Unit has begun piloting a CBMI internally within PHAC. As an internal-facing effort, this initiative does not fall within PHAC’s Program Alignment Architecture (PAA). This initiative supports the implementation of a talent management culture within the organization, thereby facilitating the development and management of PHAC’s workforce skills. The CBMI involves the identification of competency profiles for PHAC staff, informed by a broader competency dictionary. Competency profiles enable informed discussions between managers and staff regarding learning and development needs of employees. The intent is that this initiative will be implemented Agency-wide, however currently it is only in the pilot phase.

2.3 Program Narrative

According to PHAC’s 2016-2017 Performance Measurement Framework, the expected result of the public health workforce program area is “Public health partners and stakeholders have the abilities necessary to execute their public health functions.” The activity areas, outputs, immediate and intermediate outcomes to achieve this final outcome are described below.

Three main activity areas lead to the achievement of outcomes: professional development and training, workforce direction setting and coordination, and transfer payments management. These activity areas are not mutually exclusive, with many of the programs contributing to more than one activity stream.  

• The professional development and training activity stream is focused on the training of new and existing public health professionals. This stream also supports the placement of students and trainees within public health organizations so that they acquire practical, hands-on experience in public health.
• The workforce direction setting and coordination stream is focused on the development of public health competencies and the creation of competency-based products and tools to support implementation.
• The transfer payment management stream supports the development of products and tools for training and professional development in public health that meet the current needs of public health organizations and their staff.

The preceding activity streams are expected to contribute to two immediate outcomes. Firstly, public health professionals will have access to opportunities to enhance their knowledge and skills through a variety of training. Secondly, public health organizations will have access to skilled professionals and tools, including the Core Competencies, to fill capacity gaps.^{Footnote 8}

The three activity streams and two immediate outcomes are expected to contribute to two intermediate outcomes, being that public health professionals will enhance their public health knowledge and skills, and public health partners will have access to the competencies and capabilities necessary to execute their public health functions. These outcomes, in turn, will support the expected result of public health stakeholders having the abilities to execute their essential public health functions. Ultimately, a public health workforce with the skills and knowledge to meet 21st century public health needs will be a core component of a robust public health infrastructure that is able to manage public health threats of domestic and international concern.

Successful functioning of Canada’s public health system depends on collaboration and cooperation across, as well as within, multiple jurisdictions (i.e., F/P/T). The interdependent nature of the work presents a risk to both program delivery and performance reporting given that it requires active collaboration and partnering of the program area with internal and external partners, with PHAC having direct influence, but not direct control, over the achievement of some outcomes.

The connection between the program activity areas and the expected outcomes is depicted in the program logic model (see Appendix 2). The evaluation assessed the degree to which the defined outputs and outcomes have been achieved.

2.4 Program Alignment and Resources

The programs are part of PHAC’s PAA, Program 1.1: Public Health Infrastructure, Sub-Program 1.1.1: Public Health Workforce. The combined programs’ planned budgets for the years 2010-2011 through 2014-2015 are presented below (Table 1). Overall, PHAC’s PHWD activities had a planned budget of approximately $40 million over these five years, with another $5.7 million transferred to CIHR (through the Annual Reference Levels Update process).  The variance between the programs’ budgets and expenditures are found in Table 3. A more detailed breakdown (by program/activity area) of planned budgets and expenditures can be found in Appendix 3. 

In 2014-2015, approximately 72 full-time employees within the CPHI supported PHAC’s PHWD activities (note that there were an additional 17 vacant positions^{Footnote c}). Staff worked within two Divisions: the Field Services Training and Response Division (n=46), and the Public Health Capacity Division (n=26). The staff complement was composed of 2 directors, 18 managers, 15 public health officers, 12 field epidemiologist, 9 policy analysts, 7 administrative positions, 4 epidemiologists, 3 advisors, 1 learning officer, and 1 nurse.

3.0 Evaluation Description

3.1 Evaluation Scope, Approach and Design

The scope of the evaluation covered the period from April 2010 to December 2015, and as noted earlier, covered PHAC’s PHWD activities identified under Sub-Program 1.1.1: Public Health Workforce of the PAA, as well as the internal-facing Competency Based Management Initiative.

A number of other program and activity areas in PHAC also contribute to the development of the public health workforce, however these were out of scope for this evaluation (as they do not fall within PAA 1.1.1), specifically: the Public Health Law and Ethics Program (which ended in 2012), the National Collaborating Centres for Public Health (recently evaluated in 2014-2015), other PHAC training efforts and the emergency response work of the CPHI. The development of the All Events Response Operations (AERO) database is housed within PAA 1.1.1, however it will be included in an evaluation of PHAC’s emergency preparedness and response activities scheduled for 2016-2017.

The evaluation issues were aligned with the Treasury Board of Canada’s Policy on Evaluation (2009) and considered the five core issues under the two themes of relevance and performance, as shown in Appendix 4. Corresponding to each of the core issues, specific questions were developed based on program considerations and these guided the evaluation process.

An outcome-based evaluation approach was used for the conduct of the evaluation to assess the progress made towards the achievement of the expected outcomes, whether there were any unintended consequences and what lessons were learned. The evaluation has been designed to demonstrate the likely contributions of the programs to the expected outcomes, rather than demonstrate direct causal links between the programs and outcomes. The Treasury Board’s Policy on Evaluation (2009) also guided the identification of the evaluation design and data collection methods so that the evaluation would meet the objectives and requirements of the policy. A non-experimental design was used based on the evaluation plan, which detailed the evaluation strategy for this program and provided consistency in the collection of data to support the evaluation. As a non-experimental design, the evaluation relied on correlation to demonstrate effect, without implying causation.

Data collection started in March 2015 and concluded in December 2015. Data for the evaluation was collected using various methods, which were: document/file review, financial data review, key internal and external informant interviews, literature review, performance data review and a national survey of public health employees and employers. Data were analyzed by triangulating information gathered from the different methods listed above. The use of multiple lines of evidence and triangulation were intended to increase the reliability and credibility of the evaluation findings and conclusions. Additional detail on the data collection and analysis methods is provided in Appendix 4.

3.2 Limitations and Mitigation Strategies

Most evaluations face constraints that may have implications for the validity and reliability of evaluation findings and conclusions. The following table outlines the limitations encountered during the implementation of the selected methods for this evaluation. Also noted are the mitigation strategies put in place to ensure that the evaluation findings can be used with confidence to guide program planning and decision making.

4.0 Findings

This section provides a summary of the findings organized under two broad headings:

• Relevance: the need, priorities and federal public health role in public health workforce development
• Performance: the effectiveness, efficiency and economy of PHAC’s activities in this area.

4.1 Relevance: Issue #1 - Continued Need for the Program

There is a continued need to respond to the uneven distribution of public health personnel to address capacity disparities and to enhance the knowledge and skills of new and existing public health staff. Enhanced national level leadership and collaboration efforts may assist in addressing higher level system needs.

Prior to and following the SARS crisis in 2003, seminal reports highlighted key workforce challenges affecting Canada’s public health system, which impeded the ability of the public health sector to carry out its essential functions and respond to emergencies. Based on the performance of Canada’s public health system in responding to past public health emergencies, public health experts questioned if public health organizations in all jurisdictions had adequate infrastructure to carry out core public health functions^{Footnote d}, address emerging health issues and respond to public health events.^{Footnote 9} Reports identified workforce-related shortfalls as one aspect of a larger set of public health infrastructure needs.^{Footnote 10 Footnote 11 Footnote 12} Concerns regarding the public health workforce were centred on difficulties in assuring the availability of sufficient numbers of competent personnel in all regions, and addressing knowledge and skill deficiencies among new and existing public health staff.^{Footnote 13 Footnote 14}

At that time, shortages of key personnel (e.g., public health physicians, inspectors, epidemiologists) were noted across Canada, disproportionately affecting rural and remote regions. ^{Footnote 15 Footnote 16} Varying public health needs, deficiencies in available budgets, an ageing workforce and the inability to recruit and retain personnel were all identified as contributing factors. As a result, it was seen that existing personnel were faced with multiple demands, limiting their capacity to respond to emergencies while trying to maintain core public health functions and consequently, potentially exposing populations to increased public health risks.^{Footnote 17 Footnote 18}

Knowledge and skill deficits among public health personnel were also noted across Canada. Reports highlighted a lack of post-secondary education programs needed to develop new public health personnel, and explained that existing programs had in many cases been subsumed within faculties of medicine.^{Footnote 19} It was seen that there were challenges related to including public health content into the nursing curriculum^{Footnote 20} and recruiting medical students into public health specializations.^{Footnote 21 Footnote 22} Furthermore, it was recognized that practicum opportunities were not readily available for students, nor were there enough continuing education training opportunities to develop the skills of the existing workforce.^{Footnote 23 Footnote 24 Footnote 25}

Knowledge and Skill Deficiencies and Capacity Disparities Continue to Exist

As noted in section 2.1, efforts have been made over the last decade, by PHAC and others, in response to identified public health workforce needs. Despite fragmented progress associated with these efforts, overall workforce challenges noted in the immediate post-SARS period have been seen to continue.

Shortages of workforce personnel with adequate knowledge and skills, and therefore capacity disparities, continue to exist across the country. While this is more pronounced in P/Ts that lack a provincial public health agency, these shortages are noted in all P/Ts.  Key informants (i.e., P/T representatives and public health experts) noted that public health organizations in rural, remote and northern locations, including Aboriginal public health organizations, continue to experience difficulty budgeting for, attracting and retaining public health professionals. Stakeholders from P/Ts without schools of public health or MPH programs noted that they have particular difficulty in recruiting graduate-level trained public health professionals.

Similarly, according to key informants (i.e., public health experts, P/T representatives, and program staff), knowledge and skill deficiencies among new and existing public health personnel persist. In terms of education, the proliferation of MPH programs has resulted in an influx of MPH graduates into the public health workforce. However, key informants (i.e., public health experts, academic representatives, P/T representatives) feel that these graduates tend to lack essential public health experience and skills to complement their education and knowledge, and in most cases, these graduates tend to fill policy or administrative (managerial) roles, as opposed to working as front line practitioners. In addition, there is still a lack of public health content in nursing and medical degree programs (among others). For example, nursing education is under pressure to emphasise clinical acute care knowledge, leaving the nursing workforce unprepared for public health functions. There was no evidence that the recruitment of nursing and medical students^{Footnote 27} into public health practice has improved.^{Footnote 28}

With respect to professional development, although training opportunities have been made accessible through programs like Skills Online and the National Collaborating Centres for Public Health, knowledge and skill deficits among existing public health personnel are still reported by key informants. Stakeholders suggested that there are still not enough continuing education/ training opportunities for existing personnel to keep pace with changes in public health practice^{Footnote 29 Footnote 30} and for career advancement. Among respondents in the survey of public health employees (n=1411), 65% felt that gaps in public health continuing education opportunities exist. Specific topic areas noted as gaps by survey respondents included leadership (44%), knowledge synthesis, translation and exchange (38%), program planning (37%), evidence-informed public health (36%), healthy public policy (36%) and information assessment and analysis (36%).

Another possible contributing factor to the persistence of knowledge needs, despite increased availability of training, relates to the fact that choice of training does not always appear to be linked with professional development or organizational-related need. For example, in the survey of public health employees, when asked to indicate their reasoning for taking training, 86% of respondents (n=1779) indicated that it was for personal interest/development, compared to only 6% who expressed that it was for professional development. This, despite the fact that, among 1839 respondents, 48% noted that they participate in at least 20 hours of continuing education per year, and an additional 27% participate in at least 10 hours per year.

Broader Public Health Infrastructure Needs also Remain

In addition to these ongoing workforce needs, key informants expressed concerns about persisting higher-level system or infrastructure needs. For example, public health experts noted that system-level challenges highlighted in seminal reports (including inadequate federal/provincial/ territorial coordination and collaboration on workforce and system issues, and insufficient workforce data to accurately identify gaps and enable human resources planning) remain and may limit the impact of ongoing efforts to build the workforce. Further, key informants noted that public health workforce planning still faces similar challenges.

The complexity of population health issues discussed above, the involvement of multiple jurisdictions and disciplines, and the inter-professional nature of practice necessitating a mix of skills at different levels has not changed.^{Footnote 31 Footnote 32} In addition, there has not been any significant progress in enumerating the workforce.^{Footnote 33} These factors have limited the information necessary to determine the right number and mix of public health services and personnel required to address the needs for different populations. In addition, planning decisions made in one jurisdiction have implications for other jurisdictions, so work must be coordinated through and across jurisdictions and organizations. Key informants noted that these higher-level system needs would benefit from federal government leadership (discussed in Section 4.3).

4.2 Relevance: Issue #2 - Alignment with Government Priorities

Developing Canada's public health workforce is broadly aligned with federal government and Agency priorities.

Government of Canada

PHAC’s efforts to develop Canada’s public health workforce are broadly aligned with the Government of Canada’s priorities to address the health and safety of Canadians, and to develop a highly skilled workforce.

A number of previous Speeches from the Throne (2010 and 2013) have stated that “protecting the health and safety of Canadians and their families^{Footnote 34}” and creating the right conditions for “a highly skilled and flexible workforce” are priority areas^{Footnote 35}. The 2015 Federal Budget  committed to “Investing in the Health of Canadians”^{Footnote 36} , while the 2012 Federal Budget committed to “improving response capabilities to food-borne illness emergencies”, underlining the level of importance that the Government of Canada places on the country’s capacity to respond to public health demands^{Footnote 37}.

While not specific to public health, “training a highly skilled workforce” and “training the workforce of tomorrow” were highlighted in recent Budgets (2014, 2015) as priority areas. In support of this priority, ‘the Government has put in place concrete measures to directly support the development of a skilled, mobile and productive workforce’^{Footnote 38}.

Public Health Agency of Canada

PHAC has identified improving public health capacity as a priority in many corporate planning documents over the past five years.

Reports on Plans and Priorities (from 2010 to 2015) have consistently highlighted improving public health capacity as a priority^{Footnote 39 Footnote 40 Footnote 41 Footnote 42 Footnote 43}.  More specifically, they state the need to ‘facilitate Canada-wide coverage of public health professionals with the capacity to respond to public health events’. This includes placing public health practitioners in northern sites and the recruitment and mobilization of field epidemiologists and PHOs across Canada.

PHAC’s Corporate Risk Profile (2011) identifies public health workforce and capacity as one of its top ten public health risks, stating that there is a risk that ‘Canada lacks the appropriately trained workforce, tools, organization capability, and inter-jurisdictional system to respond to and plan for public health threats’^{Footnote 44}.  The 2012-2013 Corporate Risk Profile also notes that Canada lacks the appropriately trained public health workforce, organizational capability, and inter-jurisdictional systems to plan for and respond to public health threats^{Footnote 45}.

The objectives of the PHWD activities align with PHAC’s key workforce priorities outlined in the Strategic Plan 2007-2012 and Strategic Horizons 2013-18. The current document includes “strengthened public health capacity and science leadership which includes activities such as building public health capacity through a focus on provincial and territorial public health needs”, “enhanced public health security (emergency preparedness and response) which includes the capacity to mobilize the health emergency surge capacity” and “excellence and innovation in management, which includes strengthening strategic investments and partnerships that contribute to public health sustainability across Canada”. ^{Footnote 46}

Alignment between Priorities and Current Activities

At the Branch and Centre level, the workforce development components of the Health Security Infrastructure Branch (HSIB)’s strategic plan and the CPHI’s strategic and operational plans reflect priorities set out by PHAC and the Government of Canada. One of the key directions of the Branch’s strategic priorities is to “strengthen Canada’s public health system and workforce through capacity building initiatives”^{Footnote 47}. The vision articulated in the Branch’s Public Health Workforce Development Strategic Plan (2015-2019) aligns with PHAC’s commitment to help improve the capability of the public health workforce across Canada. Further, the Centre’s operational plan (2015-2018) has committed to “support the establishment and maintenance of a sufficient and competent public health workforce within PHAC and within the public health system in Canada”^{Footnote 48}, as a centre-specific priority.

4.3 Relevance: Issue #3 - Alignment with Federal Roles and Responsibilities

PHAC has a role in developing Canada's public health workforce. Current efforts to respond to the uneven distribution of public health personnel and to enhance the knowledge and skills of new and existing public health personnel align with this role. It is unclear, however, whether they represent the ideal mix for PHAC to be delivering in order to address existing needs.

In Canada, public health is a shared undertaking between federal, provincial and territorial governments, but also involves municipal governments, the private sector, non-governmental organizations, health professionals and the public. While provinces and territories have the primary responsibility for their respective human resource and capacity needs, the consequences of their capacity gaps can affect the public health system as a whole. For example, responses to public health events may be national in scope and beyond the capacity of any particular province or territory to address by itself. As such, the Government of Canada may choose to play a supporting role in this situation. Subject to governmental policy considerations, the spending powers may be used to create non-regulatory programs in areas of provincial jurisdiction.

The Department of Health Act^{Footnote 49}, as well as the Public Health Agency of Canada Act^{Footnote 50}, provide PHAC with the legislative authority to support the development of Canada’s public health workforce. The Department of Health Act establishes the Minister of Health’s powers, duties and functions relating to health, including “cooperation with provincial authorities with a view to the coordination of efforts made or proposed for preserving and improving public health”. The Public Health Agency of Canada Act established the Public Health Agency for the purpose of assisting the Minister in exercising or performing the Minister's powers, duties and functions in relation to public health. The objectives of PHAC’s PHWD activities are directly linked to the functions described above, given that strengthening the public health workforce supports the preservation and improvement of public health. The degree, however, to which current activities reflect ‘coordination of efforts’ versus the actual ‘doing’ is subject to perspective and is discussed in more detail further along in this section.

The PHAC role is further defined in the numerous program authorities (1975-2006) for the PHWD activities within scope for this evaluation. These authorities provide PHAC with a role to address public health workforce needs through activities to:

• build a skilled and responsive public health workforce across Canada, and
• provide the critical public health capacity required to prepare for and respond during a public health emergency.

More specifically, PHAC has received authorities to develop the knowledge and skills of public health professionals in Canada through training and development activities. This includes the provision of professional development through distance learning and continuing education activities, as well as applied training for epidemiologists. The authorities also outline a role for PHAC to support skill development of front line professionals through the production of tools and best practices. This includes identifying the core competencies for public health (i.e., define and obtain national agreement on them), and supporting their implementation in practice. A role for PHAC to support public health training in Canadian universities (e.g., bursaries, scholarships) is also outlined.

With respect to ensuring that Canada’s public health system has the capacity to carry out its day to day functions and is able to prepare for and respond during public health emergencies, PHAC received authorities to augment recruitment and development activities to increase the number of experienced public health professionals available for positions across Canada. To further support recruitment activities, authorities were also granted for the provision of academic awards, to attract more people to the field of public health from all jurisdictions. In addition, program authorities outline a role to support placement of PHAC staff at strategic points in P/T and local levels to support their organizational capacity needs. Other PHAC activities outlined included supporting improved linkages between universities and local public health units, and supporting the development of products and tools to meet needs of local/regional/national public health employees.

With respect to stakeholder opinion, key informants, including P/T representatives and senior public health officials, generally support PHAC’s role in developing Canada’s public health workforce, but expressed a desire for PHAC to enhance its efforts as a leader and convenor or coordinator.  This is discussed in more detail later on in this section.

Alignment between Role and Current Activities

Overall, PHAC’s PHWD activities have remained consistent with their original authorities.
With respect to the development of knowledge and skills, PHAC continues to provide training through the Skills Online program, CFEP and the FSTR Training unit. PHAC also demonstrates this role by supporting knowledge synthesis activities and sharing best practices and tools through the work of the PHSCBI program (i.e., Public Health Workforce Products and Tools Contributions Program, APHC), and the Public Health Workforce Unit (i.e., Core Competencies for Public Health in Canada Release 1.0). Current PHAC efforts with an internal focus towards implementing a systematic approach to developing and managing Agency workforce skills (Competency Based Management Initiative) also align with this role. 

With regards to ensuring Canada’s public health workforce has the capacity to respond to both day to day needs and in response to public health events, PHAC continues to support development programs (i.e., CFEP) and the placement of Agency staff in P/T and local levels across Canada (i.e., CPHS and CFEP). Finally, with respect to supporting public health training in universities, while some shifts in programming have occurred (e.g., PHAC previously offered fellowship awards, doctoral research awards, as well as a Community Medicine Residency Program) the APHC program, the Dr. James Rossiter MPH Practicum Awards program and the CPHS student program all continue to align with this role. 

Other players are also contributing towards Canada’s public health workforce needs through similar efforts. While there does not appear to be duplication, there is a lack of formal coordination among these players.

As part of this evaluation, an environmental scan of key stakeholder groups was conducted, consisting of website reviews and key informant interviews. Included in this review were P/T governments, professional associations (e.g., Canadian Public Health Association, Canadian Nurses Foundation, Canadian Nurses Association, and the Community Health Nurses of Canada), the National Collaborating Centres for Public Health, and academic institutions offering Masters of Public Health programs. Based on this environmental scan, there does not appear to be duplication of PHAC’s efforts among the stakeholder groups examined. A high level summary of each stakeholder group’s activities related to public health workforce development is provided below. 

• P/T governments - While the specific efforts of P/Ts vary, evidence of workforce development activities, led by Provincial Agencies, Ministries of Health or their equivalents, was observed for all, primarily related to the needs of their own respective jurisdiction. In the majority of P/Ts, financial incentives (e.g., tuition cash backs, relocation costs, bursaries) have been the most frequent means used to help address the shortages of health and public health professionals and their respective professional training needs; however, provinces with provincial agencies (i.e., British Columbia, Ontario and Quebec) appear to have greater capacity to offer training and continuing education. The majority of training and continuing education offered by P/Ts tends to take the form of one-time events such as workshops, conferences and webinars, and as such do not appear to duplicate or overlap with PHAC’s training offerings. While these same P/Ts were, in some cases, observed to offer online courses, they were typically targeted at specific audiences, most often those in the acute care setting (e.g., physicians, paramedics).
• Academic institutions – While perhaps not formally part of the public health system, the academic sector is nonetheless a key stakeholder and partner whose activities influence public health practice in Canada. Indeed, universities play a significant role in nurturing the capacity of future public health professionals. In addition to providing formal public health education through both undergraduate and graduate level training, universities also serve as the leading provider of online training courses for public health personnel^{Footnote 51} in Canada. While the accessibility and format of these courses limits the degree of duplication with Skills Online courses, the online training offered by each of these stakeholder groups may overlap with respect to topic areas. Universities also play a significant role with respect to public health research in Canada, a role that is currently being supported by PHAC’s PHWD activities (i.e., APHC).
• National Collaborating Centres for Public Health – The National Collaborating Centres for Public Health contribute to public health workforce development through efforts aimed at promoting the use of scientific research and other knowledge towards strengthening public health practices, programs and policies. As a knowledge hub, the six National Collaborating Centres identify knowledge gaps, foster networks and provide the public health system with evidence-based resources, multi-media products, and knowledge translation services. To foster knowledge exchange in public health, the National Collaborating Centres host webinars, workshops, and other training events. They also connect with public health practitioners at regional and national conferences, sharing their resources at outreach booths and convening networking events. While webinars and other forms of online training offered by the National Collaborating Centres tend to be free, workshops and other in-person training events tend to be associated with costs. Given that each National Collaborating Centre focuses on a specific public health topic or domain, there does not appear to be overlap between their activities and PHAC’s workforce development activities. Of note, among the National Collaboration Centres associated with universities, opportunities for term employment or student practicums are also provided.
• Professional associations– In general, the public health workforce development activities of professional associations tend to focus on facilitating knowledge exchange and providing opportunities for professional development. Knowledge exchange opportunities exist via annual conferences offered by certain professional associations and through knowledge brokering activities where professional associations link members to external resources such as training (including Skills Online) and bursaries. Additionally, some professional associations participate and share their perspectives in national working groups or networks (e.g., CPHA acts as the secretariat for the Network of Schools and Programs of Public Health). With respect to professional development, some of the professional associations included in this scan offer bursaries or financial resources to members in support of development. Similarly, some associations provide members with opportunities to volunteer or find employment within the association. 

In addition to this environmental scan, a survey of public health personnel was also undertaken as a component of this evaluation. As a means of identifying and confirming the key players involved in the delivery of public health training specifically, public health personnel were asked to indicate which training/continuing professional education providers they most commonly use. Please note that respondents were encouraged to choose all that apply. The six most commonly noted training providers (in order of popularity) among survey respondents (n=1832) corresponded to the stakeholder groups included in the environmental scan above and were: employer (60%), Skills Online (38%), Professional Associations (36%), P/T public health associations (33%), National Collaborating Centres for Public Health (31%) and Academic Institutions (31%). Details on the factors that influence the selection of training by trainees is provided in section 4.2.

While minimal evidence of overlap or duplication with PHAC activities was observed, so too was there minimal evidence of coordination among the various players and their efforts. Through interviews, key informants from all stakeholder groups (i.e., Program staff, P/T representatives, academic representatives) expressed concerns over the lack of coordinated approaches to public health workforce development in Canada. More specifically, key informants noted the importance of a collaborative effort to respond to this multi-jurisdictional challenge, as well as the risk of duplication of efforts and associated inefficiencies in the absence of such efforts (discussed in more detail in Section 4.5). 

There is an opportunity to re-think the federal investment, in terms of level and scope, by considering a stronger Agency leader/convener role.

Currently, the majority of PHAC’s PHWD activities involve the direct delivery of programs and/or activities (e.g., the delivery of training through Skills Online), as opposed to the coordination of efforts. While key informants (i.e., P/T representatives, academic representatives, and Program staff) support PHAC’s current efforts, as previously noted, they also expressed a desire for the Agency to enhance its efforts as a leader and convenor. This sentiment is consistent with recommendations made in the Naylor report that PHAC provide national leadership in the field of public health.

Particular areas where key informants have expressed a desire for PHAC leadership relate to the higher-level public health system needs that were discussed in section 1.1.  These include a desire for PHAC leadership in public health human resource planning, primarily facilitated through improved F/P/T coordination and collaboration on workforce and system issues. This improved coordination and collaboration would enable the federal government to convene the necessary players and to ensure clear roles and responsibilities moving forward, while minimizing the risk of potential duplication of efforts.

While the Public Health Infrastructure Steering Committee of the Public Health Network provides an F/P/T forum for workforce-related discussion, key informants (including members from the Public Health Network) have expressed that competing priorities (e.g., Multi-lateral Information Sharing Agreement) have kept workforce-related issues from their agenda over the last five years. Previously, the Public Health Network also established a Public Health Human Resources Task Group to advance the implementation of a public health human resource framework. This multi-stakeholder expert group was composed of representatives from regional, provincial and federal public health employers, CIHR, universities, and broader health system human resource planning. This group, supported by the PHAC secretariat, pursued and advanced a number of workforce development initiatives; however, the group was discontinued in 2012 as part of broader Public Health Network restructuring. A primary conclusion at a recent workshop involving public health experts, P/T and academic representatives and PHAC staff was that there has since been an absence of multi-stakeholder, pan-Canadian public health workforce development activities, hampering the ability to make national-level systemic advancements on these issues.

4.4 Performance: Issue #4 - Achievement of Expected Outcomes (Effectiveness)

4.4.1 To what extent have opportunities to enhance knowledge and skills been made accessible to public health and public health professionals? To what extent have these opportunities been accessed?

Opportunities to enhance the knowledge and skills of public health professionals and students were made available through formal training, informal training and practicum placements, and were accessed by public health professionals across the country. In many cases, demand for participation exceeds availability of opportunities.

A variety of training modalities have been offered through the PHWD programs. These include formal training, informal training (including mentoring and experiential learning), and practicum placements. Despite limited materiality for programs with national scope, these training modalities provide a range of opportunities for public health professionals and students to enhance their knowledge and skills in public health. The intensity and reach of these activities are presented as a table in Appendix 5.

Formal Training

Formal training, including courses offered through the Skills Online Program and the FSTR Training Unit, as well as training offered through the CFEP, provides opportunities for a wide range of public health professionals to enhance their knowledge and skills. 

• The Skills Online program offers 13 facilitated and 4 self-directed training courses annually. The courses cover a range of topics from epidemiology and surveillance to planning and research methods, and content that reflects the Core Competencies. Through this web-based format, participants are able to access training from any location at their convenience.As of 2014, approximately 8,200 unique individuals (this includes repeat participants) had enrolled in a Skills Online course. From 2011-2013, an average of 2,400 participants enrolled in facilitated courses annually; however, enrolment rates dropped in subsequent years after the introduction of a course fee ranging from $75 to $175. Enrollment decreased by 61.5% in 2013, and a further 12.7% in 2014. Of note, however, is that in 2013, the first year a fee was charged for facilitated courses, completion rates for these courses increased by approximately 10% to around 82% and have remained stable since.
• Skills Online courses attract participants from all provinces and territories. Results from the post-module surveys from 2011-2014 for self-directed courses (Table 3) indicate that the greatest percentage of registered participants were from Ontario (45%, n=3,172), Alberta (12.5%, n=879) and British Columbia (11%, n=792), while the lowest were from Prince Edward Island (0.68%, n=48), Nunavut (0.43%, n=30) and Yukon (0.24%, n=17). Table 4 shows the top five provinces by number of Canadian participants that completed a facilitated course per year from 2011-2014. Of note is that, prior to the implementation of fees in 2013, the three provinces with the greatest capacity and population also had the highest numbers of participants completing courses (i.e., Ontario, Québec and British Columbia).However, after the fee implementation in 2013, Nova Scotia and New Brunswick had the second and third highest numbers of participant course completion after Ontario. Nova Scotia continued to rank second of all P/Ts in 2014. Based on this, it appears that there is a greater need for facilitated courses in lower capacity provinces (i.e., Nova Scotia and New Brunswick). This is aligned with feedback collected through interviews, whereby key informants (e.g., P/T representatives, PHAC staff) confirmed that Skills Online was a source of training for all P/Ts, but stressed that P/Ts defined as having greater capacity gaps were dependent on it given that other sources were not available to them. 

According to the survey of public health professionals, Skills Online also attracts participants frommany different occupational categories. The survey showed that, over the past five year period, 30% of respondents who reported having participated in Skills Online were Public Health Nurses (n=205); 9.2% were managers (e.g., supervisors and middle management (n=63)); 7.6% were consultants/specialists (e.g., scientists, engineers, clinical resource nurses (n=52)); and 6.3% were health promoters (n=43). Of note is that approximately 11% of reported participants in Skills Online were PHAC or Health Canada employees.

Results from the survey of public health professionals also showed that awareness and perceived quality of the Skills Online program among public health professionals is high. For example, 68% (n=1,843) of respondents from the survey were aware of the program and of those, 55% (n=1,257) had participated in a course in the previous five years. Of those that who were aware, but had not taken a course(n=561), 26% stated that course offerings were not relevant to their role, while others identified cost (19.1%) and previously established preferences for training (10.6%). Few respondents indicated program quality (2%) or program credibility/reputation (2%) as a reason.

• The FSTR Training Unit provides training on a range of issues including outbreak investigation methods, advanced epidemiology analysis, surveillance, mass gathering preparations, and leadership and resiliency skills. In addition, Training Unit staff also routinely provide one-on-one training/technical support to field staff as needed (e.g., statistical data analysis for outbreak investigations, advanced social network analysis) and have assisted in the development and delivery of numerous other training activities led by partners (e.g., supported the International Health Regulations (IHR) focal point within PHAC to train IHR ambassadors within PHAC program areas).

  From 2010-2015, the Training Unit has provided a total of 30 training events. In 2014-2015 and 2015-2016, over 100 spaces at these training sessions were offered to PHAC employees (e.g., PHOs, CFEP field epidemiologists) and over 70 were offered to external public health professionals (e.g., local, provincial, territorial and federal partners, medical residents of population health and preventive medicine programs/veterinary programs across Canada, and international partners). Demand for the training offerings is high, with 5-10 external public health professionals applying for every one space available for learners outside of PHAC.

• The CFEP trains epidemiologists through two-year placements with provincial, territorial, regional or local government department or public health agency. It involves formal training, knowledge exchange activities, and the requirement for graduates to meet professional experience guidelines. Since 2010-2011, 26 field epidemiologists (with another 11 in progress) have successfully completed program requirements, demonstrating that they meet their professional experience guidelines. This includes having conducted or participated in at least one field investigation of a disease outbreak or other potentially serious public health problem that requires a timely response.

  Since 2010, 37 epidemiologist trainee positions have been filled with a mix of internal and external staff. Demand for this program exceeds supply. In the last two cohorts (i.e., 2014-2016, 2015-2017), 399 individuals applied to participate, however, with strict admission requirements and the intensive nature of the program, only 11 trainees (representing 3% of applicants) were hired.

Informal Training

Informal training opportunities, such as mentoring and experiential learning, have been made available through the CPHS and the APHC programs. These programs are aimed at a specific target audience and, unlike the formal training offered through Skills Online, are not accessible to public health professionals at large.

• Although not the primary outcome of the CPHS program (given the shift in strategic focus following the 2013-2014 evaluation), PHOs were provided opportunities to enhance their knowledge and skills through experiential learning, mentoring and knowledge exchange activities. Since 2010-2011, a total of 28 PHOs have participated in 46 site placements. There are currently 17 active PHOs.
• Through the APHC program, opportunities to enhance knowledge and skills through experiential learning were made available to academic researchers (funding recipients), public health professionals and students. Including both rounds of the APHC, a total of 29 academic researchers have been funded through the program, working in close collaboration with public health practice based partners (number not tracked). In addition, approximately 275 undergraduate/ graduate students have worked and trained with the funded Chairs over the two rounds of funding.

Practicum Placements

Opportunities to enhance knowledge and skills have also been made available in the form of practicum placements for MPH students. This has occurred through the CPHS and the Dr. James Rossiter MPH Practicum Awards programs, whereby students gain public health experience during placements in public health organizations.

• Through the student practicumplacement component of the CPHS program, students are placed in northern public health organizations to gain practical experience. Student PHOs are also provided with opportunities to enhance knowledge and skills through experiential learning, formal training and knowledge exchange activities. Since 2013-2014, a total of 170 students have applied and 44 students have participated (data is not available for previous years).
• The Dr. James Rossiter MPH Practicum Awards program also supports placement of MPH students in public health organizations; however, while there is guidance from PHAC for these placements to occur in areas of lower capacity, due to the funding model in place (whereby student placements are organized through participating universities) the exact locations of the students is not known. A total of 262 practicums were funded since 2013, involving nine institutions in 2013-2014, eleven institutions in 2014-2015, and 13 institutions in 2015-2016. Demand for the program is unknown as the total number of student applications received for practicum awards was not available.

4.4.2 To what extent have these opportunities contributed to enhanced public health knowledge and skills among public health and public health professionals?

The knowledge and skills of public health professionals and students were enhanced through opportunities provided by the PHWD programs. While national level impact is unlikely, given program materiality, evidence of knowledge and skill gains being used to support public health practice were noted.

Through a combination of formal training, mentoring, and practicum placements, PHWD activities have contributed to the improved knowledge and skills of public health professionals and students. Armed with enhanced knowledge and skills, public health professionals have applied their learnings to support public health practice. 

Formal Training

Formal training, including courses offered through the Skills Online Program and the FSTR Unit, as well as training offered through the CFEP, has enhanced the knowledge and skills of public health professionals and students. 

• Feedback from Skills Online participants indicates a link between their participation and the acquisition of new knowledge and/or skills. The majority of participants of both facilitated and self-directed Skills Online courses (i.e., 97.5% based on program data, 90% according to survey data collected as part of this evaluation) felt they learned something new as a result of their participation. According to the survey of public health professionals, 87.5% (n=596) of those that participated in Skills Online in the last five years confirmed that the program increased their knowledge/skills/competencies relevant to their public health role, 46.5% (n=317) indicated that participation influenced their individual practice, and 42.1% (n=287) stated that Skills Online increased knowledge within their organization.

  Survey data for facilitated Skills Online courses from 2012-2014 demonstrates a strong, self-reported, immediate use of knowledge gained among English learners (82.7%).  While immediate usage among French learners was lower (48.8%), the majority (77.5%) indicated that they intended to apply the knowledge gained through Skills Online within their organization.

  The Skills Onlineprogram has received critical acclaim, having earned the 2011 Canadian Society for Training and Development Silver Award for training excellence in the e-learning category, and it is cited as a best practice by the Pan-American Health Organization. Results of the survey collected as part of this evaluation identified that 76% (n=528) of the respondents indicated that the content of Skills Online was above satisfactory or excellent, while 22.6% (n=157) found the content to be satisfactory.

• Participants in the CFEP acquired the necessary knowledge/skills to obtain competency in their professional experience guidelines. Through exit interviews in 2012-2013 and 2013-2014, 100% of CFEP trainees (n=15) agreed that the program prepared them to confidently perform as a field epidemiologist and to perform tasks that are related to their future career goals. This was confirmed in key informant interviews with former participants who stated that the CFEP placement filled knowledge gaps they had related to the delivery of local public health, thus providing a unique opportunity to experience and learn “first hand” the functionality of the local public health system. Through exit interviews, 87% of participants felt that the knowledge and skills acquired during the placements would prepare them for domestic and international mobilizations. This was echoed by all supervisors from the 2013-2014 exit interviews (n=4) who indicated their confidence in their field epidemiologist’s performance on a mobilization.

  The CFEP is well respected among public health professionals, with many key informants (e.g., P/T representatives, PHAC management, program participants) indicating that previous CFEP participants now hold key leadership positions in a number of public health organizations across Canada.

• Improved the knowledge of a wide range of public health professionals, including CFEP trainees and PHOs. For example, during exit interviews in 2013-2014 and 2014-2015, CFEP trainees cited the three-week Epi in Action orientation training as successful, whereby 100% indicated that the orientation and training received enabled them to confidently perform on both domestic and international mobilizations. Interviews with current and former CFEP participants confirmed that the quality of the training is high and that it effectively prepares the field epidemiologists for their placements.

  Public Health Officers in the CPHS program have also benefited from training delivered by the FSTR Training Unit. In exit interviews, all PHOs (n=5) agreed that they received adequate and sufficient training that enabled them to perform their roles. 

  Survey data collected by the Training Unit following each training event confirms this, with 94% of participants (n=160) feeling that they have learned something new from their training. In general, feedback from participants who received training from the FSTR Training Unit demonstrates that the Unit is successfully providing the right training to field staff in order to increase their knowledge and skills.

Informal Training

Mentoring and experiential learning opportunities through the CPHS and APHC programs have also contributed to increased knowledge and skills of Public Health professionals and students.

• As stated in the 2013-2014 CPHS evaluation, current and former PHOs have acquired transferable knowledge and skills (e.g., improved knowledge of the federal government’s role in public health, improved knowledge of technical tools and processes) as a result of their site placements.
• Through the APHC program, evidence of knowledge and skills gains was collected for all stakeholder groups involved. Former APHC recipients (academic researchers) noted through key informant interviews that their participation in this funding opportunity had contributed to their knowledge and skill development. This included improved understanding of research methodologies in public health. Similarly, former APHC recipients felt that their public health practice-based partners (including practitioners and policy makers), who are involved in all aspects of the research, also demonstrated knowledge and skill improvements. With respect to students (i.e., undergraduate, masters, doctoral and post-doctoral), former APHC recipients noted that they were provided the opportunity to learn and employ a variety of research methods (e.g., concept mapping, social network analysis, complexity science). While impact data is not collected on students’ learning, former APHC recipients indicated that the award attracts graduate students who wish to be trained, thus developing the next generation of researchers and practitioners. Some students under the mentorship of the Chairs have since won major awards (such as the Fulbright and the Frederick Banting and Charles Best Graduate Scholarships), while others have obtained various CIHR research awards. While this may not be an indication of the learning that took place during their experience with the Chair, it is indicative of the calibre of students the program attracts and their commitment to public health as a career choice.

Practicum Placements

Student placements through both the CPHS and Dr. James Rossiter MPH Practicum Awards programs have contributed to the increased knowledge and skills of MPH students through exposure to a variety of public health practicum experiences.

• Through exit interviews (2013-2014 and 2014-2015), students funded through the CPHS program indicated that their placement allowed them to: enhance their knowledge and skills in a practical setting that complemented their academic studies; be exposed to the public health context, work and living conditions of northern communities; be introduced to stakeholders and networks, such as community representatives and public health experts, as well as other students; fulfilled their educational requirements for graduation, and confirmed their choice of public health as a career path. Key informants interviewed as part of this evaluation indicated that the CPHS program provides students with opportunities for knowledge and skill development. 
• Qualitative evidence collected from students funded by the Dr. James Rossiter MPH Practicum Awards program reveals that students feel that the practicum placements helped them become competent public health professionals. Most students reported participating in a range of public health activities, from secondary research to knowledge exchange activities and evaluation and policy-related tasks. Students worked across a range of public health topic areas (e.g., health promotion, chronic disease prevention, infectious disease), and the majority of students reported gaining experience in at least three public health Core Competencies areas via their practicum.

4.4.3 To what extent have Agency staff and products/tools been made accessible to public health and health organizations? To what extent have they been accessed/used?

Public Health Agency staff and MPH students were made accessible to public health organizations in support of day to day and emergency response functions.

As summarized in Appendix 6, through the CPHS program, CFEP, and the Dr. James Rossiter MPH Practicum Awards program, PHAC has made resources available to public health organizations to help fill capacity gaps and meet system requirements through a variety of mechanisms.

• The CPHS programplaces PHOs in public health organizations for two-year placements, with the primary intent being for these organizations to access the skills and expertise of the PHOs. Since 2010-2011, the Program has placed 28 PHOs in 39 different locations (for a total of 46 placements).  As of January 2015, there were 17 placement sites hosting a PHO. Following the 2013-2014 CPHS evaluation, site selection criteria were revised to favour placement sites with known capacity needs (i.e., northern, rural and remote settings). Since this shift in strategic focus, 38% of PHOs (n=16) have been placed in these priority regions, however, 75% have focused their efforts on these areas.
• Through the CFEP, placement sites host CFEP trainees for two years in order to provide the trainee with the opportunity to obtain competency in their professional experience guidelines. While not the main objective of the program, the trainees do operate as an employee of the host organization and are therefore positioned to increase their short-term capacity. Since 2010-2011, 37 field epidemiologist placements have taken place with 22 distinct organizations. For the 2015 site selection process, 22 sites applied to host a field epidemiologist, with five being successful in obtaining one. Sites are selected based on their ability to host a trainee (e.g., IT infrastructure, mentoring ability, work opportunities for the trainee). Trainees have been placed primarily in Ontario (32%), British Columbia (22%) and Québec (19%).
• Through the CFEP and CPHS program, field epidemiologists and PHOs have also supported PHAC’s response to public health events. Over the past five years, PHAC has responded to all requests from provinces and territories for support during emergency response efforts, and have primarily drawn their resources from these two program areas. Between 2011-2012 and 2015-2016 (data not available for previous years), CFEP (n=63) and CPHS (n=22) have supported a total of 85 mobilizations, covering a range of issues including mass gathering event surveillance, outbreak investigations and natural disaster response. While the majority of mobilizations have taken place in Canada (n=60) (e.g., investigation and management of a syphilis outbreak in Nunavut), staff have also been deployed internationally (n=25) (e.g., Ebola response in West Africa). Of note is that the CFEP program requires field epidemiologists to conduct or participate in at least one field investigation of a disease outbreak or other potentially serious public health problem that requires a timely response. While PHOs are not subject to the same requirement, there is an expectation that they will participate as required.
• Through the CPHS and the Dr. James Rossiter MPH Practicum Awards programs, MPH students are funded to complete at minimum a twelve-week long practicum placement at a public health organization. Since 2013-2014, 44 students have been placed through the CPHS Program, and 262 through the Dr. James Rossiter MPH Practicum Awards program. CPHS student practicum placements take place only at sites with identified needs (i.e., Northern Canada). While placement location data was not collected for students funded through the Dr. James Rossiter MPH Practicum Awards program, the 13 universities that received funding in 2015-2016 were located in seven provinces^{Footnote e} across Canada. It was expected that a majority of these student practicums took place in the home province of each university, however, some of the funding was used to support students to undertake practicums in other regions.

Products and tools have been developed; however, their availability and the extent to which they have been accessed has been inconsistent.

Products and tools have been developed through the Workforce Development Products and Tools Contribution Program and the APHC Program, as well as the work of the Public Health Workforce Development Unit. While some tools have been made available to the public health workforce writ large, others appear to have remained solely with the funded organization.

• In 2007, following extensive national consultation on their development, PHAC released the Core Competencies for Public Health in Canada: Release 1.0. The Core Competencies, which are described as the essential knowledge, skills and attitudes necessary for the practice of public health, were made available online and in print for access by all. Environmental scans were conducted in 2009 and 2015 by the Public Health Workforce Development Unit, which included a survey of organizational representatives, demonstrated that awareness and organizational support for the Core Competencies across public health organizations in Canada had improved over this period. As of 2015, all organizations that responded to the survey (n=44) were aware of the Core Competencies and were carrying out their own awareness building activities related to them, compared to 47% of organizations (n=60) in the 2009 scan. In addition, data from the survey of public health professionals reveals that among 1434 respondents, approximately 70% were aware of the Core Competencies. Since the development of the Core Competencies, the Public Health Workforce Development Unit has also produced a number of resources and tools to support implementation of the Core Competencies by PHAC and other public health organizations (e.g., Core Competencies orientation module, competency based job descriptions and self-assessment). These resources and tools have been made available to public health professionals and organizations through the Skills Online Core Competencies for Public Health in Canada Toolkit.
• Through the most recent cycle of theWorkforce Development Products and Tools Contribution Program (2012-2015), there were 14 projects funded to develop training products designed to improve the skills and capacity of public health practitioners (e.g., tools to enhance use of the Core Competencies for specific professions or organizations), as well as tools to raise the profile of public health as a career for health sciences students. Funding recipients included universities, public health units, professional associations and one provincial public health agency. As part of the development process, lead organizations engaged a total of 78 partners representing local, regional and national public health stakeholders (e.g., academic, professional and public sector), thus increasing the national relevance and reach of the projects.

  While products were developed, a survey of websites related to the projects revealed that only half of the resulting products or tools (n=7) were available online (as of May 2016). While it may be the case that all lead organizations and their partners may continue to have access to the products and tools, a small number of key informants stated that the project funds were used for development and additional resources would be required to make them widely available.

• Chairs funded through both rounds of the APHC Program have created research products covering a wide range of priority public health topics, such as health promotion and chronic disease prevention in vulnerable populations, the influence of environments on health, planning and training to support the public health system, and the development and implementation of public health policies. Research topics in the second round of funding were prioritized and selected via consultations between PHAC and CIHR (i.e., Indigenous Health, Oral Health). To date (including both rounds of funding), 720 publications and 830 presentations have been generated and made accessible to public health and health professionals through their research activities. These research products have been made available to public health organizations through such means as traditional research publications, technical reports, policy briefs, presentations and knowledge tools. Bibliometric analysis (2008-2015) of published journal articles produced through Round 1 of the APHC program provides supporting evidence that knowledge generated by the Chairs is being accessed, given that on average there have been approximately eight citations per article.
• Within PHAC, the Public Health Workforce Development Unit is currently piloting a Competency Based Management Initiative (CBMI). A significant focus for this initiative is to improve PHAC’s preparedness to respond to public health events by increasing the development of and access to relevant skills across a broad range of staff and business functions. Through this initiative, competency profiles will be developed for all PHAC positions up to the Executive level, which will support HR and learning and development processes. The intent is that this initiative will be implemented Agency-wide; however, to date it has been implemented only with selected early adopters. Based on internal key informant interviews, there is limited awareness of the timelines and goals of this initiative.

4.4.4 To what extent have public health and health organizations' capacity been enhanced as a result of the Agency's Public Health Workforce Development activities?

Public Health Agency staff and MPH students have contributed to improved short-term capacity in the public health organizations where they have been placed. In some instances, efforts have also been made to support longer-term capacity.

PHAC staff, as well as MPH students, have contributed to improved short-term capacity in the public health organizations where they have been placed. Through staff placements, efforts have been made to support longer-term capacity through the development of systems and tools, and the transfer of knowledge. The increased emphasis to focus PHOs’ and CPHS student efforts in regions with known capacity needs has helped respond to public health system needs.

• As documented in the 2013-2014 CPHS evaluation, PHOs have contributed to increased short-term capacity of host sites by enhancing organizational ability to take on priority projects, facilitating greater flexibility for organizational response to emerging public health issues, and through the transfer of knowledge and skills to organizational staff.^{Footnote 53} In some instances, PHOs have made a concerted effort to build long-term capacity at placement sites through the development of tools and processes as well as the training or mentoring of staff. Recent exit interviews with site supervisors (n=3) corroborated these findings and added that PHOs helped to develop networks or relationships for the organizations as well as tools or products. Documents show that PHO project work has spanned a range of public health topics that include: communicable, zoonotic, and non-communicable diseases, mass-gathering injuries and environmental health impacts.^{Footnote 54}

  These findings were reflected in the survey of public health employees, where approximately 45% of those who self-identified as participating in the CPHS program, (n=91) noted that the PHO placement had increased the host organization's short-term capacity (61% in government organizations and 29% in public health/regional units). Furthermore, longer-term capacity gains were also reported. For example, approximately 52% indicated that the PHO placement had increased knowledge within the host organization and 35% identified that the program had positively influenced policy or practice in the organization.

  As mentioned in section 4.2.3, following the 2013-2014 evaluation, the CPHS program revised its site selection criteria to favour placement sites with known capacity needs (e.g., northern, rural and remote settings). The increased emphasis to focus PHOs’ efforts in regions with known capacity needs has helped respond to public health system needs.

• Although the CFEP’s primary purpose is to increase capacity within individuals through applied training opportunities, placements sites also stand to benefit from the increased capacity. Reporting on the 2012-2014 cohort of CFEP trainees (n=4), 75% of placement site supervisors expressed that the placement contributed to their organization’s capacity to respond to public health events. Data collected from the survey of public health employees reveals that of those who self-identified as having been involved in the CFEP program (as either a trainee or representative from host site) (n=134), approximately 48% noted that participation in the program increased the host organization's short-term capacity, primarily within government (62%) or public health/regional units (32%).

  With respect to longer-term capacity gains, 46% of survey respondents indicated that participation in the program had increased knowledge within the host organization, and 31% agreed that the program had positively influenced policy or practice in the organization. Similarly, key informants (i.e., current and former CFEP trainees, site supervisors) speaking about the CFEP indicated that trainees have not only provided organizational support during their placements, but in some cases they have also contributed towards sustained organizational impacts through the development of ongoing surveillance systems. In exit interviews, all site supervisors noted that the trainees had transferred valuable skills and knowledge to their organization and that they wished to participate again as a placement site. As noted in 4.4.3, CFEP trainees also represent a capacity resource for organizations responding to emergencies. CFEP, along with the CPHS program, have helped PHAC to answer all requests for assistance over the last five years.

• Through practicum placements, MPH students funded through the CPHS and Dr. James Rossiter MPH Practicum Awards programs have provided support to public health organizations by undertaking short-term priority projects. The majority of CPHS site supervisors indicated that the students gave them the capacity to accomplish discrete projects within desired timeframes.^{Footnote 55} All Dr. James Rossiter students specified that they had developed knowledge products for their host organizations. Both they and their academic institutions declared that the practicums had provided useful temporary capacity to their hosts. This was echoed by one host organization interviewee who said that students did represent a useful resource, but that funds were lacking in rural and remote regions to host them, pointing to the need for these programs.

The extent to which Agency products and tools are addressing capacity gaps and helping to meet system needs is inconsistent.

The use of PHAC products and tools, such as the Core Competencies, tools developed through the Public Health Workforce Products and Tools Contributions program, and the research products developed by the APHC program, is inconsistent.

• Through an environmental scan conducted in 2015, evidence of uptake and use of the Core Competencies for Public Health Release 1.0 by public health organizations (n=44) was collected. Organizational representatives reported using the Core Competencies to: orient new employees (64%), develop job descriptions for job postings (52%), define selection requirements for hiring (50%), and to define professional development objectives for staff (48%). Data collected from the survey of public health employees reveals that 640 respondents noted that their organization made use of the Core Competencies. The most common ways in which they were reportedly used were for: learning/training/development plans (75%), performance management (48%) and hiring /recruiting activities such as creating job descriptions or candidate assessments (32%).

  Examples of the Core Competencies being applied by organizations to strengthen human resources systems include the provincial governments of Nova ScotiaFootnote 56 and British Columbia.Footnote 57 They have also informed curricula for MPH programs by virtue of their inclusion in the voluntary Guidelines for MPH Programs in Canada.Footnote 58 While the Core Competencies have been widely used, interviewees confirmed that there is a sense among stakeholders that an update may be needed; however, there is uncertainty regarding who should lead this process.

• Minimal evidence exists to understand how tools created through the Workforce Development Product and Tools Contribution Program have impacted the lead organization or other partners and stakeholders. The following examples of impact exist; however, this is not consistent across all the projects.
• In Round 1, the Association of Faculties of Medicine received funding to develop a Primer on Population Health - A virtual textbook on Public Health concepts for clinicians. Web analytics from 2012 reveal that it was viewed by 53,600 individuals that year, representing all P/Ts and many international jurisdictions. The Association received funding in Round 2 to continue the work of raising the profile of public health for medical students, including a survey on best practices for public health elective rotations.
• The Canadian Association of Schools of Nursing developed the Entry-to-Practice Discipline-Specific Public Health Nursing Competencies, which have been used across Canada to design and/or update education and training courses and recruitment tools.

While examples of impact exist, challenges related to the sustainability of the efforts post-funding have also been noted. For example, Round 2 funding for Peel Public Health was used to develop and pilot five introductory-level modules on the use of data and evidence that will be offered through the University of Waterloo's Centre for Extended Learning. However, national marketing and long-term sustainability issues for the initiative have not been resolved.

• While there is evidence to demonstrate that the Chairs have contributed new knowledge to the domain of public health and that this knowledge is being accessed, it is unclear to what extent these research products have addressed capacity gaps (although selection of intervention research topic areas have been linked to public health priority issues). Two examples demonstrating use of tools developed from program are:
  • Toronto Charter for Physical Activity: developed by a Round 1 Chair, this tool was subsequently used as a basis for the City of Montreal’s active living strategy partnership  and an implementation toolkit from Middlesex-London Health Unit; and
  • Healthy Alberta Communities: developed by a Round 1 Chair, this online capacity-building tool is a guided questionnaire for community organizations that was used by Alberta Health Services and Alberta Cancer Prevention Legacy Fund to support community efforts across the province.

  Additional examples exist where research products and partnerships of the Chairs have continued post-funding.^{Footnote f} Similarly, examples of research activities influencing public health policy and planning have been observed.^{Footnote 59} In addition, the second round of APHC program included an applicant requirement to formally “identify a decision-maker as a collaborator who will be directly engaged in their research program”.^{Footnote 60} This is likely to ensure that research results will be applied in practice settings to the benefit of public health organizations.^{Footnote g} Outside of research programs, instances have been noted of Chairs assuming leadership roles in public health associations, as well as contributing additional capacity to their academic home organizations through training students and, in some cases, by helping to develop public health education programs.

• Within PHAC, piloting of the CBMI has just begun. The roll out of this initiative has been based on the identification of early adopters and volunteers, and it currently lacks a formal governance structure to guide efforts moving forward. Performance data for this initiative are not yet available.

4.4.5 Are there early indications that public health and health organizations will have the abilities necessary to execute their public health functions?

PHAC's PHWD programs have enhanced the knowledge and skills of public health professionals and have contributed to improved capacity of public health organizations. However, the extent to which these efforts have resulted in the public health workforce having the abilities to execute their public health functions is not clear.

As noted in sections 4.4.2 and 4.4.4, PHAC’s PHWD programs have contributed towards improved knowledge and skills among individual public health professionals, and have supported organizational capacity across the country, particularly in organizations with known capacity challenges. It is challenging, however, to measure the extent to which they have contributed to the system-level change, associated with the achievement of the long-term outcome “enhancing public health stakeholders’ ability to execute their public health function”. Given the materiality, as well as the dispersed nature of these programs, it is ambitious to expect that program efforts would result in overall system-level change.

As mentioned in Section 3.1, the evaluation has been designed to demonstrate the likely contributions of the programs to the expected outcomes, rather than demonstrate direct causal links between the programs and outcomes. It is acknowledged that there are many other influences that may have an impact on changes in an individual’s knowledge, skills and health behaviors. Given that these outcomes are broad in nature, it is also understood that there are a number of other players and factors contributing to the achievement of these same outcomes.

Further, while programs can demonstrate an increase in knowledge and skills of their participants, it is unclear to what extent these efforts align with system needs as data on existing knowledge/skill deficiencies in the public health workforce are not available. In addition, it is unclear what specific professions or disciplines within public health require training and where PHAC efforts would be best placed.  

There was some evidence of PHAC’s PHWD activities having positive medium-term impacts on public health professionals and organizations. However, in most cases, these were isolated examples and not necessarily systematically observed. Of note, the following examples provide evidence of PHAC’s PHWD activities contributing to broader system-level outcomes.

• The CFEP contributes to system-level impacts given that the program is designed to facilitate the re-entry of all CFEP graduates to the public health workforce at different points throughout the system, bringing with them the skills and knowledge they have developed through the program. Unfortunately, graduates are not tracked and a detailed analysis of where they end up is not possible.
• In addition to evidence of uptake and use of the Core Competencies among individual public health organizations across Canada, there are also indications of stronger impacts. For example, environmental scans and key informant interviews revealed that the Core Competencies are influencing program planning for education of students and training for professionals across the country. In addition, there are isolated examples of the Core Competencies being integrated into human resources systems at both the local and provincial level. Feedback from public health stakeholders regarding the Core Competencies included that they have helped to create shared language and goals among public health disciplines, a clearer definition of public health within the larger health care system, as well as improved the quality of specific programs.

4.5 Demonstration of Economy and Efficiency

While examples of internal collaboration exist, there are opportunities to improve efficiencies through more strategic engagement.

The Treasury Board of Canada’s Policy on Evaluation (2009) and guidance document, Assessing Program Resource Utilization When Evaluating Federal Programs (2013), define the demonstration of economy and efficiency as an assessment of resource utilization in relation to the production of outputs and progress toward expected outcomes. This assessment is based on the assumption that departments have standardized performance measurement systems and that financial systems link information about program costs to specific inputs, activities, outputs and expected results.

The data structure of the detailed financial information provided for the programs within scope did not facilitate the assessment of whether program outputs were produced efficiently, or whether expected outcomes were produced economically. Specifically, the lack of output/outcome-specific costing data limited the ability to use cost-comparative approaches. Considering these issues, the evaluation provided observations on economy and efficiency based on findings from the literature review, key informant interviews and available relevant financial data.

In addition, the findings below provide observations on the adequacy and use of performance measurement information to support economical and efficient program delivery and evaluation.

Funding

When looking at the combined financial information for all programs within scope, there were slight variances between planned spending and expenditures during the period evaluated. As illustrated in Table 5 below, over the last five years, the variances by fiscal year were not significant and ranged between minus 10% and plus 1.3%. Overall, programs have spent their allocated budgets.

Economic Program Delivery

As mentioned in Section 2.4, the funding allocation for PHAC’s PHWD activities have remained fairly constant since 2010-2011, at an average of approximately $8M per year (not including the amount transferred to CIHR through the ARLU process). Spread across six program areas (i.e., CFEP, CPHS, FSTR, PHWU, PHSCBI and SEPH), PHAC’s PHWD activities use this funding to accomplish many objectives on a national scale. As a result, PHAC is able to be involved in a range of activities; however, the depth to which they can be involved in any given activity is limited. With the same funding allocation, an investment could be made in select key strategic priority areas, rather than delivering across a breadth of activities. Investing more resources into a smaller range of key priority areas may also support the extent towards which achievement of intended outcomes could be expected.

Conversely, as discussed in Section 4.3, PHAC may choose instead to shift its role or approach from one of “doing” to one of coordinating and/or leading. Indeed, given that a range of stakeholders are involved in similar efforts, improved coordination among all players could minimize duplication of efforts and support system-level efficiencies. In this situation, PHAC resources could support a coordination role of bringing together the relevant stakeholders to collaboratively identify, and plan a coordinated response to, Canada’s public health workforce needs.

While minimal evidence of leveraging of additional funding was observed for the programs within scope, the exception to this was the APHC program which leveraged $7.9M during the two rounds of funding due to its partnership approach (an unintended but positive outcome). It is estimated that this additional leveraged funding provided for eight additional research chairs.

Of note, partnering with CIHR to support program delivery of the APHC and the Dr. James Rossiter MPH Practicum awards programs, allowed PHAC to employ CIHR’s infrastructure and to benefit from existing relationships with academic institutions.

Internal Collaboration

The 2013 internal audit of PHAC’s PHWD activities concluded that there was a need for the programs to have a consistent approach to planning and monitoring. In response, the Centre developed a Strategic Plan to provide an overarching vision for PHAC’s PHWD activities. This document focuses primarily on internal efforts to build capacity; however, it does not clearly articulate how external facing programs will contribute to this vision. During interviews, program staff were not consistently aware of the Strategic Plan document and among those who were, several noted that they did not see their program area reflected in it. As a result, there may be opportunities to enhance or formalize internal collaboration efforts across the programs.

Despite the lack of formal mechanisms to promote collaboration among the PHWD program areas, examples of collaboration have occurred including:

• Skills Online and the Training Unit of the FSTR unit recently collaborated to produce an on-line training course to prepare PHAC responders for domestic response for Ebola.  This has resulted in a new collaboration to produce an online course in contact tracing and social network analysis.
• The Public Health Capacity Development and Field Services Training and Response Divisions are currently collaborating on competency profiles for field epidemiologist and PHO positions as part of the CBMI.
• The Skills Enhancement for Public Health program and the Public Health Workforce Development Unit are presently collaborating on the development of an inventory of training and professional development opportunities in public health.

Examples of collaboration between the PHWD programs and other areas of PHAC have also been observed, including:

• Skills Online has worked with the HSIB Surveillance Unit to produce internal training for all PHAC personnel involved in surveillance activities.
• The rollout of the CBMI is collaborative in nature and to date has engaged the labs, Border Health, Centre for Food-borne, Environmental and Zoonotic Infectious Diseases, physicians, and is beginning to engage the Policy Excellence Program and cross-PHAC emergency management preparedness.
• The CPHS program, Centre for Food-borne, Environmental and Zoonotic Infectious Diseases, Regional Offices and the provinces have collaborated on the Lyme disease tri-lateral partnership. CPHS has hired three PHOs to work with provinces, Regional Offices and the Centre to advance work on Lyme disease across all provinces in Canada.
• Several PHWD programs (i.e., Skills Online, the Public Health Workforce Development Unit and the APHC) have worked closely with the National Collaborating Centres for Public Health. As knowledge hubs, the National Collaborating Centres have helped make products and tools from these programs more accessible, and have also collaborated on specific APHC projects, particularly in the areas of indigenous health and the environment.

Of note, efforts in support of more formalized approaches to collaboration have also been observed as a result of the PHWD strategic plan document. For example, there have recently been internal efforts to improve collaboration and communication across the learning programs within the HSIB branch. Following a brainstorming session in February 2015, the HSIB Learning Group (comprised of participants from learning teams across the Branch) put forward the following priorities with action steps upon which they are currently working:

• Enhancing collaboration laterally between programs to reduce duplication of effort, and
• Mapping out what expertise exists in-house and where it resides to better leverage one another’s resources.

Similarly, the third pillar of the CPHS Strategy Policy Framework focuses on Knowledge and Networks to share information, ensure sustainable knowledge transition and facilitate connectivity between programs and sites.
Improvements to other internal processes, such as governance, would also support improved collaboration efforts.  As noted in section 4.4.4, formal governance mechanisms are required to oversee PHAC’s PHWD activities, particularly related to the CBMI. Of note, the current PHWD strategic plan document includes improved coordination and monitoring of program activities as a goal.

Performance Measurement

While not all of the individual programs in scope have their own performance measurement strategy, at the PAA level the performance monitoring activities of the various PHWD programs are guided by the recently drafted Public Health Workforce (PAA 1.1.1) Performance Measurement Strategy. As part of this Performance Measurement Strategy, a PHWD-level logic model was also developed (representing all programs involved). A unique challenge faced by some of the programs in scope, however, is that not all of their activities fit within PAA 1.1.1 (e.g., elements of CFEP support PAA 1.3.1 Emergency Preparedness and Response). As a result, performance measurement data reviewed by specific PAA levels, may not always tell a complete program picture.

In general, the PHWD programs have been active in monitoring performance, although this varies by program area. In some cases, gaps in performance data were noted, and programs would benefit from improved consistency in annual monitoring of performance data. For the most part, performance data on immediate and intermediate level program outcomes is available; however, longer-term or integrated outcome data is more difficult to measure. For example, it is challenging to understand how gains in knowledge and skills have been used to support public health practice at a system level, or the ways in which products and tools have influenced broader public health practice.

Each of the programs has tailored the collection of performance data to suit their needs. For example, staff and student placement programs (i.e., CFEP, CPHS) have made use of mid-term and exit interviews with participants and supervisors to collect valuable performance information. Skills Online routinely collects performance data through surveys following each course. The Public Health Workforce Unit has used environmental scans to provide insight into the uptake of the Core Competencies across public health organizations in Canada.  The CIHR has been instrumental in collecting performance measurement data for the APHC program and the Dr. James Rossiter MPH Practicum Awards program through midterm and annual reporting. In addition, CIHR has also made available output data related to both programs on request. However, there are challenges associated with the timeliness of outcome information being made available (e.g., final reports from Round 1 Chairs were not yet available to inform this evaluation).

Among the programs, the Public Health Workforce Products and Tools Contributions program would benefit most from more systematic tracking of performance data. The identification of specific performance indicators to capture both output and outcome data, along with definitions for various terms (i.e., products and tools) would aid in this effort. Of note, this program has recently taken steps to address these issues via the establishment of a performance measurement strategy for PHSCBI. In addition, discussions have taken place between PHAC and CIHR to formalize an agreement on information sharing related to the APHC chair reports. In addition, other long-term performance measures, such as tracking the post-program employment of CFEP participants, would support future program management decisions and evaluations.

The degree to which performance data is systematically analyzed and used to inform program delivery was less clear, although evidence of this was noted. For example, following each Skills Online course, program staff analyze participant feedback and use this information to improve and guide future course delivery. Similarly, mid-term and final reports are compiled for the APHC program by the CIHR to summarize program achievements and guide future program delivery. Moving forward, improved consistency across the various PHWD programs to systematically review and use performance data is needed.

5.0 Conclusions

5.1 Relevance Conclusions

5.1.1 Continued Need

PHAC’s PHWD activities continue to be relevant programs in Canada, as challenges affecting the public health system continue and may impede the ability of the public health sector to carry out its essential functions. PHAC’s current efforts provide support, specifically in response to public health workforce needs, related to insufficient numbers of competent personnel in all regions and knowledge and skill deficiencies among new and existing public health staff.

Shortages of workforce personnel and therefore capacity continue to exist across the country. While this is more pronounced in the P/Ts that lack a provincial public health agency, these shortages are noted in all P/Ts. In addition, according to key informants (i.e., public health experts, P/T representatives, and program staff) knowledge and skill deficiencies among new and existing public health personnel persist. Combined, these challenges consequently, potentially expose populations to increased public health risks.

In addition to these ongoing workforce needs, key informants expressed concerns about higher-level system or infrastructure needs (including inadequate federal/provincial/territorial coordination and collaboration on workforce and system issues, and insufficient workforce data to accurately identify gaps and enable human resources planning) that have persisted since they were first raised in post-SARS reports and which may limit the impact of ongoing efforts to build the public health workforce.

5.1.2 Alignment with Government Priorities

PHAC’s efforts to develop Canada’s public health workforce are broadly aligned with the Government of Canada’s priorities to address the health and safety of Canadians, and to develop a highly skilled workforce, as noted in Government of Canada strategic policy and planning documents, including the Speeches from the Throne.

Within the Public Health Agency, improving public health capacity has consistently been noted as a priority in many corporate planning and strategic documents (i.e., Reports on Plans and Priorities, Strategic Horizons Document, and the Corporate Risk Profile) over the past five years.

5.1.3 Alignment with Federal Roles and Responsibilities

It is appropriate for the federal government and PHAC to support the development of Canada’s public health workforce.

Existing legislative and/or program authorities speak to the federal role and provide PHAC with a mandate to develop the knowledge and skills of public health professionals through training and development activities, to support skill development of front line professionals through the production of tools and best practices, and to support public health training in Canadian universities (e.g., bursaries, scholarships). These documents also provide PHAC with a mandate to augment recruitment and development activities to increase the number of experienced public health professionals available for positions across Canada, and to support improved linkages between universities and local public health units.

Other players (e.g., P/Ts, Academic Institutions, Professional Associations, and National Collaborating Centres for Public Health) are also contributing towards Canada’s public health workforce needs through similar efforts. While minimal evidence of overlap or duplication with PHAC activities was observed, so too was there minimal evidence of coordination among the various players and their efforts. As a result, there is an opportunity to re-think the federal investment, in terms of level and scope, by considering a stronger PHAC leader/convener role. This is supported by public health stakeholders, who support PHAC’s current PHWD efforts, but also expressed a desire for the Agency to enhance its efforts as a leader and convenor.

5.2 Performance Conclusions

5.2.1 Achievement of Expected Outcomes (Effectiveness)

In general, PHAC’s PHWD activities are demonstrating progress towards achieving intended immediate and intermediate outcomes.

Through the provision of opportunities for, and delivery of, formal training, mentoring, experiential learning, and practicum placements, PHWD programs (i.e., Skills Online, CFEP, CPHS, APHC and Dr. James Rossiter MPH Awards) have contributed to the improved knowledge and skills of public health professionals and students. By supporting the synthesis of evidence-based knowledge (APHC) PHWD activities have also contributed to this outcome.

PHAC’s PHWD activities have resulted in greater access to Agency staff and, in some cases, tools for public health stakeholders. Through these programs (CFEP, CPHS, Dr. James Rossiter MPH Awards), PHAC staff and MPH students have been placed in public health organizations across the country to support of day to day and emergency response functions. Through these placements, PHAC staff and students have contributed to improved capacity in host organizations. While tools have been developed through these programs (i.e., Core Competencies for Public Health, Public Health Workforce Products and Tools program), the extent to which they have been made available to public health organizations has been inconsistent. Although some tools have been accessible to the public health workforce writ large, others appear to have remained solely with the funded organization. Impact and use data associated with these tools also varies, however, evidence of uptake and use of the Core Competencies for Public Health by public health organizations was collected.

PHAC’s Workforce Development programs have contributed to enhanced knowledge and skills of public health professionals and to improved capacity of public health organizations. However, the extent to which these efforts have resulted in the public health workforce having the abilities to execute their public health functions is unclear. Given that knowledge and skill deficiencies in the public health system are not well defined, the degree to which knowledge and skill improvements align with public health workforce needs is unclear. A similar statement can be made in regards to the extent that PHAC tools are addressing capacity gaps and helping to meet system needs.

5.2.2 Demonstration of Economy and Efficiency

PHAC’s PHWD activities are funded at approximately $8M annually and the funding is spread across six program areas.  As a result, PHAC is involved in a range of activities, however, the depth of involvement in any given activity is limited.  With the same funding allocation, an investment could be made in select key strategic priority areas, or as discussed in Section 4.3, PHAC may choose instead to shift its role or approach from one of “doing” to one of coordinating and/or leading.

Currently, examples of collaboration within the PHWD program areas, as well between the PHWD programs and other areas of PHAC, exist; however, there is a lack of a formalized approach to collaboration. Formal governance mechanisms are required to oversee PHAC’s PHWD activities, particularly related to the CBMI. Of note, the existing PHWD strategic plan document cites improved program coordination and governance as a goal.   In general, the PHWD programs have been successful in monitoring performance, and each program has tailored the collection of performance data to suit their needs. The degree to which performance data is systematically analyzed and used to inform program delivery was less clear, although examples of this were noted. Moving forward, improved consistency across the various PHWD programs to systematically review and use performance data is needed.

6.0 Recommendations

The findings from this evaluation of PHAC's Public Health Workforce Development activities have resulted in the following three recommendations.

Recommendation 1

Re-assess the federal investment (in terms of level and scope), and determine the appropriate federal role (mix of activities) in building the public health workforce. Consideration should be given to the need to:

• Convene stakeholders and enhance efforts as a leader/convener;
• Streamline ‘like’ efforts underway within PHAC; and
• Maximize program impact and efficiency by investing in activities with sustainable impacts.

PHAC currently invests approximately $8M per year (not including the amount transferred to CIHR) towards the PHWD activities. Spread across six different programs (i.e., CFEP, CPHS, FSTR, PHWU, PHSCBI and SEPH), PHAC uses this funding to accomplish many objectives on a national scale. This allows for PHAC to be involved in a breadth of activities, at the expense of the depth to which it can be involved in any given one. However, there may be opportunities for streamlining investments, such as the two programs which fund MPH student practicums (i.e., CPHS, Dr. James Rossiter MPH Practicum Awards program). In addition, program impacts could be enhanced by prioritizing the sustainability of desired outcomes in program design (e.g. supporting the dissemination of successful products and tools). Other players (e.g., P/Ts, Academic Institutions, Professional Associations, and National Collaborating Centres for Public Health) are also contributing towards Canada’s public health workforce needs through similar efforts. These stakeholders have expressed a desire for PHAC to take on a stronger leadership or convener role in this area. In general, PHAC’s current efforts reflect more of a ‘doing’ role than a ‘leading’ or ‘convening’ role. It is timely for PHAC to re-assess current PHWD efforts with a mindset towards identifying the most appropriate federal role moving forward.

Recommendation 2

Following the first recommendation, develop a strategic plan to articulate and communicate the federal role related to public health workforce development.

A strategic plan document was recently developed for PHAC’s PHWD programs in response to a 2013 internal audit. While this document provides an overarching vision for PHAC’s PHWD activities, it focuses primarily on internal efforts to build capacity, and does not clearly articulate how external facing programs will contribute to this vision. In light of the first recommendation (to determine the appropriate mix of activities moving forward), an update to this document will be necessary to clearly articulate the federal role with respect to PHWD moving forward.

Recommendation 3

Establish formal governance to guide PHAC’s internal workforce development activities (i.e., CBMI), and formalize a roll out with clear timelines, objectives and expectations.

Recently, PHAC’s PHWD activities have been directed internally to support the Agency’s workforce. This includes the CBMI, which aims to support a talent management culture within PHAC. To date, the roll out of this initiative has been based on the identification of early adopters and volunteers from across PHAC, with no formal, senior management-approved, collaborative work plan, led by a formal governance structure, to guide the process. There is an opportunity to establish a formal governance structure to guide this initiative moving forward. With this governance in place, timelines, objectives and expectations related to the initiative could be clarified for all stakeholders.

Appendix 1 - Professional Experience Guidelines for Field Epidemiologists

All Field Epidemiologists are required to complete the CFEP Professional Experience Guidelines during their two-year term.

Appendix 2 – Logic Model

Source: Health Security Infrastructure Branch, PHAC (2015). Performance Measurement Strategy: Public Health Workforce, p. 19.

Appendix 3 – Detailed Financial Breakdown by Program Area

Appendix 4 – Evaluation Description

Evaluation Scope

The scope of the evaluation covered the period from April 2010 to March 2015, and covered PHAC’s activities identified under Sub-Program 1.1.1: Public Health Capacity Building of the Program Alignment Architecture located within the CPHI in the Health Security Infrastructure Branch. The Center’s Workforce Development activities included in this evaluation were the:

• Canadian Field Epidemiology Program (CFEP);
• Canadian Public Health Service (CPHS);
• Public Health Scholarship and Capacity Building Initiative (PHSCBI);
• Skills Enhancement for Public Health (SEPH); and
• Work conducted by the Public Health Workforce Development (PHWD) Unit, including efforts to build the workforce capacity of PHAC.

Evaluation Issues

The specific evaluation questions used in this evaluation were based on the five core issues prescribed in the Treasury Board of Canada’s Policy on Evaluation (2009). These are noted in the table below. Corresponding to each of the core issues, evaluation questions were tailored to the program and guided the evaluation process.

Data Collection and Analysis Methods

Evaluators collected and analyzed data from multiple sources. Sources of information used in this evaluation included:

Document and file review

The main purpose of conducting the document review was to obtain a comprehensive understanding of the scope and delivery of the intended activities, as well as results observed over the five-year period covered by the evaluation.

Approximately 210 documents pertinent to public health workforce development were reviewed, including those focused on performance data below. The majority of these were held by the CPHI. Other documents came from different organizations within PHAC as well as external groups, notably CIHR.

Two prior PHAC evaluations were included in this document review:

• the Evaluation of the Canadian Public Health Service Program at the Public Health Agency of Canada, July 2013; and
• the Public Health Scholarships and Capacity-Building Initiative Summary Evaluation, November 2010 (not posted) which included an annexed report by CIHR, the APHC at the Mid-Term: A report of progress to date, January 2010.

Performance data review

Analyses included a review of data on performance of program activities collected by the CPHI between 2010-2011 and 2014-2015, including:

• CFEP participant and placement site supervisor exit interviews from 2012 and 2014 cohorts, as well as an inventory of placement site locations;
• CPHS PHO participant and host organization exit interviews from 2013, 2014 and 2015, as well as an inventory of placement site locations;
• CPHS student and supervisor exit interviews from 2013-14, 2014-15, and 2015-16, in addition to numbers of student applications and placements;
• PHSCBI programs:
  • information on outputs of the APHC Chairs provided by CIHR-IPPH as available, and a bibliometric analysis of publications produced by the Round One Chairs;
  • summarized information from the Dr. James Rossiter MPH Practicum Award program reports submitted by participating students and academic organizations for 2014-15 and 2015-16;
  • four case studies of four projects funded by the Workforce Development Products and Tools contribution program;
• A summary of FSTR training events, and a listing of domestic and international mobilizations from 2011 to 2015 (this includes mobilizations of CFEP and CPHS participants);
• Performance measurement reports from Skills Online for 2012, 2013 and 2014, as well as a case study on how it has been used in Quebec; and
• 2009 and 2013 environmental scans on the implementation of the Core Competencies for Public Health.

Financial data review

• An analysis of financial data helps respond to questions of effectiveness, efficiency and economy.
• A review of financial data from 2010-2011 to 2014-2015, which focussed on planned and actual expenditures.

Key informant interviews

• Interviews were conducted with 42 stakeholders: Public Health Agency of Canada (n=20); other federal government departments or agencies (n=1); program recipients (n=7); academic stakeholders (n=2); professional associations (n=4); provincial/territorial stakeholders (n=6); other stakeholders (n=2).
• Interview questionnaires were developed and slightly modified and tailored for each specific stakeholder group. Guides were based on the core evaluation issues and questions identified in Table 1. They were developed using a semi-structured format, including probes where helpful. This format was used in order to define the areas to be explored, and also to allow the interviewer or key informant to pursue an idea or response in more detail. The flexibility of this approach, particularly when compared to structured interviews or focus groups, also allowed for the discovery or elaboration of information that was important to participants but may not have been previously thought of as pertinent by the evaluation team.
• Interviews were conducted in person when possible or by telephone.
• Interviews were recorded, with the participant's consent, transcribed and summarized as necessary. Data was coded and analysed with the aid of Excel spreadsheet software.

Literature review

• A search conducted by the Health Library for English-language Canadian literature from the 2004 to 2016 using search terms of “public health”, “workforce”, “human resources”, “manpower”, “hr”, “staff” returned 455 documents.
• After examining documents to ensure relevance, 44 articles were reviewed.
• In addition, 15 grey literature documents were included in the review featuring landmark policy documents (such as the 2003 ‘Naylor Report’ Learning from SARS: Renewal of Public Health in Canada), and internal research documents.
• Pertinent information was extracted and/or summarized from each document, and then coded in an Excel spreadsheet.

Survey

• The PHAC Office of Audit and Evaluation partnered with the Canadian Public Health Association (CPHA) to develop and field two survey questionnaires on public health workforce issues as it was discovered in the design phase of the evaluation that both organizations were independently planning similar studies. The three objectives of the collaborative Public Health Workforce Survey were to: i) gain knowledge on workforce size, composition and levels of education and training, ii) identify training priorities, supports and barriers, iii) better understand how the Core Competencies for Public Health Canada Release 1.0 have been adopted, and iv) describe challenges to recruitment and retention of qualified public health professionals.
• The Survey was comprised of:
  • a bilingual questionnaire for individuals working in public health that was launched on December 4th, 2015 and closed on January 18th, 2016. Invitations to complete the survey were disseminated through the CPHA membership, PHAC employees, and requests to partner organizations. It is estimated that 5000 individuals would have received an invitation, of which 1344 completed the survey.
  • a bilingual questionnaire for employer-level representatives of public health organizations that was launched on January 26th, 2016 and closed on February 11th, 2016. Invitations to complete the survey were sent to key CPHA contacts in approximately 74 organizations, including the 17-member Pan-Canadian Public Health Network Council, of which 11 completed the survey. Given the low response rate and therefore the limited generalizability of the information, the decision was made to not use the data from this survey as a line of evidence in the evaluation.   

Data were analyzed by triangulating information gathered from the different sources and methods listed above. This included: systematic compilation, review and summarization of data to illustrate key findings; thematic analysis of qualitative data; and comparative analysis of data from disparate sources to validate summary findings.

Appendix 5 – Intensity and Reach of PHWD Activities

Appendix 6 – Summary of PHWD Activities (inputs and outputs)

Endnotes