Evaluation of the National Collaborating Centres for Public Health Program 2008-2009 to 2013-2014

Prepared by
Evaluation Directorate
Health Canada and the Public Health Agency of Canada

April 2014

Table of Contents

• Executive Summary
• Management Response and Action Plan
• 1.0 Evaluation Purpose
• 2.0 Program Description
  • 2.1 Program Context
  • 2.2 Program Profile
  • 2.3 Program Logic Model and Narrative
  • 2.4 Program Alignment and Resources
• 3.0 Evaluation Description
  • 3.1 Evaluation Scope, Approach and Design
  • 3.2 Limitations and Mitigation Strategies
• 4.0 Findings
  • 4.1 Relevance: Issue #1 - Continued Need
  • 4.2 Relevance: Issue #2 - Alignment with Government Priorities
  • 4.3 Relevance: Issue #3 - Alignment with Federal Roles and Responsibilities
  • 4.4 Performance: Issue #4 - Achievement of Expected Outcomes (Effectiveness)
  • 4.5 Performance: Issue #5 - Demonstration of Economy and Efficiency
• 5.0 Conclusions
  • 5.1 Relevance Conclusions
  • 5.2 Performance Conclusions
• 6.0 Recommendations
• Appendix 1 - Mandates of the NCCs
• Appendix 2 - Evaluation Description
• Appendix 3 - Examples of knowledge products
• Appendix 4 - Examples of partnership activities
• Appendix 5 - Examples of impacts
• Appendix 6 - Public Health Agency/NCCPH Interactions
• Appendix 7 - Summary of international approaches to KSTE
• Appendix 8 - References

List of Tables

• Table 1: Planned versus actual expenditures 2008-2009 to 2013-2014
• Table 2: Limitations and Mitigation Strategies
• Table 3: Summary of annual knowledge product development by NCCPH program
• Table 4: Summary of annual partnerships by NCCPH program
• Table 5: Summary of total annual visits to NCCPH websites
• Table 6: Core Evaluation Issues and Questions
• Table 7: Breakdown of Key Informant Interviews
• Table 8: Summary of 2009-2013 NCCAH Interactions with the Public Health Agency
• Table 9: Summary of 2009-2013 NCCDH Interactions with the Public Health Agency
• Table 10: Summary of 2009-2013 NCCEH Interactions with the Public Health Agency
• Table 11: Summary of 2009-2013 NCCHPP Interactions with the Public Health Agency
• Table 12: Summary of 2009-2013 NCCID Interactions with the Public Health Agency
• Table 13: Summary of 2009-2013 NCCMT Interactions with the Public Health Agency

Executive Summary

This evaluation covered the National Collaborating Centres for Public Health (NCCPH) program for the period from the fall of 2008 to the fall of 2013. The evaluation was undertaken in fulfillment of the requirements of the Financial Administration Act and the Treasury Board of Canada's Policy on Evaluation (2009).

Evaluation Purpose and Scope

The purpose of the evaluation was to assess the relevance and performance of the NCCPH contribution program (hereafter referred to as the NCCPH program). In the assessment of relevance, the evaluation examined the continued need for the program and its alignment with federal government and Public Health Agency of Canada roles and priorities. In the assessment of program performance, the evaluation examined the progress made towards expected outcomes and the efficiency and economy of the program.

Program Description

Initiated in 2005, the NCCPH program is a contribution program that funds activities designed to promote and support the use of knowledge and evidence in public health programs, policies and practices. To support achievement of the program objectives, funding is delivered through contribution agreements with recipient host organizations of six regionally placed National Collaborating Centres (the NCCs). These centres each specialize in a different priority area of public health, and carry out activities in three key areas: knowledge translation, identification of knowledge gaps and network development. Together, these activities are intended to lead to improved evidence-informed decision making (EIDM) and ultimately towards improved public health programs and policies in Canada.

The six centres, and their recipient host organizations, are:

• National Collaborating Centre for Aboriginal Health, University of Northern British Columbia, Prince George, British Columbia
• National Collaborating Centre for Determinants of Health, St. Francis Xavier University, Antigonish, Nova Scotia
• National Collaborating Centre for Environmental Health, British Columbia Centre for Disease Control, Vancouver, British Columbia
• National Collaborating Centre for Healthy Public Policy, Institut national de santé publique du Québec, Montréal, Québec
• National Collaborating Centre for Infectious Diseases, International Centre for Infectious Diseases, Winnipeg, Manitoba
• National Collaborating Centre for Methods and Tools, McMaster University, Hamilton, Ontario.

Each centre has an Advisory Board that provides guidance on work plans and deliverables. Overall accountability for the funding program resides with the NCCPH program secretariat, situated in the Centre for Public Health Capacity Development, within the Health Security Infrastructure Branch, at the Public Health Agency.

Conclusions - Relevance

Continued Need

There continues to be a need to strengthen Canada's public health capacity. More specifically, there is an on-going need for effective knowledge synthesis, translation and exchange (KSTE) mechanisms to enhance evidence-informed decision making in public health in support of programs and policies that protect and promote the well-being of the public.

Alignment with Government Priorities

The objectives of the NCCPH program broadly align with Government of Canada and Public Health Agency priorities. The NCCPH program recipients' activities (hereafter referred to as program activities) are intended to enhance evidence-informed decision making in public health practice. While not explicitly mentioned as a priority, it is recognized that improved evidence-informed decision making facilitates timely and effective public health programs and policies that protect and promote the health and well-being of the public. This outcome continues to be reflected as a priority in a variety of parliamentary and corporate strategic reports.

Alignment with Federal Roles and Responsibilities

Addressing Canada's public health capacity needs is an appropriate role for the Government of Canada and the Public Health Agency. The Department of Health Act, as well as the Public Health Agency of Canada Act, provide the Public Health Agency with the legislative authority to take on this role. Through the Agency's support of the NCCs' efforts aimed at improving evidence-informed decision making in public health, the Agency is carrying out this role.

A number of other organizations are involved in KSTE to address public health capacity. In addition, the Public Health Agency participates in a number of KSTE efforts, most of which have been established since NCCPH program inception. It is unclear to what extent these various efforts, particularly within the Agency, align with or complement the NCCPH program, or each other. As a result, it is unclear if there is a continued need for the NCCPH program as currently designed.

Conclusions - Performance

Achievement of Expected Outcomes (Effectiveness)

Based on the information that was available, including performance data and key informant interviews, the NCCPH program has made progress towards increasing the use of evidence to inform public health practice; however progress varies by Centre. In general, progress has been accomplished through the KSTE work of the NCCs, including their networking and partnership activities and the production of relevant knowledge products. The networking and partnership activities of the NCCs have supported knowledge gap identification and contributed to decreased duplication of public health efforts nationally in the topic areas of the Centres. The knowledge products developed and disseminated by the NCCs have been in response to gaps identified through their networking activities. The dissemination of these credible synthesized knowledge products and tools have supported public health practitioners across the country in their efforts to incorporate evidence into their practice. The design of the program along specific themes, however, has restricted the ability of the Centres to respond to emerging public health priorities outside of the topics areas of the NCCs.

Demonstration of Economy and Efficiency

A number of efficient practices are regularly employed by each of the NCCs; however, several inefficiencies exist at the program level. The lack of centralized guidance and coordination for the Centres (including on issues such as approach to joint work-planning, approach to KSTE and partnership activities) has led to potential duplication of activities among the NCCs and missed opportunities for collaboration.

In addition, limited awareness and engagement of the program and its products among Public Health Agency staff and the Public Health Network (facilitated by the NCCPH Program Secretariat) has resulted in potential duplication of activities between the Public Health Agency and the NCCs. There is a need for the Program Secretariat to enhance program-level performance measurement in support of program decision making.

Recommendations

Recommendation 1

In consultation with key stakeholders, including P/Ts, and considering the Public Health Agency's various efforts in KSTE, determine the appropriate role for the Agency in addressing the public health system's KSTE needs.

There is evidence of an on-going need to strengthen Canada's public health capacity through effective KSTE mechanisms in support of improved evidence-informed decision making. However, the Public Health Agency is currently involved in a number of KSTE efforts, most of which have been established since NCCPH program inception. While there is still a role for the Public Health Agency in responding to this need, it is not clear that it is through the NCCPH program as currently designed. To best support the public health system, the Agency's efforts in this area should be coordinated, responsive to emerging public health priorities, and not duplicate or overlap the work of others.

Recommendation 2

Review NCCPH program mandate and determine if there is continued alignment with the articulated Agency role. If necessary, revise as appropriate.

Once the Public Health Agency's role in supporting the public health system's KSTE needs has been articulated, the mandate of the NCCPH program should be reviewed to ensure it aligns with the Agency role, and that it is complementary with other Agency efforts in this area.

Recommendation 3

Adjust program design to reflect renewed mandate.

Following the review of the program mandate, the design of the program should be examined and adjusted (if necessary) to most effectively carry out this mandate. Should the program design remain consistent with its current format, several adjustments are suggested based on evaluation findings. The NCCPH program should:

1. Enhance collaboration among the Centres, between the Centres and Agency, and with other players.
2. Maximize program flexibility to respond to emerging priorities.
3. Ensure ongoing strategic guidance and coordination are in place for the program.

Recommendation 4

Ensure performance measurement is implemented and used to assist in programmatic decision making.

Irrespective of the resulting program mandate and design, it is essential that a performance measurement system be implemented that would assist in programmatic decision making.

The current NCCPH program lacks a formal performance measurement strategy to guide the collection of performance data. While each Centre collects, analyzes and uses performance data, there is limited performance measurement in place at the program level. Centralized guidance to direct performance measurement activities of the Centres is required, including common definitions and consistent templates. Improvements to the performance measurement approach would enhance the ability to assess program success on a continuous basis.

 

Management Response and Action Plan

 

1.0 Evaluation Purpose

The purpose of the evaluation was to assess the relevance and performance of the Public Health Agency of Canada's National Collaborating Centres for Public Health (NCCPH) contribution program (hereafter referred to as the NCCPH program) for the period of 2008-2009 to 2013-2014. In the assessment of relevance, the evaluation examined the continued need for the program and its alignment with federal government and Public Health Agency roles and priorities. In the assessment of program performance, the evaluation examined the progress made towards expected outcomes and the efficiency and economy of the program.

The evaluation was required by the Financial Administration Act and the Treasury Board of Canada's Policy on Evaluation (2009). The NCCPH program was last evaluated in 2008.

 

2.0 Program Description

2.1 Program Context

The origin of the NCCPH program can be traced back to reviews and consultations undertaken by the Canadian Institutes of Health Research, Institute of Population and Public Health (CIHR-IPPH) from 2001 to 2003 on the state of knowledge generation and use in the Canadian population and public health sector.^{Endnote 1} This work found that there were important challenges faced by the sector regarding access to high quality and relevant research evidence and its use to inform decision making. Subsequently, following the SARS crisis of 2003, several Canadian commissions, consultations and reports, including the Naylor Report, concluded that there had been insufficient investments in Canada's public health infrastructure, resulting in Canada having an inadequate knowledge base to inform the development of public health programs and policies. These reports strongly recommended enhancements to Canada's public health capacity through improved evidence-informed decision making (EIDM) practices.^{Endnote 2,Endnote 3,Endnote 4,Endnote 5} In response to these concerns, the federal government committed to renew and strengthen public health in Canada. In 2004, in support of this commitment, the NCCPH program was announced by Cabinet, along with the Public Health Agency of Canada and the Pan-Canadian Public Health Network.

The NCCPH program was initiated in 2005 with an annual budget of $9.15 million. The program was established to enhance Canada's public health capacity by creating collaborative processes among researchers, the public health community and other stakeholders to analyse priority population health issues and to provide evidence and expertise for the development of mechanisms and tools to improve public health across Canada. The NCCPH Program is funded through the federal government's contribution agreement mechanism.

The NCCPH program is currently led by the Knowledge Integration and Mobilization Unit in the Centre for Public Health Capacity Development, within the Health Security Infrastructure Branch (HSIB) of the Public Health Agency of Canada. The Centre for Public Health Capacity Development is currently led by an Executive Director who has a direct reporting relationship with the Branch Head within HSIB.

2.2 Program Profile

The NCCPH Program is designed to promote and support the use of knowledge and evidence in public health programs, policies and practices through knowledge translation, identification of knowledge gaps and network development.

It is important to note that, across the public health landscape, many terms are used interchangeably when discussing activities related to knowledge translation, including knowledge synthesis, translation and exchange (KSTE), knowledge management and knowledge development and exchange (KDE). For the NCCPH program, both knowledge translation and KSTE are used. Knowledge synthesis, knowledge exchange and knowledge translation are defined by the program as follows.

• Knowledge synthesis is the systematic gathering and analysis of information about a particular topic.
• Knowledge exchange refers to the collaborative problem-solving that occurs between researchers and decision-makers that is made possible through linkage and exchange. To be effective, knowledge exchange requires the interaction between practitioners, researchers and decision makers, which leads to shared learning through the process of planning, producing, disseminating, and applying existing or new research evidence in decision making.
• Knowledge translation is a dynamic and iterative process that includes synthesis, dissemination, exchange and the ethically sound application of knowledge.^{Endnote 6}

Similarly, various terms and definitions are used to describe EIDM. In recent scientific literature, EIDM is defined as "the intentional and systematic processes of bringing the best available scientific evidence on specific questions together with other relevant information to help weigh options and inform decisions that will affect priorities, policies, programs and practices".^{Endnote 7}

2.2.1 National Collaborating Centres

Six regionally placed Centres (the NCCs) were established in 2005, each focusing on KSTE activities for a different topic area of public health. The specific mandates of each NCC are provided in Appendix 1. The topic areas of the Centres were determined at the outset of the program to align with commonly accepted public health priorities at that time.

Each NCC is situated within a host organization (three universities, two provincial government organizations, one non-governmental organization). Funding is provided under contribution agreements with these host organizations, and in addition, the hosts provide support to the activities of the centres, via human resources, administrative, information technology and office space support. The host organizations are also responsible for complying with the terms of the multi-year contribution agreements with the Public Health Agency. Under the terms and conditions, the Centres must work together to address common priorities, while fulfilling a national mandate for their individual topic areas.

The centres carry out activities in three key areas: identifying knowledge gaps in specific public health topic areas; facilitating networking across Canada's public health system with respect to these topics; and undertaking KSTE to address identified knowledge gaps and support EIDM in public health practice in Canada. These core activities and how they align with and complement program objectives are further described in the discussion on program logic and theory (Section 2.3). Each NCC consists of highly specialized subject matter experts from a diversity of fields including recognized experts in critical assessment methodology and planning. These experts help develop critical paths for knowledge generation and use for the Centres. The six NCCs, along with their host organizations and locations, are:

• National Collaborating Centre for Aboriginal Health (NCCAH), University of Northern British Columbia, Prince George, British Columbia
• National Collaborating Centre for Determinants of Health (NCCDH), St. Francis Xavier University, Antigonish, Nova Scotia
• National Collaborating Centre for Environmental Health (NCCEH), British Columbia Centre for Disease Control (BCCDC), Vancouver, British Columbia
• National Collaborating Centre for Healthy Public Policy (NCCHPP), Institut national de santé publique du Québec, (INSPQ), Montréal, Québec
• National Collaborating Centre for Infectious Diseases (NCCID), International Centre for Infectious Diseases (ICID), Winnipeg, Manitoba
• National Collaborating Centre for Methods and Tools (NCCMT), McMaster University, Hamilton, Ontario.

2.2.2 NCCPH Program Secretariat

Situated within the Centre for Public Health Capacity Development, the NCCPH Program Secretariat was established to be responsible for overall program accountability. Program authorities specified the provision of three full-time staff to carry out this role. The accountability activities of the Program Secretariat, as stated in the 2005 NCCPH program Results-Based Management and Accountability Framework (RMAF) consist of:

• acting as the main point of public communications and internal briefing on the NCCPH program for the Minister and the Agency
• ensuring compliance with financial and management accountability frameworks
• negotiating and managing the Contribution Agreements with the recipient host organizations
• performance monitoring and evaluation of Centre outputs and activities to ensure alignment with priorities of public health
• supporting and coordinating a national program Advisory Council (described below)
• facilitating priority-setting for the program as a whole to inform Centre work plans and collaborative priorities (with the participation of the Centres and the Advisory Council)
• promoting networking between the six Centres, as well as between the Centres and the Agency, the Health Portfolio (Health Canada and the Canadian Institutes for Health Research) and other federal organizations in order to better coordinate knowledge translation activities and to promote the use of knowledge products to inform policy development and program implementation
• showcasing and promoting the NCCPH Program, both within and outside the federal government.^{Endnote 8}

While the above roles for the Secretariat were outlined in their RMAF, the contribution agreements with the recipient host organizations outline the parameters for and nature of the relationship between the funder and the recipient. These parameters include that no party (funder or recipient) "shall have the right to obligate or bind the other party in any manner"^{Endnote 9}, and that the parties shall not enter into a relationship of principal-agent, employer-employee, partnership, or joint-venture. As a result, the Centres have the autonomy and independence to manage their own operations within national program guidelines. The arms-length relationship between the Public Health Agency and the NCCs limits the ability of the Agency to influence the work of the NCCs, and subsequently restricts the role of the NCCPH Program Secretariat.

2.2.3 Advisory Boards

Each National Collaborating Centre is connected to the public health community through an Advisory Board consisting of key stakeholders in the specific topic area of the Centre. These Advisory Boards consist of decision-makers, practitioners and academics from a variety of regions. In some cases, staff from one Centre may participate on the board of another Centre in order to promote collaboration. Advisory Board members provide guidance and advice on Centre work plans and deliverables, as well as assess the progress of the Centre's activities. Board members also play an important networking role, making their own personal connections available as a means to gather intelligence to inform Centre activities and to distribute knowledge products.

2.2.4 Advisory Council

The recently disbanded Advisory Council was put into place at the initiation of the program, and helped to review the original proposals for each of the six Centres. The Council was composed of national and international experts representing different sectors of public health, including a balance of researchers and practitioners. According to the Council's terms of reference, the Council provided program-level public health expertise and advice to the Program Secretariat, as well as scientific advice and insight and guidance to the individual Centres. Advisory Council responsibilities included:

• acting as a national advisory body to the Public Health Agency on the priorities for the NCCPH program
• reviewing NCC workplans and providing advice and recommendations to the NCCPH program on the underlying science and relevance of the proposed workplans to national/international public health priorities
• reviewing the work of the NCCs and providing the Public Health Agency with an assessment and evaluation of the quality of NCCs' work (or activities) and the NCCs' progress in meeting NCCPH program objectives
• advising on such other matters relating to the NCCPH program as may be requested in writing by the Public Health Agency.^{Endnote 10,Endnote 11,Endnote 12}

The Advisory Council (inactive since June, 2012) was disbanded in 2013 as part of a broader Agency review and renewal of all external advisory boards. The Program Secretariat is currently working to rebuild elements of this reduced capacity (for example, through accessing internal subject matter expertise within the Agency), and is in the process of developing a new external advisory function focused on broader knowledge translation in public health.

2.3 Program Logic Model and Narrative

Given that the NCCPH program uses the Terms and Conditions for the Promotion of Population Health Contributions as the basis for funding of the six National Collaborating Centres, the program logic model (2009) links to the overarching logic model of the Fund via the common long term outcome of "Improved public health programs and policies".

The activities, outputs and outcomes (immediate, intermediate and long-term) to achieve this final outcome were documented in the 2005 Results-Based Management and Accountability Framework and Risk Assessment for the National Collaborating Centres. The connection between these activities, and the expected program outcomes are also describe in the section below (2.3.1). This evaluation assessed the degree to which the defined outputs and outcomes were being achieved over the evaluation timeframe.

Immediate outcomes for the NCCPH program include the synthesis, translation and exchange of knowledge across diverse sectors in public health, the identification of public health knowledge gaps, increased collaboration across all levels of public health, and increased opportunities for enhancing public health through collaboration with the Health Portfolio and the NCCs. Given that the program has been active since 2005, the evaluation assessed progress made towards intermediate program outcomes, which include:

• increased availability of knowledge and knowledge products for evidence-informed decision making in public health and improved mechanisms and processes to access knowledge
• partnerships developed with external organizations
• increased use of evidence to inform public health programs, policies and practices.

The intended reach for the program consisted of the following stakeholder groups: National Collaborating Centres, research and education communities, regional and municipal local health authorities, provincial and territorial governments, the Health Portfolio, public health practitioners, policy makers and non-governmental organizations.

2.3.1 Description of the Logic Model^{Appendix 8 Footnote i}

The logic model for the National Collaborating Centres for Public Health Contribution Program, which depicts the program's long term outcome "improved public health programs and policies".

The logic model also identifies the program's immediate and intermediate level outcomes.

The logic model includes the following intermediate outcomes: increased availability of knowledge and knowledge products for evidence informed decision making in public health; increased use of evidence to inform public health programs, policies and practices; partnerships developed with external organizations; and mechanisms and processes to access knowledge. These intermediate level outcomes contribute towards three outcome areas that link to the PHAC Population and Public Health Fund (under which the NCCPH program is funded) including: Research/Knowledge, Capacity Building and Intersectoral Collaboration.

The logic model includes the following immediate level outcomes: increased opportunities for enhancing public health through collaboration with the Health Portfolio and NCCs; knowledge translation: the synthesis of knowledge in existing and emerging areas of public health, and the dissemination, translation and exchange of knowledge across diverse sectors and disciplines in public health; knowledge gap identification: gaps are identified and act as catalysts for applied or new research; and networking: increased collaboration with NCCs occurs among and across public health at all levels.

Lastly, the logic model also describes program activities and outputs which are expected to lead to the intended program outcomes.

Program activities include:

• Program management: NCCPH Program Secretariat, at PHAC, is responsible for managing the contribution agreements established with six host organizations that each house an NCC, promoting and providing advice and guidance to the program, providing secretariat support for the National Advisory Council and supporting the priorities of public health. It also acts as a liaison between PHAC, the Health Portfolio and NCCs.
• The NCCPH Program Secretariat also monitors and evaluates NCC outputs and activities to ensure alignment with priorities of public health, and facilitates alignment of knowledge translation activities between PHAC and the NCCs; the National Advisory Council for the NCCPH.

Program outputs include:

• National Advisory Council materials, meetings with NCCs, priority setting documents, promotion tools for NCCPH program; managed contribution agreements with six host organizations that each house a Centre (at arm's length from PHAC, and within program parameters, each NCC focusses on its own public health priority, develops its own and contributes to a collaborative workplan, disseminates its own knowledge products and builds networks of partners that include regional, national and international subject matter experts, policy makers, practitioners, researchers, academics and others); NCC Collaborative: it includes representation from each NCC and provides a forum for collective action across all NCCs. Supported by the Leads Secretariat.

NCCPH program activities are intended to reach the following audiences: the NCCs; research and education communities; regional and municipal local health authorities; provincial and territorial governments; public health practitioners; policy makers; NGOs and all Canadians.

2.4 Program Alignment and Resources

The program was reflected as part of the Public Health Agency's 2013-2014 Program Alignment Architecture (PAA): Public Health Infrastructure program (PA 1.1) in the Public Health Information and Networks sub-program (SP 1.1.2). The program's financial data for the years covered by this evaluation (2008-2009 through 2013-2014), as available at the time of writing this report, are presented below in Table 1. Overall, the program had a budget of $54.7 million over the past six fiscal years.

Over the five-year period from 2008-2009 to 2012-2013, actual program spending totalled approximately $48 million compared to a budget of $45.9 million. This difference is due to actual contributions spending over the five years exceeding the planned amount by approximately $2.6 million, or 6%, which was funded through the reallocation of lapsed contribution funds from other Agency programs. It should also be noted that the actual spending on salary, operations and maintenance (O&M) and Employee Benefit Plan (EBP) over the five years was below the planned amount by approximately $550 thousand, or 13%, primarily as a result of program understaffing.

Budget 2012 savings measures resulted in a planned reduction of approximately 30% to the program budget at both the Secretariat and Centre levels which will be implemented in 2014-2015.

 

3.0 Evaluation Description

3.1 Evaluation Scope, Approach and Design

The scope of this evaluation includes an assessment of the relevance and performance of the Public Health Agency of Canada's NCCPH program for the period between fall 2008 and fall 2013. The scope excludes KSTE activities conducted by the Public Health Agency or the NCC recipient host organizations (separate from the NCCs). An outcome-based evaluation approach was used for the conduct of the NCCPH evaluation to assess the progress made towards the achievement of expected outcomes. This approach involves determining what outcomes a program helps achieve, and the extent to which the change can be attributed to program activities. This was the second evaluation of the NCCPH program (last evaluation conducted in 2008).

The evaluation addresses requirements in the Treasury Board of Canada's Policy on Evaluation (2009) and considers the five core issues under the two themes of relevance and performance.

In the assessment of relevance, the evaluation examined the continued need for the program and its alignment with federal government and Public Health Agency roles and priorities. In the assessment of program performance, the evaluation examined the progress made towards expected outcomes and the efficiency and economy of the program. Corresponding to each of the core issues, specific questions were developed based on program considerations and these guided the evaluation process.

Data for the evaluation were collected using various methods: a literature review, a document review, case studies of each NCC and key informant interviews. The evaluation also included an international review to examine the role of other federal public health organizations (Australia, England, the Netherlands, New Zealand, and the United States) in KSTE. Following the initial data collection process, the scope of the evaluation was expanded to include an environmental scan to provide information on how the public health landscape (particularly related to the need for KSTE) has changed since program inception. More specifically, this environmental scan would help to identify and document the various organizations involved in KSTE activities in public health in Canada in support of accurately assessing the current need for the NCCPH program, and the most appropriate role for the Public Health Agency in delivering it. An additional round of data collection was conducted within the time frame of the evaluation; however, the evaluation scan that resulted is not exhaustive.

Data were analyzed by triangulating information gathered from the different methods listed above. The use of multiple lines of evidence and triangulation were intended to increase the reliability and credibility of the evaluation findings and conclusions. More specific details on the data collection and analysis methods are provided in Appendix 3.

3.2 Limitations and Mitigation Strategies

Most evaluations face constraints that may have implications for the validity and reliability of evaluation findings and conclusions. The following table outlines the limitations encountered during the implementation of the selected methods for this evaluation. Also noted are the mitigation strategies put in place to ensure that the evaluation findings can be used with confidence to guide program planning and decision making.

 

4.0 Findings

4.1 Relevance: Issue #1 - Continued Need

FINDING #1 - There continues to be a need to strengthen Canada's public health capacity. More specifically, there is an on-going need for effective KSTE mechanisms to enhance evidence-informed decision making in public health in support of programs and policies that protect and promote the well-being of the public.

As noted in the introduction of this report, Canada's public health system's capacity challenges were brought to the forefront in the landmark report issued by Dr. Naylor in response to the SARS crisis in 2003. This report noted that inadequate EIDM in the development of public health programs and policies has contributed to Canada's public health capacity needs. The report outlined several recommendations towards improving Canada's public health system's capacity, including the need to improve knowledge synthesis activities in support of EIDM.^{Endnote 13}

A review of literature published over the last five years reveals an ongoing need for improved EIDM across all levels of public health in Canada. Researchers note that the gap between evidence and accessing/applying it into public health practice to support EIDM continues to grow.^{Endnote 14} While a number of factors contribute to this gap, one factor is the sheer quantity of information that is currently available to public health practitioners. The amount of new information that is released on a daily basis in all public health fields makes it impossible for public health practitioners to consistently remain up-to-date on new evidence-based research and spend an appropriate amount of time on their professional duties.^{Endnote 15} It then follows that keeping current on practice-based evidence is equally challenging for public health practitioners. That the gap between evidence and merging it into practice continues to widen is also partially attributable to the fact that decision-makers frequently lack access to important information. Many public health practitioners do not have the knowledge and/or skills to conduct and critically appraise systematic reviews of the research literature, and many public health organizations lack the infrastructure to support these activities.^{Endnote 16}

In order to help close the gap between evidence and its application in practice (EIDM), there is a documented need in the literature for research to be translated between researchers and users in a practical manner.^{Endnote 17} In addition to knowledge synthesis and translation activities, this may include the creation of partnerships that facilitate the sharing of ideas between knowledge producers and users, enabling them to build on strengths, learn from experiences and best practices, and avoid duplication of efforts. Knowledge translation strategies with the ability to examine, analyze and synthesize evidence-based information have been linked to an increased success in policy decisions with the greatest likelihood of beneficial health effects for the general population.^{Endnote 18}

Feedback from key informants (including NCCPH staff, users of NCCPH products, Public Health Agency senior management, public health experts and members of the Public Health Network Council) supports the findings from the literature review. These key informants consistently noted the ongoing need for improved EIDM in Canada's public health workforce, and the importance of effective KSTE activities in helping to address this need.

4.2 Relevance: Issue #2 - Alignment with Government Priorities

FINDING #2 - Strengthening Canada's capacity in public health is aligned with Government of Canada and Public Health Agency priorities. While not explicitly mentioned as a priority, it is recognized that improved EIDM facilitates timely and effective public health programs and policies that protect and promote the health and well-being of the public.

In public health practice, timely and informed decisions and actions are needed to protect and promote the well-being of the public.^{Endnote 19} Through knowledge translation activities, the NCCPH program was established to provide this timely information, in support of improving public health programs and policies for the public. Over the last five years, improving knowledge transfer in the workforce and protecting the health of Canadians have been identified as priorities of the Government of Canada. The Budget 2012, which focused on jobs and growth in Canada, specifically highlighted the need to strengthen knowledge transfer in Canada's workforce.^{Endnote 20} The Speech from the Throne (2010) noted that "protecting the health and safety of Canadians and their families is a priority of our Government." ^{Endnote 21}

The NCCPH program broadly aligns with Public Health Agency priorities. Over the last five years, various corporate documents (Report on Plans and Priorities, Departmental Performance Report, and Five year strategic plan) have identified the need to improve public health capacity in Canada as a priority. For example, each of the Public Health Agency's Report on Plans and Priorities documents produced between 2006-2013, include the need to address public health capacity as an organizational priority.^{Endnote 22,Endnote 23,Endnote 24,Endnote 25,Endnote 26,Endnote 27,Endnote 28} The 2012-2013 Report on Plans and Priorities specifically identifies "Enhancing Public Health Capacity" as one of four organizational priorities, explaining that enhancing pan-Canadian and Agency capacity will contribute to building a stronger public health system and, in turn, promote health, reduce health inequalities and prevent and mitigate disease and injury. Specific plans noted to support addressing this priority include the following, which align with NCCPH program objectives:

• strengthen the capacity of the public health workforce
• improve public health tools, including the advancement of systematic knowledge sharing and use of best practices in public health.

The Public Health Agency's five year strategic plan (Strategic Horizons 2013-2018) indicates four strategic directions including the need for strengthened public health capacity and science leadership.^{Endnote 29}

Additionally, the Public Health Agency's Corporate Risk Profile 2012-2013 identifies an "inability to plan and respond to public health threats" as a risk, if Canada lacks the appropriately trained workforce, tools, organizational capability, and inter-jurisdictional systems. The NCCPH program is identified as a mitigating action towards minimizing this risk.^{Endnote 30}

4.3 Relevance: Issue #3 - Alignment with Federal Roles and Responsibilities

FINDING #3 - Strengthening Canada's public health capacity through efforts aimed at improving EIDM is an appropriate role for the Government of Canada and the Public Health Agency.

In Canada, public health is a shared responsibility between federal, provincial and territorial governments, but also involves municipal governments, the private sector, non-governmental organizations, health professionals and the public. While provinces and territories have the primary responsibility for their respective public health capacity needs, the consequences of their capacity gaps can affect the public health system as a whole. As such, the Government of Canada may choose to play a supporting role in this situation.

The Department of Health Act, as well as the Public Health Agency of Canada Act, provide the Public Health Agency with the legislative authority for a contribution program designed to support efforts aimed at improving KSTE, in support of EIDM and enhancing public health capacity. The Department of Health Act establishes the Minister of Health's powers, duties and functions relating to health, including "cooperation with provincial authorities with a view to the coordination of efforts made or proposed for preserving and improving public health" and "subject to the Statistics Act, the collection, analysis, interpretation, publication and distribution of information relating to public health".^{Endnote 31} The Public Health Agency of Canada Act established the Public Health Agency for the purpose of assisting the Minister in exercising or performing the Minister's powers, duties and functions in relation to public health.^{Endnote 32} The NCCPH program is directly linked to the functions described above through efforts aimed at addressing Canada's public health capacity needs through improved evidence-informed decision making. The program authorities received to support building a pan-Canadian public health system provide further definition of the Public Health Agency role in this area. Specifically, the Public Health Agency received funding to support national collaborating centre efforts to advance understanding and action on key priority issues in public health.

Additional support for a federal and Public Health Agency role in this area was collected from the literature review, international comparison and from the interviews with key informants. With respect to the literature review, the Naylor Report noted the public health system's collective ability to respond to public health events is limited by the weakest jurisdiction in the chain of the public health system. This led Naylor to conclude that there is a federal responsibility to strengthen public health system capacity. This resulted in a recommendation for the creation of a federal public health agency, with a variety of responsibilities, including knowledge translation.^{Endnote 33} Similarly, the international comparative analysis revealed that although the specific strategy varies by country, respective federal bodies are consistently involved in enhancing public health capacity. Finally, there was a broad consensus among internal and external key informants that addressing Canada's public health capacity needs through efforts aimed at improving EIDM is an appropriate role for the Public Health Agency of Canada.

FINDING #4 - There are a number of organizations, including other Public Health Agency efforts, involved in KSTE to strengthen Canada's public health capacity needs. It is unclear to what extent these various efforts align with or complement the NCCPH program, or each other.

A mapping exercise, or environmental scan, of the public health landscape in Canada was undertaken to document how the landscape has changed since the inception of the NCCPH program. The scope of the scan was deliberately kept broad, so as to include key organizations that contribute to building public health capacity through any activity spanning the spectrum of knowledge generation to knowledge exchange. It did not include an assessment of organizations involved in KSTE related to the specific topic areas of the Centres. It is important to note that the resulting list is not exhaustive, and reflects only those organizations that were identified during the environmental scan within the context of this evaluation.

The environmental scan revealed that a variety of governmental (federal, provincial and territorial and municipal/regional), academic and non-governmental organizations are involved in a range of capacity building activities. It also confirmed that the Public Health Agency has numerous investments in the area of KSTE, both directly and as partner and/or co-funder.

While overall efforts to enhance public health capacity have improved since the NCCPH program was launched in 2005, the extent to which these various efforts align with, or complement, the NCCPH program, or each other, is not clear. More specifically, despite the numerous KSTE efforts within the Public Health Agency, there is currently no KSTE strategy for the Agency in place. This has resulted in a number of KSTE-related activities operating in silos, without an overarching strategy to guide them. In addition, while the NCCs recognize the KSTE efforts underway, and to varying degrees, have made efforts to link with some of these organizations (e.g. all of the NCCs link with the Canadian Public Health Association in support of their annual conference), there does not appear to be a formal mechanism in place for facilitating such collaboration.

While an environmental scan of organizations involved in KSTE related to the specific topic areas of the Centres was not conducted as part of this evaluation, it was identified that capacity in some of these areas has been reduced since the onset of the NCCPH program. For example, based on key informant interviews, the primary source of KSTE work in the area of Aboriginal health is now the NCCAH. There had previously been KSTE capacity within the National Aboriginal Health Organization, founded in 2000 with a goal of providing health programs and a common research base for Canada's diverse Aboriginal populations, however they closed on June 30, 2012. Furthermore, it was noted by a number of key informants that KSTE capacity in Aboriginal health, environmental health, and healthy public policy does not exist internal to the Agency. A summary of KSTE-related organizations and initiatives, identified during the mapping analysis, are summarized below.

Public Health Agency of Canada

In addition to providing support to the KSTE activities of the NCCs under the NCCPH program, the Public Health Agency itself directly carries out the following KSTE-related activities.

• Centre-level KSTE activities - Across the Public Health Agency, programs and centres are actively involved in the production and dissemination of guidelines, tools, and knowledge products via websites, conferences, published research and networks. For example, the Intervention and Best Practices Division develops KSTE tools, workshops and learning modules, and also produces a scientific journal called Chronic Diseases and Injuries in Canada.
• Canadian Best Practices Portal for Chronic Disease - This resource was developed in 2005-2006 and is managed by the Intervention and Best Practices Division of the Centre for Chronic Disease Prevention. The Best Practices Portal for Chronic Disease consolidates multiple sources of trusted and credible information related to chronic disease interventions and tools to support EIDM, in one place, to provide access to comprehensive information to support planning, implementation and evaluation of effective public health programs and policies. The overall objective of the portal is to strengthen evidence-informed policy, practice and programming in chronic disease/injury prevention and healthy living in Canada.
• Public Health Agency Regional Operations - Beginning in 2013-2014, the Regional Operations have been undergoing a transformation that will see them become capacity enablers. Included in this transformation is an intent for Regional Operations to assume a more active role in knowledge development and exchange. This would include identifying knowledge gaps, helping stakeholders articulate and access best practices and contributing to the synthesis and translation of knowledge. In carrying out this role, Regional Operations would support provinces and territories through capacity building efforts and by facilitating their linking with programs and expertise within the Public Health Agency.

In addition to these efforts, the Public Health Agency is also involved as a partner (co-funder) in the following KSTE-related activities.

• Programmatic Grants in Health and Health Equity - In 2011, the Public Health Agency, in partnership with the Canadian Institutes of Health Research and others, began supporting 11 Programmatic Grants in Health and Health Equity. These grants are intended to support two research priorities - pathways to health equity and population health intervention research. The specific objectives of this funding include supporting programmatic research that contributes new knowledge to improve health and health equity at a population level, supporting interdisciplinary collaborations that involve researchers and knowledge-users in public health, and facilitating effective KSTE approaches that enhance the integration and use of new and existing knowledge to inform decision making in public health. This is a five year grant program.
• Applied Public Health Chairs- Also in partnership with the Canadian Institutes of Health Research and others, the Public Health Agency provides grant funding to the Applied Public Health Chairs program. This program has been in place since 2008, and was re-launched in 2013. It strives to increase national capacity for effective research, mentoring, education, knowledge translation and use of public health research evidence. The articulated objectives for this program are to stimulate the application of innovative theories, methods and approaches in research and knowledge translation that promote reciprocal learning within and between countries, and to catalyze interdisciplinary and inter-sectoral collaborations between researchers and knowledge users that contribute to evidence-informed decision making and use of knowledge by public health and other sectors.

In addition to the Public Health Agency's numerous KSTE-related contributions, a number of external organizations are also actively involved in capacity building in support of Canada's public health system. While not all are involved specifically in KSTE, they are included to demonstrate the changing landscape since the inception of the NCCPH program. Organizations identified during the mapping analysis are summarized below.

• Provincial Public Health Agencies (British Columbia Centre for Disease Control (BCCDC), Institut national de santé publique du Québec (INSPQ) and Public Health Ontario) - Three of Canada's provinces have provincial public health agencies in place. While each of these agencies are distinct from one another in some aspects, they all share the broad role of protecting and promoting the health of their province's inhabitants and reducing inequalities in health through the provision of provincial leadership in public health. Similarly, while each agency approaches their role in KSTE in their own unique way, all three of these provincial agencies do participate in KSTE activities, primarily aimed at practitioners within their own province. For both INSPQ (created in 1998) and BCCDC (became an agency in 2001), knowledge translation activities, including the dissemination of best practices and knowledge products, are carried out through the various institute/agency service lines. The Institute also contributes to the management of public health information by launching and operating a collective information infrastructure (Québec Public Health Infocentre) available to all those working in the health system, linked with strategic public health information systems. The KSTE efforts within Public Health Ontario have recently been centralized to a knowledge services group. This knowledge services group also administers the Supporting Research and Program Evaluation, Education and Knowledge Exchange in Public Health program which builds capacity in Ontario's 36 public health units to better enable them to generate, access, exchange and use evidence.
• Canadian Institutes of Health Research - Institute of Population and Public Health.In addition to the granting programs described above, in which the Public Health Agency is involved as a partner, this national granting agency (created in 2000) aims to improve the health of populations and promote health equity in Canada and globally through research and its application to policies, programs, and practice in public health and other sectors. Elements of KSTE, for which the Institute of Population and Public Health provides funding, include:
  • Knowledge synthesis - Funding of primary and secondary research (including evidence reviews and realist reviews).
  • Knowledge translation & exchange - Connecting researchers with decision makers, and linking researchers to KSTE mechanisms.
• Canadian Public Health Association (CPHA) - The CPHA (founded in 1910) is a national, independent, not-for-profit, voluntary association representing public health in Canada, with links to the international public health community. CPHA's mission is to constitute a special national resource in Canada that advocates for the improvement and maintenance of personal and community health according to the public health principles of disease prevention, health promotion and protection and healthy public policy. To help achieve this mission, the CPHA is actively involved in supporting evidence-based decision making. In addition to publishing the Canadian Journal of Public Health, the CPHA also participates in the following KSTE activities:
  • Knowledge generation/synthesis - Initiates, encourages and participates in research directed at the fields of disease prevention, health promotion and healthy public policy and design, develop and implement public health policies, programs and activities.
  • Knowledge exchange - Organizes and leads the annual CPHA conference, bringing together public health practitioners from across Canada to participate in knowledge exchange activities.
• HealthEvidence.org - HealthEvidence.org (launched in 2005), based out of McMaster University in Hamilton, Ontario, is actively involved in contributing to improved EIDM in public health in Canada. This organization aims to make evidence easily available, while also developing organizational and individual capacity for evidence-informed public health decision making. Through their online registry of critically appraised systematic reviews, along with training and consultation services, HealthEvidence.org aims to facilitate searching, interpreting and applying research evidence for policy makers in their local contexts. Specific KSTE-related elements of this organization's work include:
  • Maintaining a registry of critically appraised systematic reviews. This registry provides access to over 3730 quality-rated systematic reviews evaluating the effectiveness of public health interventions.
  • Delivering tailored workshops and presentations (to public health organizations or individual practitioners) addressing the 'how to' of using research evidence in public health decision making.
  • Providing knowledge brokering services, which involves working with individuals, teams, divisions, and organizations to customize and facilitate a locally tailored approach to evidence-informed decision making.
• Networks of Centres of Excellence of Canada - The Networks of Centres of Excellence program was created in 1989. The program is jointly administered by the Canadian Institutes of Health Research (CIHR), the Natural Sciences and Engineering Research Council (NSERC) and the Social Sciences and Humanities Research Council (SSHRC) in partnership with Industry Canada and Health Canada. The program supports large-scale interdisciplinary academic research networks that involve the active participation of non-academic sectors to assist the economic development and well-being of Canadians. Network funding is available for a five-year period with the possibility of renewing for up to two more five-year cycles. Currently, there are 14 networks receiving funding under the Networks of Centres of Excellence program. The Networks cover a broad spectrum of non-health-related topics such as biofuels, new media technologies and understanding the Arctic. Health-related networks address clinical care for the elderly, application of stem cell research, understanding neurodevelopmental disorders, reducing the burden of stroke and arthritis, and research the connection between allergies, genes and the environment.
   
  The program has a KSTE mandate as described in its guidelines, but it has an additional role to focus on the commercialization and job creation approach to innovation and technology. The program's funding model permits it to administer funds through grants in order to facilitate knowledge mobilization and partnerships between companies, government and non-government organizations. It also has a mandate to fund new centres for excellence, and these can potentially shift every 4 to 5 years. The individual centres of excellence provide publications on their research, similar to the model of the individual NCCs. In 2010 the Networks of Centres of Excellence Knowledge Mobilization initiative was introduced to support the networking and knowledge mobilization of a network - these are funded for a four year cycle with a possibility of one renewal of three years.
• Master of Public Health (MPH) programs - In response to numerous reviews and reports citing the need to improve public health capacity in Canada, a number of academic institutions are at various stages of planning/implementing professional master's level training programs in public health (MPH programs). These programs are intended to provide students with a broad mastery of the subject matter and the methods used in the field of public health practice. These programs are primarily designed around the Core Competencies for Public Health in Canada and typically require students to develop the capacity to organize, analyze, interpret and communicate knowledge in an applied manner. It is unclear how the emergence of these programs has contributed to addressing public health capacity needs in Canada. However, it has been suggested through key informant interviews that the MPH graduates who enter the public health workforce represent a receptive capacity for the work of the NCCPH program, in that they have the skills and abilities to use NCC products in support of EIDM.

4.4 Performance: Issue #4 - Achievement of Expected Outcomes (Effectiveness)

4.4.1 To what extent has the NCCPH program contributed to an increased availability of knowledge and knowledge products for evidence-informed decision making in public health?

FINDING #5: The NCCPH program has developed and made accessible knowledge products including evidence reviews, fact sheets, tools and online training modules. These products are perceived as filling identified knowledge gaps in the specific topic areas of the Centres.

While NCCPH program authorities describe knowledge products as "the results of publishing research knowledge in plain language and in accessible formats", there is currently no common definition in place for the program of what specifically constitutes a knowledge product. Further, the description from the program authorities is not broad enough to include products informed by practice-based knowledge (including tacit and experiential), which represent a number of the knowledge products produced by the NCCs. While each Centre is asked to report regularly on their progress towards the development of knowledge products (through semi-annual and annual progress reports), the lack of a formal, agreed-upon definition has resulted in a variety of interpretations being used by the Centres. A further consideration is that knowledge products take on a variety of forms (including evidence reviews, fact sheets, tools, case study summaries and online training modules), each requiring different levels of effort and resource investment. All of these factors have led to a wide range in the number of knowledge products being reported across the NCCs. Examples of knowledge products for each Centre are provided in Appendix 3.

Table 3 below provides an overview of the number of knowledge products developed by the program over the last four full fiscal years. In general (with the exception of 2011-2012) the number of products produced has increased each year.

Across the NCCPH program, the KSTE efforts of the Centres are influenced by target audience needs. On an annual basis, each NCC conducts an environmental scan (which may be informed by surveys of the Centre's target audience, literature reviews and consultations with their Advisory Board) to identify knowledge gaps in the topic area of the Centre. These knowledge gaps then help inform the Centre's work plan for the upcoming year, including which topic areas the knowledge products will focus on. In this way, the products developed by the Centres are relevant and appropriate in that they are being developed in response to identified knowledge gaps. This finding is supported by feedback collected from key informants, in particular, the users of the NCCs.

The specific needs of each NCC's target audience greatly influence the format of the knowledge products that are developed, as well as the means by which they are disseminated. For example, NCCAH has designed their products to ensure that they are culturally sensitive, relatable and appropriate. Similarly, while NCCAH produces a variety of knowledge products, from academic papers to informative DVDs informed by national gatherings, they tailor their deliverables to the needs of the specific audience they are trying to reach. For example, booklets have proven to be an effective dissemination vehicle for the NCCAH to reach community members. In addition, to support their target audience's need for timely information, the NCCID has produced a series of rapid reviews, including ten questions and answers for current topics such as the Middle East Respiratory Syndrome Coronavirus and Influenza H7N9. One of the primary means by which the NCCs make their knowledge products accessible is through online repositories/databases/registries available on NCC websites. These registries are searchable libraries of knowledge products produced by, or reviewed and translated by the NCCs. Other common dissemination channels used across the NCCs include workshops and webinars, newsletter mail outs to list serves, networks and partners, conferences and workshops and social media (Twitter or Facebook).

Through the product dissemination efforts of the NCCs, including Centre websites, registries and published NCC journal articles, there has been broad reach of NCC knowledge products.

This is supported by various lines of evidence including centre-level performance data. Web analytic data collected by the various NCCs reveals that the NCC websites have been visited by target audience members from across the country. In addition, citation analysis of NCCPH products reveals that NCC materials are commonly cited in both academic and grey literature. For example, in the first six months of 2012 alone, citation analysis for only seven of the NCCAH's many products revealed 81 citations of NCCAH resources including 42 citations in academic journals or textbooks and 39 in published grey literature sources. There are also indications that NCC products have had international reach. For example, the NCCHPP has developed methodology for synthesizing knowledge and summarizing relevant literature about healthy public policy. This has been developed into a format that is relevant for policy-makers and decision-makers. These products have been disseminated internationally including to the public health networks of various countries, and the World Health Organization's Evidence Informed Policy-Net group has also recommended the documents through its network. Additionally, a public health documentation centre from the Piemont region in Italy has profiled the work of NCCHPP and translated materials from the NCCHPP to include in their online registry.

In general, there is a high level of credibility associated with the knowledge products developed by the NCCs. Key informants, including Agency staff, Advisory Board members, and users of the individual Centres, spoke of the expertise of the individuals staffed by the Centres and the credibility provided due to the association with the host organizations. More specifically, it was noted by many key informants that the association with a provincial health organization or university provides a certain level of prestige to the NCC; however, of note was the concern expressed regarding the suitability of a non-governmental organization as a host for an NCC. In particular, the sustainability of the International Centre for Infectious Diseases (the host for the NCCID) was questioned. Many key informants also felt that the arms-length relationship between the NCCs and the Public Health Agency elevated the perceived credibility of the NCC products, due to the level of independence the NCCs have from the political environment associated with that of any federal government department/agency.

Challenges

Of note is that the six NCCs use various criteria with respect to the type of material which are included in their online registries. For example, some NCCs include summary statements with each knowledge product posted (explaining what the product is, how the product can be used by public health practitioners, its strengths/weaknesses); however, not all of the NCCs do this. Anecdotal evidence suggests however that public health practitioners prefer to receive information and knowledge in short, concise, summarized formats.

4.4.2 To what extent have the National Collaborating Centres developed partnerships with external organizations?

FINDING #6 - The NCCPH program has formed partnerships with a wide range of organizations, and has improved national networking among public health practitioners. Both the partnerships and networking efforts of the NCCPH program has resulted in reduced duplication of activities nationally and increased knowledge gap identification.

The NCCPH annual progress report template defines partnerships as "joint activities with other organizations in which resources/skills/knowledge are contributed by each partner". This definition is broad, and the lack of specificity has resulted in numerous interpretations being used by the Centres. For example, based on this definition, an activity as small as co-presenting a webinar, or an activity as large as co-partnering with a public health organization to develop a suite of practice-informed tools, could be interpreted as a partnership. Due to this, there is a wide range in the number of partnerships formed by each of the NCCs on an annual basis.

Table 4 below provides an overview of the number of partnerships developed by the program over the last four full fiscal years. This table shows that the number of partnerships reported at the program level have increased annually over the each of the last four full fiscal years. Examples of partnerships for each Centre are provided in Appendix 4.

In general, as noted in the Centre progress reports, all of the NCCs have been successful in engaging with a variety of partnering organizations over the last five year period. For example, the NCCs have engaged local and provincial/territorial-level organizations (including public health units, Regional Health Authorities, and Ministries of Health), other NCCs, NCC host organizations, non-governmental organizations working in the area of public health, Canadian and international public health and knowledge translation organizations, and academic institutions. While the Centres naturally have partnerships at the local level, all Centres have expanded beyond the geographic district of their Centre to encompass organizations located in other regions and provinces.

Based on key informant interviews, it appears that partnerships most commonly revolved around working together on projects or events, and the formal sharing of knowledge and best practices. While some partnerships were initiated based on requests received by the Centres for assistance, others were strategically selected based on their knowledge/skills or the joint ability to maximize resources. In general, partnerships were developed with the end goal of together being able to better address needs related to evidence-informed decision making. The overall benefits associated with NCC partnership activities are discussed in conjunction with the benefits of NCC networking activities (below).

Networking

Although networking is recognized as a key activity of the NCCs, the NCCPH program does not define networking and therefore there are inconsistent interpretations utilized amongst the various Centres. For the purpose of this evaluation, a network was defined as "a system of interconnected individuals, groups or organizations within a specific domain of knowledge that interact socially and share knowledge with each other".^{Endnote 34} The primary distinction between networking efforts and partnerships would be that the latter is more formal in nature and requires that there be a mutual benefit to both parties.

The NCCs support the development and expansion of their networks through approaches such as:

• participating or presenting at public health conferences
• disseminating electronic newsletter (via mail out) to distribution lists
• hosting or participating in online communities of practice
• working through existing networks (including other NCCs) to establish new contacts
• capitalizing on established networks of Advisory Board members.

Through these various networking efforts, the NCCPH program has contributed to the establishment of national networks in the topic areas of the Centres. For example, DialoguePH, a network developed by the NCCMT, has brought together public health practitioners to share their knowledge and experience related to moving research evidence into practice. Similarly, in 2012, the NCCDH launched "Health Equity Clicks", an on-line community of interest for public health practitioners and researchers that supports problem-solving, topical and experiential learning, and resource sharing. The NCCDH moderates the community, facilitates conversations and synthesizes conversations into alternate official language. Focus groups demonstrate that the community increases connections, enhances learning and is applicable to members' work.

At times, the networking activities of the NCCs have contributed to the formation of on-going, sustainable working groups and/or committees. For example, the NCCEH was a catalyst in forming a national Personal Service Working Group (supported by the Public Health Agency), and facilitating this Working Group by connecting stakeholders, identifying gaps, and stimulating knowledge exchange. A major focus was the development of appropriate legislation, regulation and inspection procedures. Once these were established, the NCCEH, while continuing to participate in the Working Group, assumed a less prominent role.

In general, key informants noted a variety of benefits resulting from both the partnering and national networking efforts of the NCCs. Primarily these involved the sharing of knowledge to help reduce duplication of activities nationally and to improve knowledge gap identification in the public health topic area of the Centre. The development of their own networks has also assisted the NCCs in their ability to support the networking efforts of others. For example, the NCCs receive ongoing requests from public health practitioners across Canada for assistance or direction. When possible, the NCCs endeavour to respond to these requests directly; however, the outcome often involves linking the practitioner in need with other practitioners across the country working in similar areas. This type of service would not be possible without an in-depth knowledge of the key players in their field.

Challenges

Despite the apparent success of the NCCs in developing partnerships and networks, of note would be the minimal guidance to Centres on this topic, including the lack of criteria to inform the development and/or selection of partners. While the Advisory Council previously provided guidance on networking and partnerships to each NCC through feedback letters, over the last couple years in the Council's absence this has not occurred. This is unlike the Centres' annual work planning process, where the topic areas of focus are informed through environmental scans or discussions with stakeholders. This minimal guidance may have resulted in gaps in the partnerships developed. For example, while some Centres have engaged the Pan-Canadian Public Health Network Council, there is no formal approach in place to linking with and/or partnering with it, despite program planning documents indicating a desire for such a linkage.^{Endnote 35} Representation on the Centres' Advisory Boards reflect these comments, as only three individuals from the Public Health Network Council currently sit on Centre Advisory Boards.

4.4.3 To what extent has the National Collaborating Centres for Public Health program increased the use of evidence to inform public health programs, policies and practices?

FINDING #7 - Based on available evidence, the NCCPH program has made progress towards increasing the use of evidence to inform public health practice. However, the ability to demonstrate this varies by Centre.

In general, there appears to be a high level of awareness of the NCCPH program among the target audiences of the Centres. Output data collected by the individual Centres, provided in table 5, reveals the number of visitors to NCCPH websites. While this output data doesn't necessarily reflect use of NCCPH products, it is useful as a proxy measure for assessing target audience awareness of the Centres, and also helps support the notion that members of the target audience value the NCCPH as a resource and are keeping up to date with Centre products (by visiting websites). Of note is the fact that these output numbers are continuing to trend upwards each year.

Members of the NCCPH target audience are not only aware of the Centres and their products; they are also using these resources in their professional practice. Evidence of NCCPH program impact was collected through interviews with NCC users and staff. In general, key informants from all of the Centres provided positive feedback on the usefulness of the products, and were able to provide an explanation as to how they had used them to implement positive change at their organization. For example, they frequently noted making use of NCC products, networks and expertise to gain information and knowledge to inform their practice. Despite the generally positive feedback collected on NCC products, mixed feedback was received from users of the NCCID, and other key informants who spoke of their interactions with this Centre. In addition to this inconsistent feedback on NCCID's work, the majority of informants had minimal knowledge of the products the Centre had produced and were not able to comment on the perceived usefulness of the products or to document the impact that the work had.

According to key informants, and as documented in Centre-level performance data, there are a variety of ways in which NCCPH products have been impactful. For example, NCCPH products have been used in the development of guidelines; have been incorporated into academic curriculums; have facilitated the sharing of new knowledge and raised awareness around key issues; and have helped increase public health practitioner capacity in EIDM. Through these various examples of use and impact, NCCPH products have been incorporated into decision-making processes to inform public health programs and policies. Multiple examples of use across the various types of impact were collected for each Centre; however, specific examples of NCCPH product impact are summarized below.

• NCCPH products used in the development of guidelines: As a result of the NCCEH's substantial initial investigative work in assessing the health risks of, and management strategies for, personal service establishments (tattoo parlours, piercing and decorative surgical establishments), the Centre was invited to review proposed health guidelines for authorities in British Columbia, Nova Scotia, Northwest Territories and Saskatchewan which were later codified into legislation and/or guidelines. Similarly, the NCCAH's Framework Web of Being has been used by the Society of Obstetricians and Gynecologists of Canada to inform their new Guidelines for Health Professionals Working with First Nations, Inuit, and Metis.
• NCCPH products incorporated into academic curriculums: Despite the growing popularity of health impact assessments in Canada, few training opportunities are available. In collaboration with educational technologists, academics and an expert pioneer in health impact assessments, the NCCHPP has developed two self-paced courses on health impact assessments, now accessible throughout the country. One is a 12-hour continuing education course open to all practitioners and is recognized for continuing education credits. The other, a 45-hour University course offered through the University of Montréal's School of Public Health, is a graduate level credited course available to all Canadian students, including students from other universities. Both courses are offered in separate English and French sessions. These health impact assessment online courses have contributed to increased capacity for graduate level community health and preventive medicine programs and Schools of Public Health, and have increased capacity for professional development in public health.
• NCCPH products facilitating knowledge sharing: The NCCDH recently held knowledge translation forums in Manitoba. The central forum, conducted in June 2013, aimed to inform participants about effective community engagement, inter-sectoral approaches and organizational mechanisms to address social determinants of health. It was attended by 111 public health practitioners, researchers, decision makers, students and community leaders from throughout Manitoba. Feedback was received from 53 participants, with all agreeing (58% very strongly agreed and 42% strongly agreed) that the event strengthened their commitment to advancing health equity and addressing the social determinants of health, and 77% indicating they were likely to integrate what they had learned into their work. Complementary training was provided to the Public Health Branch of Manitoba Health, and similar forums have been held in Nunavut, Newfoundland/Labrador, PEI and NS.
• NCCPH products raising awareness of key issues: The NCCID has worked with colleagues and partners in public health on the persistent problem of Antimicrobial Resistance/ Antimicrobial Use. When other organizations were unable to lead the collaboration, NCCID stepped in to chair the Communications and Education Task Group on Antimicrobial Resistance which initiated the first-ever Antibiotic Awareness Day in 2010, in keeping with similar campaigns in Europe. The Antibiotic Awareness Day has since developed into an Antibiotic Awareness Week, coordinated by NCCID and picked up by numerous groups each successive year. The NCCID-led group developed a website, posters as well as prescription pad for doctors who need an alternative "prescription" for patients with suspected viral infections who are seeking antibiotics which are not medically indicated. Over 1,000 prescription pads have been distributed, and NCCID receives regular requests for renewed supplies for physicians using them. Interest in the website (antibioticawareness.ca) and its products has grown exponentially, with more than 70,018 visits to the website in the last 2 years.
   
  In 2009 the NCCAH partnered with UNICEF Canada on Aboriginal Children's Health: Leaving No Child Behind - Canadian Supplement to the State of the World's Children 2009. The report was designed to address the lack of awareness and concern about the equity gap in health between Aboriginal children and other Canadian children by bringing evidence-based awareness of the gaps that exist. In referring to the success of the partnership UNICEF representatives indicated that the Centre's ability in forging relationships enabled it to combine mainstream academic knowledge and practice with Aboriginal knowledge. According to the UNICEF partners, it is unlikely that the project would have gone ahead without the support of the NCCAH. This report has been widely disseminated and academics, advocates, and policy makers have and continue to engage with the report. As of 2011, the UNICEF had distributed 65,000 copies of the report world-wide.
• NCCPH products have helped increase public health practitioner capacity in EIDM: In 2010, NCCMT began developing online learning modules to help practitioners overcome barriers to practicing evidence-informed public health. The modules support skill development at each of the seven steps of the evidence-informed public health process. Users' perceived self-efficacy and knowledge of content, as of one year ago, have been assessed prior to starting the module (pre-test) and again after completion (post-test). Certificates of completion are provided upon request. Results indicate a statistically significant increase in knowledge following completion of the Research Designs module and a statistically significant increase in self-efficacy following the completion of the EIDM and Critical Appraisal of Intervention Studies modules. So far, the modules have been accessed by over 4,000 users, with 1,476 people earning a certificate. The modules are used in staff training and have also been incorporated into curricula at several post-secondary institutions across Canada.

Additional examples of impact for each Centre, collected from key informant interviews, are included in Appendix 5.

Challenges:

The achievement of intermediate and long term outcomes requires that sufficient time elapse in order for change to occur and to be documented. While the NCCPH program was established in 2005, not all contributions agreements were signed until the end of 2006. In addition, all of the Centres report having gone through a two to three year developmental phase before they were able to become fully operational. Evaluative evidence, including Centre-level performance data and feedback from key informants, demonstrates that the NCCPH program has made progress towards increasing the use of evidence to inform public health practice.

While the NCCs support one another through networking efforts and the dissemination of one another's knowledge products, there is inconsistent use of one another's knowledge products among the NCCs. This is particularly interesting given that in some cases, for some products, standards and/or tools were developed and/or promoted by one of the Centres, but not adopted by others. For example, the NCCMT registry includes a knowledge synthesis tool for appraising public health interventions. And while this tool and others like it have been shared with the other Centres, uptake has been inconsistent. This lack of leveraging of in-program expertise is further discussed in section 4.5.2.

Despite the positive evidence on impact and use of the NCCs, key informant interviews with senior managers and staff at the Public Health Agency identified that there does not appear to be strategic usage of NCCPH products across the Agency, outside of the few areas within the Public Health Agency that have partnered with the NCCs in the past five years. A summary of NCCPH/PHAC interactions for 2012-2013 is included in Appendix 6, and this issue is further discussed in section 4.5.2.

While the NCCs have contributed to an increased use of evidence to inform public health practice, this has occurred primarily in the specific topic areas of the NCCs (notwithstanding the broad and cross cutting nature of some of the Centres). Due to the design of the program along specific themes, whereby Centres work primarily in their specific topic area, it was noted by key informants that as a whole, the NCCPH program lacks an ability to respond to emerging public health priority areas. As a comparison, England's Department of Health, which also supports KSTE and EIDM for public health in that jurisdiction, has mechanisms to ensure that knowledge is available for emerging priority issues. For example, the Department of Health in England has established the Public Health Research Consortium, an academic-sector initiative to increase evidence for a variety of public health interventions. Through this consortium, the Department is able to direct the choice of research projects according to priority policy needs of the Department. This is discussed further in the summary of international approaches to KSTE (Appendix 7).

4.5 Performance: Issue #5 - Demonstration of Economy and Efficiency

The Treasury Board of Canada's Policy on Evaluation (2009) and guidance document, Assessing Program Resource Utilization When Evaluating Federal Programs (2013), defines the demonstration of economy and efficiency as an assessment of resource utilization in relation to the production of outputs and progress toward expected outcomes. This assessment is based on the assumption that departments have standardized performance measurement systems and that financial systems link information about program costs to specific inputs, activities, outputs and expected results.

The NCCPH program lacks a standardized performance measurement system, resulting in an inability to compare performance across Centres. Similarly, the data structure of the detailed financial information provided for the NCCPH program did not facilitate the assessment of whether program outputs were produced efficiently, or whether expected outcomes were produced economically. Considering these issues, the evaluation provided observations on economy and efficiency based on findings from the literature review, document review, key informant interviews, Centre-level performance measurement information and available relevant financial data.

In addition, the findings below provide observations on the adequacy and use of performance measurement information to support economical and efficient program delivery and evaluation.

4.5.1 Observations on activities linked to economy and efficiency

FINDING #8 - All Centres have employed various methods to increase their efficiency.

Key informants, including NCC staff and Agency staff, highlighted a number of efficient practices that were utilized by the Centres. These practices varied across each of the NCCs. Typical ways in which the Centres improved efficiency are summarized below.

• Utilizing student placements - Student placements, both undergraduate and graduate, were utilized by most Centres. Through these placements, the students have supported and contributed to the activities of the NCCs by supporting specific projects, and providing advice/guidance. This relationship is mutually beneficial as the Centres have benefitted from the increased capacity in the Centre while the students are on board, and the students have gained valuable experience and an introduction to the public health landscape during their time working with the Centres.
• Minimizing travel expenses - All of the Centres noted that travel is kept to a minimum in an attempt to avoid the associated costs and instead they have used alternative means to share information and connect with others. For example, many of the Centres reported using webinars as an alternate to face-to-face presentations. In addition, whenever possible, travel-related expenses were cost-shared with partner organizations. For example, when an NCC visits a partner to deliver a workshop, the various costs associated with the activity are split between the two organizations.
• Maximizing target audience participation at NCC-led events - Many of the Centres spoke of their efforts to increase participation at NCC-led events. For example, when an organization requests an NCCMT-led workshop, the Centre typically responds by requesting that this organization also identify other organizations in that region, province or territory to participate. In this way, the Centre increases the reach of each activity, reduces travel costs (multiple organizations involved in cost sharing the event), and limits the need to return to the same area multiple times to present to each of the organizations individually.
• Leveraging additional funding - While this does not represent a typical way of working, a number of Centres have successfully acquired additional funding on occasion through grants or contracts with public health or research organizations (including the Public Health Agency). Additional funds have also been leveraged through cost-sharing activities with stakeholder and partner groups participating in NCC-led events.
• Placing staff in satellite offices - Although the satellite-model was only noted by the NCCDH, it is worthwhile highlighting as it is a particularly interesting example of efficiency. Rather than housing all of the Centre staff within the NCCDH-proper, some Centre staff are placed across various regions in the country (Manitoba, Ontario, BC and Québec) to provide the Centre with national representation. Key informants from the NCCDH noted that this national representation limits the need for the Centre staff located in the NCCDH proper to travel, and also enhances the national networking efforts of the Centre contributing to effective relationships and regional penetration.
• Maximizing the host/NCC relationship - A number of key informants noted that program efficiencies have occurred through the NCC/host relationship. Typical efficiencies noted included working through the existing networks of the host organization to disseminate knowledge products, accessing expertise from within the host organization, reduced fees associated with accommodations, and minimal, if any, administration fees.

4.5.2 Observations on operational challenges related to economy and efficiency

FINDING #9 - There are inefficiencies at the program level which have resulted in potential duplication and missed opportunities for greater coordination of efforts.

Between the Public Health Agency and the NCCs

Key informants noted that the NCCPH program design has contributed to several inefficiencies. Specifically, key informants attributed these inefficiencies (discussed below) to the interpretation among both Public Health Agency and NCC staff about the requirements associated with contribution agreements (re: the relationship between funder and recipient). While the provision of funds through a contribution necessitates an arm's length relationship between the Public Health Agency and the NCCs, this has generally been interpreted as 'hands off'. While the contribution agreement stipulates that the funder shall not derive any direct benefit from this agreement, in order to improve F/P/T collaboration in public health, there would be benefits to involving the Agency.

Despite the potential benefits resulting from improved Public Health Agency/NCC collaboration, and despite the fact that public health practitioners work within the Agency, the majority of key informants noted that the Public Health Agency is not seen as a target audience for the products of the NCCs. With the exception of contractual relationships between the Agency and the NCCs, which have occurred in the past, there has generally been limited collaboration between the Public Health Agency and most NCCs (Appendix 6). While some program areas have been engaged by one or more Centres to occasionally work in partnership, there does not appear to have been a strategic effort towards working with the Public Health Agency across the NCCs. Furthermore, while some Public Health Agency staff contact the NCCs for information and expertise, there also does not appear to be a strategic effort on behalf of the Agency to work with the NCCs. This is in contrast to the first immediate outcome in the NCCPH program logic model which states "Increased opportunities for enhancing public health through collaboration with the Health Portfolio and NCCs" (see Section 2.3.1). The lack of Public Health Agency representation on NCC Advisory Boards further illustrates the detached nature of this relationship. For example, a review of the membership of the six NCC Advisory Boards revealed that only one Public Health Agency employee currently participates on an advisory board. Of note, in the most recent evaluation of the NCCPH program (2008-2009), key informants suggested that to increase connectivity between the NCCs and the health portfolio, there would be benefits to having Public Health Agency representation on relevant NCC Advisory Boards.^{Endnote 36} This would be done with the understanding that Agency representation on Advisory Boards would not be for the purpose of the Agency directing the work of the NCCs, but rather to facilitate improved F/P/T collaboration and connectivity.

The NCC Program Secretariat was tasked with promoting the development of linkages between the Public Health Agency and the NCCs; however, management and administration of the program has consumed the secretariat's resources (which were reduced by one FTE in 2012-2013 as part of Budget 2012 savings measures). Consequently, key informants noted that the secretariat has not been able to carry out all of its responsibilities related to the development of linkages between the recipients and the Agency.

Other inefficiencies, possibly as a result of the 'hands off' relationship, that exists between the Public Health Agency and the NCCPH program include the lack of: joint work-planning that could potentially exist between the NCCs and relevant areas within the Public Health Agency, and awareness among Public Health Agency staff of the NCCs, their activities, networks and products. There is also limited knowledge exchange occurring between the two parties to support knowledge gap identification activities.

As a result of these issues, there is a risk that duplication of activities might occur between Public Health Agency staff and NCC staff working in similar areas (e.g. infectious disease). A similar risk was noted in the 2008 Formative Evaluation of the NCCPH program.^{Endnote 37} At that time, evaluators questioned whether, due to the lack of any linkages with the NCCs, PHAC would establish parallel internal public health KSTE capacities and resources for their own program staff, ultimately leading to duplication of effort.

Inefficiencies due to NCCs operating independently

Each NCC is mandated to operate within a defined area of public health, and is responsible for conducting its own planning, managing its resources and responding to the needs of its unique audience. However, as noted in the 2008 Formative Evaluation of the NCCPH program, there is a tendency for the NCCs to operate in silos rather than as a collective. The present evaluation identified that this continues to be the case.

Key informants noted that the independent nature of the Centres may be attributable to a combination of how the NCCs were initially established and the lack of centralized guidance currently available to the Centres. When the NCCs were established, they were set up to operate independently from one another. Each of the NCC's workplans and progress reports were reviewed separately by the Advisory Council and Program Secretariat. Key informants, including NCC leads and Public Health Agency representatives, noted that messaging related to collaboration among the NCCs has emerged in recent years; however, the program infrastructure has not changed to support this. For example, no additional resources have been provided to enable improved collaboration, nor have any systematic processes been introduced to facilitate it. Since 2010, the contribution agreements signed with each NCC have required a joint collaboration workplan funded by a contribution from each NCC.

The recent disbandment of the Advisory Council and the resulting limited centralized governance and direction available to the Centres has further impacted the NCCs in their ability to respond to the messaging related to collaboration among the Centres. As previously noted, included in the roles and responsibilities of the Advisory Council was the intent for the Council to advise on approaches to promote the coordination of NCC work plans. In the absence of the Advisory Council, such advice has not been available to the Centres. The Program Secretariat was also originally tasked with providing central direction and coordination of NCC activity and for promoting collaboration among the NCCs.^{Endnote 38,Endnote 39} However, as noted in the 2008 Formative Evaluation, not all of the envisioned Secretariat roles have been possible to carry out through the arm's length arrangement allowed under the contribution agreement funding model.

The lack of a coordinated centralized approach to NCC activities has resulted in the following inefficiencies: lack of formal joint work planning among the NCCs, lack of a formal KSTE strategy in place for the program to guide the Centres' work (based presumably on best practices related to KSTE), and inconsistent uptake of each NCCs' products among the other NCCs.

As a result of these issues, there is a risk that duplication of activities might occur among the NCCs. For example, given that some of the NCC topic areas are broad and cross-cutting with other Centres' content areas, and given that there has been limited joint work planning, there is a risk that NCCs may engage in overlapping efforts. Similarly, the lack of a formal KSTE strategy to guide the Centres has resulted in each NCC having to establish many processes and tools independently (rather than collaboratively) to guide their KSTE work. An example of the various KSTE strategies used across the Centres is the reference libraries, or product repositories hosted on each of the NCC websites. Rather than one central repository for all NCCPH work, with a common look, feel and structure, each of these repositories is constructed differently (some less navigable than others). As previously dicussed in section 4.4.1, some repositories include products (e.g. tools, evidence reviews) with summary statements - to facilitate knowledge translation - written by the NCC to help the intended user understand the strengths, limitations and potential use(s) of the products. Conversely, other repositories include no such summary statements. Finally, that there is inconsistent uptake among the Centre's of one another's products contributes to duplication of activities. For example, as noted in section 4.4.2, the NCCMT registry includes, among many others, a knowledge synthesis tool for appraising public health interventions. However, not all of the other NCCs have incorporated this tool into their methodology. As a result each Centre has invested efforts into finding their own processes. It is important to note that the KSTE efforts of each NCC vary to some extent based on the unique needs of the target audience and the topic area of the Centre. As a result, it may not always be appropriate to apply the same KSTE methodologies across all NCCs.

Despite the current lack of coordinated, centralized direction for the NCCPH program, credit is given to the individual NCC leads for the work they have done to support collaboration among the Centres. This includes the establishment and maintenance of the NCC Leads Secretariat, which helps to support the identification of shared projects addressing cross-cutting topics. At their fall 2013 meeting, it was the intent of the Leads Secretariat to include joint-work planning in their meeting agenda. Similarly, individual NCCs have made efforts to work more closely with other NCCs. For example, NCCDH typically tries to copartner (with another NCC) at all of the regional forums that they host. Key informants noted that greater efficiency and economy would result from improved collaboration and synergies of this sort.

4.5.3 Observations on the Adequacy and Use of Performance Measurement Data

FINDING #10 - While each Centre collects, analyzes and uses performance data, there is limited performance measurement in place at the program level.

The NCCPH Program Secretariat is responsible for overall program accountability. This role includes performance monitoring and evaluation of Centre outputs and activities, as noted in section 2.2.2 of this report. While the Secretariat has been actively involved in monitoring each Centre's progress towards completing planned deliverables (through review of annual NCC work plans and progress reports), there have been minimal additional performance measurement activities (collection, analysis and use of performance information) in place at the overall program level. Similarly no formal performance measurement strategy is in place to support program decision making, or to guide the collection of Centre-level performance information. Of further concern, is that while the NCC work plans and progress reports support the monitoring of NCC deliverables, these documents are completed with inconsistent levels of detail by the Centres. This is partially attributable to the broad requirements and definitions included in these templates. As a result, there is limited performance measurement data available at the program level; however, there are indications of progress by the Secretariat towards strengthening the performance measurement activities of the program, including revisions to the performance measurement strategy.

Despite the lack of formal performance measurement activities at the program level, each Centre actively participates in the collection and analysis of performance data to inform their practice. Various performance measurement activities undertaken by the Centres include stakeholder satisfaction surveys on specific published knowledge products, feedback surveys on workshops or conferences, and analysis of visits to the NCC internet web sites ('web analytics'). In addition, some NCCs have commissioned consultants to produce evaluation reports to provide a more in depth analysis of progress towards stated outcomes. It is understood that all such NCC documents have been shared with the Program Secretariat to supplement the information contained in work plans and progress reports.  

5.0 Conclusions

5.1 Relevance Conclusions

There continues to be a need to strengthen Canada's public health capacity. More specifically, there is an on-going need for effective KSTE mechanisms to enhance evidence-informed decision making in public health in support of programs and policies that protect and promote the well-being of the public.

The objectives of the NCCPH program broadly align with Government of Canada and Public Health Agency priorities. The NCCPH program recipients' activities are intended to enhance evidence-informed decision making in public health practice. While not explicitly mentioned as a priority, it is recognized that improved evidence-informed decision making facilitates timely and effective public health programs and policies that protect and promote the health and well-being of the public. This outcome continues to be reflected as a priority in a variety of parliamentary and corporate strategic reports.

Addressing Canada's public health capacity needs is an appropriate role for the Government of Canada and the Public Health Agency. The Department of Health Act, as well as the Public Health Agency of Canada Act, provide the Public Health Agency with the legislative authority to take on this role. Through the Agency's support of the NCCs' efforts aimed at improving evidence-informed decision making in public health, the Agency is carrying out this role.

A number of other organizations are involved in KSTE to address public health capacity. In addition, the Public Health Agency participates in a number of KSTE efforts, most of which have been established since NCCPH program inception. It is unclear to what extent these various efforts, particularly within the Agency, align with or complement the NCCPH program, or each other. As a result, it is unclear if there is a continued need for the NCCPH program as currently designed.

5.2 Performance Conclusions

Based on the information that was available, including performance data and key informant interviews, the NCCPH program has made progress towards increasing the use of evidence to inform public health practice; however progress varies by Centre. In general, progress has been accomplished through the KSTE work of the NCCs, including their networking and partnership activities and the production of relevant knowledge products. The networking and partnership activities of the NCCs have supported knowledge gap identification and contributed to decreased duplication of public health efforts nationally in the topic areas of the Centres. The knowledge products developed and disseminated by the NCCs have been in response to gaps identified through their networking activities. The dissemination of these credible synthesized knowledge products and tools have supported public health practitioners across the country in their efforts to incorporate evidence into their practice. The design of the program along specific themes, however, has restricted the ability of the Centres to respond to emerging public health priorities outside of the topics areas of the NCCs.

A number of efficient practices are regularly employed by each of the NCCs; however, several inefficiencies exist at the program level. The lack of centralized guidance and coordination for the Centres (including on issues such as approach to joint work-planning, approach to KSTE and partnership activities) has led to potential duplication of activities among the NCCs and missed opportunities for collaboration.

In addition, limited awareness and engagement of the program and its products among Public Health Agency staff and the Public Health Network (facilitated by the NCCPH Program Secretariat) has resulted in potential duplication of activities between the Public Health Agency and the NCCs. There is a need for the Program Secretariat to enhance program-level performance measurement in support of program decision making.

 

6.0 Recommendations

Recommendation 1

In consultation with key stakeholders, including P/Ts, and considering the Public Health Agency's various efforts in KSTE, determine the appropriate role for the Agency in addressing the public health system's KSTE needs.

There is evidence of an on-going need to strengthen Canada's public health capacity through effective KSTE mechanisms in support of improved evidence-informed decision making. However, the Public Health Agency is currently involved in a number of KSTE efforts, most of which have been established since NCCPH program inception. While there is still a role for the Public Health Agency in responding to this need, it is not clear that it is through the NCCPH program as currently designed. To best support the public health system, the Agency's efforts in this area should be coordinated, responsive to emerging public health priorities, and not duplicate or overlap the work of others.

Recommendation 2

Review NCCPH program mandate and determine if there is continued alignment with the articulated Agency role. If necessary, revise as appropriate.

Once the Public Health Agency's role in supporting the public health system's KSTE needs has been articulated, the mandate of the NCCPH program should be reviewed to ensure it aligns with the Agency role, and that it is complementary with other Agency efforts in this area.

Recommendation 3

Adjust program design to reflect renewed mandate.

Following the review of the program mandate, the design of the program should be examined and adjusted (if necessary) to most effectively carry out this mandate. Should the program design remain consistent with its current format, several adjustments are suggested based on evaluation findings. The NCCPH program should:

1. Enhance collaboration among the Centres, between the Centres and Agency, and with other players.
2. Maximize program flexibility to respond to emerging priorities.
3. Ensure ongoing strategic guidance and coordination are in place for the program.

Recommendation 4

Ensure performance measurement is implemented and used to assist in programmatic decision making.

Irrespective of the resulting program mandate and design, it is essential that a performance measurement system be implemented that would assist in programmatic decision making.

The current NCCPH program lacks a formal performance measurement strategy to guide the collection of performance data. While each Centre collects, analyzes and uses performance data, there is limited performance measurement in place at the program level. Centralized guidance to direct performance measurement activities of the Centres is required, including common definitions and consistent templates. Improvements to the performance measurement approach would enhance the ability to assess program success on a continuous basis.

 

Appendix 1 - Mandates of the NCCs

National Collaborating Centre for Aboriginal Health

The NCCAH supports a renewed public health system in Canada that is inclusive and respectful of First Nations, Inuit and Métis peoples. Using a holistic, co-ordinated and strengths-based approach to health, the NCCAH fosters links between evidence, knowledge, practice and policy while advancing self-determination and Indigenous knowledge in support of optimal health and well-being.

National Collaborating Centre for Determinants of Health

The NCCDH uses knowledge translation methods to enable public health organizations and practitioners to advance social determinants of health and reduce health disparities through public health practice. The NCCDH seeks to build public health practitioner knowledge, skill and leadership capacity; strengthen public health sector networks; and encourage researchers to pursue research that integrates social determinants of health and addresses identified evidence gaps. NCCDH products and services are knowledge syntheses/summaries, reports and fact sheets; workshops/webinars, presentations and conferences; and engaging stakeholders through online networks, social media, collaborative learning, Audio/Visual and web-enabled methodologies. The NCCDH's current focus is to assess and translate evidence regarding organizational change and decision-maker capacity to effect program and policy development.

National Collaborating Centres for Environmental Health

The NCCEH focuses on the health risks and benefits attendant to contact with the physical environment and identifies evidence-based interventions to optimize the benefits and minimize the harms of that contact. Its function is to synthesize, translate, and exchange knowledge for environmental health practitioners and policy-makers; identify gaps in research and practice knowledge; and build capacity through networks of practitioners, policy-makers, and researchers. NCCEH regularly contracts environmental scans and evaluations to inform its work plan.

National Collaborating Centre for Healthy Public Policy

The NCCHPP seeks to increase the expertise of public health actors across Canada in healthy public policy through the development, sharing and use of knowledge. This includes collaborating with different stakeholders to advance healthy public policies to improve the health of Canadians. NCCHPP accomplishes its goals by synthesizing knowledge to make it accessible, developing and participating in networks, and identifying research gaps in order to create resources and links along with practitioners, decision-makers, researchers and others.

National Collaborating Centre for Infectious Disease

The mission of the NCCID is to protect the health of Canadians by facilitating the use of evidence and emerging research on infectious diseases to inform public health programs and policy. The NCCID also aims to encourage collaborative responses by relevant stakeholders, including public health policy makers, practitioners, and researchers, to public health issues and needs in the area of infectious diseases.

National Collaborating Centre for Methods and Tools

The NCCMT supports and collaborates with individuals, organizations, and communities to share existing and/or develop new knowledge related to methods and tools that support the development, implementation, and evaluation of public health programs and policies. In particular, the NCCMT is responsible for knowledge translation, knowledge gap identification, and networking to promote the use of methods and tools to support evidence-informed decision making in public health practice.

 

Appendix 2 - Evaluation Description

Evaluation Scope

The scope of the evaluation included an assessment of the relevance and performance of the NCCPH program activities, situated in the Knowledge Integration and Mobilization Unit, Centre for Public Health Capacity Development of the Public Health Agency of Canada. The scope excluded knowledge transfer, synthesis and exchange activities conducted by the Host. The timeframe of this evaluation was from fall 2008 to fall 2013.

Evaluation Issues

The specific evaluation questions used in this evaluation were based on the five core issues prescribed in the Treasury Board of Canada's Policy on Evaluation (2009). These are noted in the table below. Corresponding to each of the core issues, evaluation questions were tailored to the program and guided the evaluation process.

Data Collection and Analysis Methods

Evaluators collected and analyzed data from multiple sources. The lines of evidence used in this evaluation included the following: literature review, document review, international comparative review, key informant interviews, case study visits to each NCC, and a mapping exercise/environmental scan of organizations involved in KSTE activities for public health in Canada. Each of these lines of evidence are briefly summarized below.

Literature Review

The literature review examined both academic and grey literature, including peer-reviewed journal articles, government and web-based documents produced between 2008 and 2013. The databases used to conduct the search included PubMed and Google Scholar. Specific search criteria included the key words "knowledge translation", "health practitioner needs", "knowledge translation activities", "knowledge translation program", "capacity building program", "knowledge translation "best practice", "research practice gap", "evidence-informed decision making" and "public health". A total of 93 articles were retrieved and reviewed of which 12 were found to be relevant.

Document Review

This consisted of a review of approximately 380 corporate and program documents on all aspects of the program's activities. The evaluation team coordinated with the program to facilitate the acquisition of program documents.

Summary of International Approaches to KSTE

The evaluation also included a review of international approaches to KSTE in public health. This review examined the role of other federal public health organizations in the following countries:

• Australia
• England
• The Netherlands
• New Zealand
• United States

In support of this review, relevant federal organizations in each of these countries were identified and descriptive material for each was collected and reviewed from organizational website pages. In addition, telephone interviews were conducted with representatives from the following organizations: Research Australia, Public Health England, the Netherlands Organisation for Health Research and Development, and the Council of State and Territorial Epidemiologists (United States).

Case studies

A key line of evidence in this evaluation was the material collected from six on-site case studies, one for each National Collaborating Centre. Each of these case studies consisted of interviews with at least seven key informants, representing different capacities within the centre. As a minimum for each NCC, interviews were conducted with the following range of key informants: the scientific/academic lead, Centre manager, Centre staff (minimum of two), a representative from the host organization, and two advisory board members (including the Chair). In addition, two collaborators (users) for each Centre (recommended by the leads) were interviewed. These individuals were interviewed as representatives from organizations that have worked closely with the NCCs and consumed their products and services.

Mapping exercise/Environmental scan

A mapping exercise, or environmental scan, of the public health landscape in Canada was undertaken to document how the landscape has changed since the inception of the NCCPH program. To inform this review, organizational mandates and activities related to KSTE were collected and summarized from website/document reviews, and additional information was captured through key informant interviews. A total of 12 key informant interviews were conducted in support of this scan with representatives from the following organizations:

• Public Health Agency of Canada
  • Health Promotion and Chronic Disease Prevention Branch (two key informants)
  • Infectious Disease Prevention and Control Branch
• Institute of Population and Public Health, Canadian Institutes of Health Research
• Canadian Public Health Association
• health evidence.org
• Network Centres of Excellence - Knowledge Mobilization (NCE-KM) initiative: Co-funded by the CIHR, the Natural Sciences and Engineering Research Council and the Social Sciences and Humanities Research Council
• Master of Public Health schools or equivalent programs from the following academic institutions: University of Alberta, Dalhousie University, Université de Montréal, University of Toronto, University of Waterloo.

Key Informant Interviews

In addition to the interviews conducted as part of the international comparison, case studies and mapping exercise, additional key informant interviews were also conducted with internal and external program stakeholders. Table 7, below, summarizes the numbers of interviews conducted by stakeholder group.

Assessment of Economy and Efficiency

To assess economy and efficiency, the evaluation:

• conducted a resource allocation review at the program level through an assessment of available financial data (spending levels, trends, issues, etc.) including interviews with OCFO to explain variances
• conducted key informant interviews regarding how resources were used, the effectiveness of program management, and the efficiency of business processes
• obtained views from key stakeholders regarding leveraging opportunities
• examined the conduct, sufficiency and use of performance information to enhance economy and efficiency.

Data Analysis

Data were analyzed by triangulating information gathered from the various sources and methods listed above, using NVivo for qualitative analysis (document review). This also included: systematic compilation, review and summarization of data to illustrate key findings; thematic analysis and coding of qualitative data; and comparative analysis of data from disparate sources to validate summary findings.

 

Appendix 3 - Examples of knowledge products

National Collaborating Centre for Aboriginal Health

The NCCAH partnered with the First Nations Health Authority to develop four child health and wellness booklets for parents and caregivers across British Columbia: Growing up Healthy; Parents as First Teachers; Fatherhood is Forever; and Family Connections. Approximately 25,000 hard copies of the booklets were disseminated across British Columbia to Friendship Centres, Aboriginal Head-Starts on and off-reserve, First Nations Health Directors, and Maternal Child Health Programs. Since the targeted mail-out, the NCCAH and First Nations Health Authority have distributed an additional 8,000 hard copies by request. These booklets are also available for download on the NCCAH website.

National Collaborating Centre for Determinants of Health

Among the NCCDH's syntheses and summaries is a systematic literature review to assess effectiveness of inter-sectoral action involving public health to advance health equity. The review resulted in a summary and full report; a peer-reviewed article; several webinars/workshops/oral presentations; and communication of a knowledge gap to CIHR's Institute of Population and Public Health and other researchers. The review methodology was rated highly (8 out of 10) by Health Evidence and the article, based on quality and online downloads/coverage, is ranked in the 97th quintile of the publisher's articles.

National Collaborating Centres for Environmental Health

The NCCEH developed two databases including one on treatment technology applicable to small drinking water systems and one on provincial legislation and governance structures around small drinking water systems. The treatment technology database was based on a synthesis of available engineering and health literature and was validated by typical knowledge users, in this case public health inspectors and engineers responsible for the oversight of small water systems. As the NCCEH treatment technology database was being developed, the United States Environmental Protection Agency, in response to a similar need there, developed a more comprehensive and ongoing database, which is now linked from the NCCEH website. The NCCEH continues to provide scientific and technical information to our target group of drinking water inspectors through webinars (two in the last year) and links to report and article citations, including an article by NCCEH/BCCDC staff on environmental and administrative factors associated with the imposition of boil water orders in rural water systems. The regularly updated legislative review covers all Canadian jurisdictions and references all relevant acts and regulations which form the basis for health protection activities around threats from water, food, air and other environmental exposures. Provincial/territorial and regional jurisdictions use the review in developing their own legislation.

National Collaborating Centre for Healthy Public Policy

The online course: "Introduction to Health Impact Assessment of Public Policies" aims to help participants develop and improve their competencies for leading a health impact assessment process related to public policies, and to do this with partners from different sectors. It is intended for participants who wish to pursue the health impact assessments of public policies.

National Collaborating Centre for Infectious Disease

The NCCID developed 12 Evidence Reviews related to the Pandemic H1N1 outbreak. The purpose of the Pandemic H1N1 Evidence Review series is to highlight lessons learned and covered topics such as Swine Surveillance for Public Health Planning, Antiviral Targeting and Distribution Strategies during the 2009 Influenza A (H1N1) Pandemic, 2009 Influenza A/H1N1 Mass Vaccination Strategy: a Multinational Comparison and the Impact of Pandemic Influenza A (H1N1) on Laboratory Services.

National Collaborating Centre for Methods and Tools

The Registry of Methods and Tools is a searchable, online collection of methods (processes) and tools (instruments) that support knowledge translation and evidence-informed practice in public health. Progress reports reveal that, as of end of fiscal year 2012-2013, the Registry had a total of 148 methods and tools. For each entry in the Registry a summary statement has been written and is available in both English and French. Visits to the site have increased significantly from 1,000 visits in 2009-2010 to 100,000 visits in 2012-2013.

 

Appendix 4 - Examples of partnership activities

National Collaborating Centre for Aboriginal Health

The NCCAH's long-standing partnership with the Canadian Paediatric Society has led to the development of 17 train-the-trainer curriculum resources and training DVDs for paediatric residents specific to Aboriginal child and youth health. The Canadian Paediatric Society is in the process of modifying this curriculum for all health practitioners using an online module. The online module will be completed by March 31, 2014 and will be accredited by the Canadian Paediatric Society and Memorial University. Note that this curriculum was developed in 2009 and expanded in 2011. To date, 305 paediatric residents have been trained in the 17 schools across Canada. A commentary on the curriculum development process and preliminary evaluation findings was published in Paediatrics and Child Health August/September 2012.

National Collaborating Centre for Determinants of Health

The NCCDH co-hosted a conference with CIHR's Institute for Population and Public Health (in partnership with NCCHPP, CIHR's Institute of Aboriginal People's Health, and the Canadian Institute for Health Information) that brought together researchers and practitioners. Event objectives were to examine approaches and tools; strengthen integration of evidence into planning, implementation and evaluation; and identify opportunities for sustained knowledge translation and linkages. As well, as progress towards the defined objectives, the conference resulted in proceedings and four cases studies, strengthened relationships with partners and other contributors, and positive profile/promotion of the NCCDH to a national audience.

National Collaborating Centres for Environmental Health

In response to three fatalities in Saskatoon, the NCCEH under contract with Health Canada, and in partnership with colleagues at the University of Montreal, developed a model to assess the lowest levels and shortest duration of exposure to carbon monoxide likely to do serious harm to the elderly, and to persons with various chronic medical conditions who are typically housed in such facilities. The model, launched in 2013, demonstrated that anemia and emphysema are important factors which would lead long-term care residents to have symptoms, which could possibly lead to death, when exposed to carbon monoxide concentrations below those which would trigger a conventional carbon monoxide monitor. The NCCEH is involved in translating the model into new monitor technology and in evaluating the implementation of appropriately protective carbon monoxide management practices in facilities in both Saskatchewan and British Columbia.

National Collaborating Centre for Healthy Public Policy

As part of the Coalitions Linking Action and Science for Prevention initiative, NCCHPP partnered with the Urban Public Health Network, the public health units of Montreal, Toronto, Peel, Fraser, Vancouver Island, and Vancouver Coastal Health, the Heart and Stroke Foundation of Canada, and the Canadian Institute of Planners. From this project, NCCHPP developed among others "Health Authorities and the Built Environment: Action to Influence Public Policies".

National Collaborating Centre for Infectious Disease

The NCCID provided expert advice and reviewed knowledge products for the Communications and Education Task Group on Antimicrobial Resistancegroup. This included partners from academia, the World Health Organization, federal and provincial organizations. The purpose of the partnership was to improve focus and direction for Canada's antimicrobial resistance work, enhance networking, and build working relationships with interested organizations.

National Collaborating Centre for Methods and Tools

The NCCMT partners with multiple organizations, regional health authorities, provinces and territories to plan and implement workshops that build individual and organizational capacity related to evidence-informed decision making. The planning of these workshops involves individuals from all public health disciplines whose roles within their organization range from management to policy- and program development to front line practitioners. Partnerships with individuals and organizations/groups from across the country have resulted in many ongoing initiatives related to EIDM in the field.

 

Appendix 5 - Examples of impacts

National Collaborating Centre for Aboriginal Health

The NCCAH has received numerous requests in 2013-2014 to use and adapt resources from organizations such as Healthy Child Manitoba, Dieticians of Canada, the Northern Ontario School of Medicine, Frontline Health, and the Society of Obstetricians and Gynecologists of Canada. These materials were used to inform organization-specific material, including curriculum, databases, websites and booklets for dissemination. For example, the Society of Obstetricians and Gynecologists of Canada has incorporated the NCCAH's Framework Web of Being to inform their new Guidelines for Health Professionals Working with First Nations, Inuit, and Metis. In addition to this, ten universities/colleges have requested permission to incorporate NCCAH resources into curriculum in 2013-2014 and another ten universities have requested multiple hard copies of NCCAH materials in the same fiscal.

National Collaborating Centre for Determinants of Health

NCCDH started a Population Health Status Reporting Initiative in late 2011 to "improve methods to produce population health status reports that better illuminate health inequities." A component of this initiative was a national learning circle of practitioners and academics that were engaged in knowledge exchange and synthesis activities. Capital Health (Halifax) served as an applied practice site. Evidence and knowledge was disseminated to the NCCDH's audience through nine synthesis documents, four videos and numerous events. Learning circle participants reported benefit in terms of the opportunities provided to discuss challenges, as well as establishing networks. Capital Health reported extreme value noting that the exercise added legitimacy and helped to gain access to 'state of the art' practices and experts. Public health practitioners reported that they had gained new knowledge (strongly agreed 42%, agreed 33%).

National Collaborating Centres for Environmental Health

In 2012, the NCCEH consulted with the Canadian Institute of Public Health Inspectors and received their support to develop an online course that caters to the needs of public health inspectors. This was an opportunity to develop expertise and provide accessible continuing education to new and experienced public health inspectors across Canada. The NCCEH/BCCDC Risk-Based Inspection of Food Premises Online Course offers a comprehensive approach to the inspection of food premises that is applicable across all provinces and territories. It focuses on the systematic identification, assessment, and control of risk factors that contribute to food-borne illness. The content of the course is based, in part, on an in-person course that was developed by the BCCDC and the British Columbia Regional Health Authorities. The self-directed course is delivered over 17 weeks and can be completed in 20 hours. As of January 2014, over 100 participants have enrolled.

National Collaborating Centre for Healthy Public Policy

While the NCCHPP's primary targets are frontline public health practitioners and decision makers, some documents published by the NCCHPP are used directly by policy and decision makers. According to key informants from a recent evaluation carried out by Sogemap Inc., an external consultant contracted by the NCCHPP, many of the Centre's documents, especially those relating to traffic calming and the built environment, have been used in the development of policies, including municipal regulations (bylaws). In some cases NCCHPP documents have served as input for final decisions or for doing advocacy work on traffic calming and built environment with other decision makers.

National Collaborating Centre for Infectious Disease

NCCID's knowledge translation, networks and partnerships in the area of HIV and sexually transmitted infections (also called STBBIs) led to the development of a suite of resources based on an identified need for systematic outreach to vulnerable populations. The Outreach Guide, the first resource, was distributed to and picked up by more than 500 public health personnel since 2010. A second edition was published two years after the first release, based on feedback and suggestions from users. A series of consultations which was attended by more than 94 people from every province and territory, including frontline workers, outreach program managers and community-based and local/regional public health unit managers, led to the development of an Outreach Checklist, for HIV and sexually transmitted infection public health programs to help with start-up and programmatic evaluation. The Outreach Checklist has been developed into an app, for use in the field. NCCID next worked with public health program staff to develop a website that creates a forum for sharing program evaluation indicators and discussions. The Outreach Program Exchange Network, was shared with target audiences in 2012 and 2013 and was re-launched with significant improvements in March, 2014.

National Collaborating Centre for Methods and Tools

The NCCMT strives to build capacity in evidence-informed decision making in public health by offering workshops and other training events across Canada, often in conjunction with established public health-related conferences and events. Since 2007, a total of 3,859 participants have attended 78 workshops on topics such as Evidence-Informed Decision-Making in Public Health, Critical Appraisal of Research Evidence and Organizational Change for Use of Evidence in Decision-Making. Evaluations reveal that participants value the opportunity to learn about the methods and tools and to practise using the resources presented in the workshops.

 

Appendix 6 - Public Health Agency/NCCPH Interactions

The tables below provide a high-level summary of the interactions between the National Collaborating Centres and the Public Health Agency (organized by NCC) over the past four years, as they were submitted by each Centre to the NCCPH program secretariat. Each table depicts, chronologically, a brief description of the activities by branch for each NCC.

 

 

 

 

 

Source: NCCPH Program Secretariat

 

Appendix 7 - Summary of international approaches to KSTE

As part of the evaluation of the NCCPH program, a review of international approaches to KSTE in public health was undertaken in order to examine the role of other federal public health organizations in the following countries: Australia, England, the Netherlands, New Zealand, and the United States. In support of this review, relevant federal organizations in each of these countries were identified and descriptive material for each was collected and reviewed from organizational website pages. In addition, telephone interviews were conducted with representatives from the following organizations: Research Australia, Public Health England, the Netherlands Organisation for Health Research and Development, and the Council of State and Territorial Epidemiologists (United States).

In general, the KSTE activities of the countries reviewed are most commonly lead by their national health research organizations, comparable to the Canadian Institutes for Health Research. Each of these organizations appear to have activities aimed at mobilizing the results of health research to inform practice, however the only system that somewhat resembles the National Collaborating Centres for Public Health program is the Knowledge and Intelligence Teams in Public Health England, although this initiative is in an early stage of implementation.

None of the other four countries had permanently-funded KSTE organizations focused on supporting public health practise on specific issues of concern, although the Netherlands does have a time-limited program that seeks to match academic researchers with regional public health delivery organizations in that country. It should be noted that the United States does have health issue-specific organizations that undertake research and knowledge translation on a very large scale (e.g. Centers for Disease Control and Prevention and National Institutes of Health) that not only serve information needs in that country but are also consulted by public health practitioners around the world. That said, none of these organizations by themselves are concentrated solely on KSTE to build practioner capacity for public health.

Australia

In Australia, the primary responsibility for healthcare provision and public health lies with the state and territory health departments under the leadership of the national Department of Health and Ageing; this mirrors the federated system in Canada. The Department's portfolio of agencies includes two key organizations that support KSTE activities for public health:

• The National Health and Medical Research Council is Australia's leading health research granting body that exists to support the creation of new knowledge, accelerate KSTE, and aid the development of capacity for research and application of knowledge in the health sector. Among other granting programs focused on knowledge translation, the Council offers five-year 'Centres of Research Excellence' grants in clinical, health services and population health research that aims to closely link research and its translation into practice. The Council has recently created a 'Research Translation Faculty' comprising over 2,800 researchers who will contribute to the identification of gaps between health research, policy and practice in fourteen health issues of national concern identified in the Council's current strategic plan. Lastly, the Council funds the Australasian Cochrane Centre which is part of the international Cochrane Collaboration. This relationship includes free access to the Cochrane Library for all residents of Australia and is therefore available to the public health sector as a source of knowledge to inform practice.^{Appendix 8 Footnote ii}
• The Australian National Preventative Health Agency was created in 2011 to act as a national coordinating body to develop preventative health policies and programs in collaboration with state and territorial governments, health promotion organizations, industry and primary healthcare providers. KSTE is incorporated into the strategic goals of the Agency, especially in the management of knowledge to inform policies and practice. The Agency currently acts as a centre of knowledge generation and translation, guided by a series of multi-stakeholder committees, on the topics of healthy weights and obesity, tobacco control, alcohol and preventative medicine. Knowledge dissemination activities include posting resources online, holding national symposia on preventative health research and organizing annual awards recognizing leadership in preventative health.

In addition to these two organizations, the Australian Institute of Health and Welfare, an independent statistical agency, monitors and produces reports on the status of a variety of health and social welfare topics. The Institute aims to produce timely information that is relevant to policy-makers and stakeholders at the local, state, territorial and national levels as well as to the broad community of researchers. The Institute also works with health and social services providers to promote data standards to facilitate the reporting of comparable information from these organizations across the country.

England

Public Health England was created in April 2013 as an executive agency of the Department of Health, England. This agency, which has regulatory powers to support carrying out its mandate, has an annual budget of approximately $790 million Canadian. The mandate of Public Health England includes two aspects centred on KSTE and EIDM: i) to undertake research and analyses on public health problems, and ii) to share information and expertise with municipalities, the private sector and the health care system in order to inform decisions that may impact the public's health. To support KSTE activities, Public Health England has a Chief Knowledge Officer directorate and is creating a centralized 'Data and knowledge gateway' web portal for public access to health information and evidence.^{Appendix 8 Footnote iii}

The organizational structure of Public Health England incorporates a number of topic-based KSTE 'intelligence networks', each addressing a specific health topic in areas such as child and maternal health, learning disabilities, health impact assessment, obesity, cancer and end-of-life care. In addition, eight regionally-based Knowledge and Intelligence Teams provide assistance to local public health units in accessing and using knowledge. Both groups support knowledge exchange and knowledge gap identification.^{Appendix 8 Footnote iv} This approach to organizing KSTE in England is much more centralized than the Canadian model, reflecting the context of a nationally-mandated public health service that is delivered through municipal government structures.

England's Department of Health also supports other important KSTE mechanisms such as:

• The National Institute for Health and Care Excellence is an arm's length agency that is responsible for developing a broad array of evidence-based guidance and standards for clinical health care, social services and public health. Topics for study and development of guidance are identified via a stakeholder identification process and selected at the ministerial level. To disseminate information to practitioners and policy-makers, the Institute maintains web portals for published guidance as well as complementary evidence-based knowledge products.^{Appendix 8 Footnote v}
• A Public Health Research Consortium among senior researchers from 11 institutions was created as an academic-sector initiative to increase evidence for a variety of public health interventions, especially on those addressing socio-economic inequalities in health. The Consortium is funded by the Department of Health Policy Research Programme which gives direction to the choice of research projects according to departmemtal policy priorities.
• The National Institute for Health Research is the leading national health and health care research granting agency funded through the Department of Health that aims to support research relevant to current needs. The Institute includes five United Kingdom Clinical Research Collaboration Public Health Research Centres of Excellence^{Appendix 8 Footnote vi} in the areas of complex interventions, KSTE, inequalities, tobacco control, diet and physical activity. In addition, the Institute initiated a funding program called Collaborations for Leadership in Applied Health Research and Care that creates regional partnerships between academic centres and district health services organizations. These partnerships exist to develop research that can be applied to priority issues as well as build the capacity of practitioners to access and use research information.^{Appendix 8 Footnote vii}

The Netherlands

Health issues in the Netherlands are addressed by the Ministry of Health, Welfare and Sport. Public health research and KSTE is conducted by the following key organizations in the health ministry portfolio:

• The National Institute for Public Health and the Environment is the leading national public health organization. The Institute has an important KSTE role in undertaking primary research as well as analysing information from other sources to develop knowledge products. Many of the Institute's subject-matter experts are cross-appointed with academic centres across the country. The Institute disseminates knowledge to public health professionals, e.g. via the 'Healthy Living Helpdesk', though specific web portals and by sharing information with health education organizations.^{Appendix 8 Footnote viii}
• The Netherlands Institute for Social Research develops evidence to inform policies and conducts evaluations of existing government social policies. Research areas include well-being, use of care services, healthy ageing, educational achievement, work and income, social inclusion and integration of minorities. The Institute maintains databases that offer a significant national evidence resource on social issues.

The Netherlands Organisation for Health Research and Development, or 'ZonMw' is the national health and healthcare research granting institute which seeks to promote innovations that address current challenges. One specific KSTE effort for public health of note is the Academic Collaborative Centres for Public Health program, created to encourage academic research that would meet current evidence needs of practitioners.Appendix 8 Footnote ix The program supports regional partnerships between community health services, academic centres and local policy makers to develop knowledge on priority public health themes within areas such as infectious disease, youth healthcare, health promotion and epidemiology; a national-scale partnership on environmental health has also emerged. The regional basis of the partnerships is intended to ensure that there is flexibility to respond to specific practitioner needs and to build a common understanding of the roles and capacities of the participating organizations, i.e. creating a platform for collaborative research (rather than simply 'translating 'existing academic research for practitioner use). Funding is time-limited, and ZonMw expects interest from regional or national stakeholders to sustain the partnerships.

New Zealand

In New Zealand the overarching health authority is the New Zealand Ministry of Health which funds 20 District Health Boards across the country to deliver health care services as well as fulfills public health promotion and protection roles. Although there does not appear to be any national KSTE centre or organization focussed on public health, the following groups have a KSTE role:

• Different units within the Ministry of Health review and publish knowledge products to guide District Health boards, other stakeholders and the public on priority topics.^{Appendix 8 Footnote x}
• The Health Research Council of New Zealand, an agency in the Minister of Health's portfolio, consults broadly to set health research funding priorities that aim to meet national needs. The Council has taken a KSTE approach through communicating and encouraging the use of research findings, especially via requirements for research findings to be relevant to practice in the short or medium term and the publication of research summaries.^{Appendix 8 Footnote xi}
• The Health Promotion Agency, another health portfolio organization, publishes fact sheets and research reports on priority topics, e.g. alcohol, immunization and mental health.
• There are several university-based health research centres such as the Tomaiora Maori Health Research Unit and the National Institute for Health Innovation at the University of Auckland and the Centre for Public Health Research at Massey University.^{Appendix 8 Footnote xii} These organizations respectively address issues of Aboriginal health, population health, environmental health. They each publish research findings and host seminars and conferences to share knowledge and foster collaboration with both local and international groups.

It should be noted that all these institutions incorporate programs and activities to specifically address aboriginal health issues in New Zealand. As part of this focus, the Health Research Council of New Zealand signed a trilateral letter of intent with the Canadian Institutes of Health Research and the National Health and Medical Research Council of Australia "to improve research capacity in Indigenous peoples' health" in 2012.^{Appendix 8 Footnote xiii}

United States of America

The US Department of Health and Human Services co-ordinates health issues in the United States. Within the Department's portfolio, there are many leading agencies involved in KSTE for the public health field in specific domains, including:

• The Centers for Disease Control and Prevention is the primary national public health protection organization that undertakes a range of activities including national level surveillance, emergency preparedness and disease prevention. The Centers for Disease Control and Prevention regularly collaborates with a spectrum of national public health stakeholder organizations.^{Appendix 8 Footnote xiv} All branches of the organization are involved to some extent in KSTE. For example, the National Center for Chronic Disease Prevention and Health Promotion created a KSTE framework and EIDM tool.^{Appendix 8 Footnote xv} The Office of the Associate Director for Science provides a KSTE model for application of science throughout the whole organization that complements the evaluation process.^{Appendix 8 Footnote xvi} The CDC also funds the Council of State and Territorial Epidemiologists, a non-government organization that promotes the use of epidemiologic data for public health as well as supporting the practice of epidemiology across the country. The Council members undertake KSTE activities by means of a wide array of topic-specifc committees that may engage in a variety of activities such as networking, disseminating information and providing training opportunities to build capacity for applied epidemiology. Topics are set according to CDC priorities which are informed over time by needs communicated from the state-level members.
• National Institutes of Health is the senior medical research agency and is comprised of 27 Institutes and Centres. Broad direction for health research priorities are determined through mechanisms such as biennial reporting to Congress and public consultations. All Institutes and Centres undertake KSTE activities; of particular note, the National Library of Medicine provides the PubMed online database which is a globally-important source of peer-reviewed knowledge which covers all health disciplines.
• The Indian Health Service, responsible for providing health care and public health services to American Indian and Alaskan Native communities, maintains a publically-accessible database on evidence-based 'best practices, promising practices, local efforts, policies and resources'.^{Appendix 8 Footnote xvii} The purpose of this tool is to share information between practitioners and with stakeholders to improve decision making and program implementation.
• The Substance Abuse and Mental Health Services Administration is the national agency addressing behavioural health. Two KSTE efforts of note focus on capacity building of practitioners: an online continuing education program for substance abuse treatment professionals and an evidence-based practice network.^{Appendix 8 Footnote xviii}

As the global reach of the PubMed database demonstrates, the scale of KSTE efforts in the United States are very large and diverse and have a global profile and reach. Given the large scale of these institutions, public health practitioners and decision-makers in the United States may still face the challenge of distilling a large volume of information in order to identify evidence that is relevant to their needs.

 

Appendix 8 - References

Aluttis, C., K. Michelsen, et al. (2013). Review of Public Health Capacity in the EU. Luxembourg, European Commission Directorate General for Health and Consumers: 168.

Armstrong, R., E. Waters, M. Dobbins, J. Lavis, M. Petticrew, and R. Christensen. Knowledge translation strategies for facilitating evidence-informed public health decision making among managers and policy-makers (Protocol). The Cochrane Library:1-14.

Benzies, K., and T. Weiden. (2002). Knowledge translation: Final report from the workshop held at the University of Calgary. Canadian Institute of Child Health:1-41.

Bos, V., and J. van Kammen. (2007). Knowledge Synthesis: A Guide. ZonMw and NIGZ.

Bosch, M., E. Tavender, P. Bragge, R. Gruen, and S. Green. (2012). How to define 'best practice' for use in Knowledge Translation research: a practical, stepped and interactive process. Journal of Evaluation in Clinical Practice:1-6.

Brownson, R. C., E. A. Baker, T. L. Leet, K. N. Gillespie, and W. R. True. (2010). Evidence-Based Public Health. Oxford: Oxford University Press.

Brownson, R. C., J. E. Fielding, and C. M. Maylahn. (2009). Evidence-Based Public Health: A Fundamental Concept for Public Health Practice. Annual Review of Public Health 30:175-201.

Brunham, R. C. (2012). BC Centre for Disease Control Report - 2012. BC Centre for Disease Control.

Choi, B. C. K. (2005). Understanding the basic principles of knowledge translation. Journal of Epidemiology and Community Health 59:93.

Ciliska, D., H. Thomas, et al. (2012). An Introduction to Evidence-Informed Public Health and A Compendium of Critical Appraisal Tools for Public Health Practice. Hamilton, National Collaborating Centre for Methods and Tools (NCCMT): 27.

Corelissen, E., C. Mitton, and S. Sheps. (2011). Knowledge translation in the discourse of professional practice. International Journal of Evidence-Based Healthcare 9:184-188.

Council, N. H. a. M. R. (2011). Drafts regarding integrating research and health care with responses.

Curren, J. A., J. M. Grimshaw, J. A. Hayden, and B. Campbell. (2011). Knowledge translation research: the science of moving research into policy and practice. Journal of Continuing Education in the Health Professions 31:174-180.

Curren, J. A., J. M. Grimshaw, et al. (2011). "Knowledge translation research: the science of moving research into policy and practice." Journal of Continuing Education in the Health Professions 31(3): 174-180.

Foley, E. (2013). Assessing the wider benefits arising from university-based research. Research Australia.

Fyfe, T., T. Hampe, C. Hardy, D. Bentham, M. MacLeod, and M. Mogus. (2007). Knowledge Synthesis, Translation and Exchange (KSTE) Cheat Sheet. University of Northern British Columbia.

Green, L. W., J. M. Ottoson, C. Garcia, and R. A. Hiatt. (2009). Diffusion Theory and Knowledge Dissemination, Utilization, and Integration in Public Health. Annual Review of Public Health 30.

Grimshaw, J. M., M. P. Eccles, J. N. Lavis, S. J. Hills, and J. E. Squires. (2012). Knowledge translation of research findings. Implementation Science 7:1-17.

Jadad, A. R., and R. B. Haynes. (1998). The Cochrane Collaboration - Advances and Challenges in Improving Evidence-Based Decision Making. Medical Decision Making 18:2-9.

LaRocca, R., J. Yost, M. Dobbins, D. Ciliska, and M. Butt. (2012). The effectiveness of knowledge translation strategies used in public health: a systematic review. BMC Public Health 12:1-15.

Lomas, J. (2000a). Connecting Research and Policy. Canadian Journal of Policy Research Spring 2000:140-144.

Lomas, J. (2000b). Using 'Linkage and Exchange' to move research into policy at a Canadian foundation. Health Affairs 19:236-240.

Medlar, B., D. Mowat, E. D. Ruggiero, and J. Frank. (2006). Introducing the National Collaborating Centres for Public Health. Canadian Medical Association Journal 175:493-494.

Metzler, M. J., and G. A. Metz. (2010). Analyzing the Barriers and Supports of Knowledge Translation Using the PEO Model. Canadian Journal of Occupational Therapy 77:151-158.

Naylor, D. (2003). Learning from SARS: Renewal of Public Health in Canada. Health Canada.

Nutley, S. M. 2007. Using evidence: how research can inform public services. Bristol: Policy Press.

Nutley, S. M. (2007). Using evidence: how research can inform public services. Bristol, Policy Press.

Nuyens, Y., and M. A. D. Lansang. (2006). Knowledge translation: linking the past to the future. Bulletin of the World Health Organization 84:590.

Organization, W. H. (2006). Bridging the "Know-Do" Gap Meeting on Knowledge Translation in Global Health. World Health Organization.

Peirson, L., D. Ciliska, M. Dobbins, and D. Mowat. (2012). Building capacity for evidence informed decision making in public health: a case study of organizational change. BMC Public Health 12:1-13.

Tetroe, J. (2007). Knowledge Translation at the Canadian Institutes of Health Research: A Primer. Focus:8.

Wilson, K. M., T. J. Brady, and C. Lesesne. (2011). An Organizing Framework for Translation in Public Health: the Knowledge to Action Framework. Preventing Chronic Disease 8:1-7.

 

List of Acronyms

BCCDC

British Columbia Centre for Disease Control

CIHR

Canadian Institutes of Health Research

CPHA

Canadian Public Health Association

EBP

Employee Benefit Plan

EIDM

Evidence-informed decision making

HSIB

Health Security Infrastructure Branch

INSPQ

Institut national de santé publique du Québec

KSTE

Knowledge synthesis, translation and exchange

KT

Knowledge translation

MPH

Master of Public Health

NCC

National Collaborating Centre

NCCAH

National Collaborating Centre for Aboriginal Health

NCCDH

National Collaborating Centre for Determinants of Health

NCCEH

National Collaborating Centre for Environmental Health

NCCHPP

National Collaborating Centre for Healthy Public Policy

NCCID

National Collaborating Centre for Infectious Diseases

NCCMT

National Collaborating Centre for Methods and Tools

NCCPH

National Collaborating Centres for Public Health

O&M

Operations and Maintenance

PHAC

Public Health Agency of Canada

P/T

Provinces and Territories

RMAF

Results-based Management and Accountability Framework

 

Footnotes

Endnotes