Evaluation of the National Collaborating Centres for Public Health Program 2018-19 to 2022-23

Prepared by the Office of Audit and Evaluation
Public Health Agency of Canada
March 2024

Table of contents

• List of acronyms
• Executive summary
• Program description
• Origin of the National Collaborating Centres for Public Health Program
• Evaluation scope and approach
• Evaluation findings
• Conclusions and recommendations
• Management Response and Action Plan - Evaluation of the National Collaborating Centres for Public Health Program, 2018–19 to 2022–23
• Appendix A: Short profiles of the six National Collaborating Centres
• Appendix B: NCCPH logic models
• Appendix C: NCCPH Timeline 2005 to 2023
• Appendix D: Data Collection and Analysis Methods
• Appendix E: NCC web site statistics highlights
• Appendix F: Key examples of NCC COVID-19 response activities
• Appendix G: NCCPH summary funding 2018-19 to 2022-23
• End notes

List of acronyms

AMR

antimicrobial resistance

CPHO

Chief Public Health Officer

COVID-19

Coronavirus disease

EIDM

evidence-informed decision making

HiAP

Health in All Policies

NCC

National Collaborating Centre

NCCDH

National Collaborating Centre for Determinants of Health

NCCEH

National Collaborating Centre for Environmental Health

NCCHPP

National Collaborating Centre for Healthy Public Policy

NCCIH

National Collaborating Centre for Indigenous Health

NCCID

National Collaborating Centre for Infectious Diseases

NCCMT

National Collaborating Centre for Methods and Tools

NCCPH

National Collaborating Centres for Public Health

OCSO

Office of the Chief Science Officer

PHAC

Public Health Agency of Canada

PHN

Pan-Canadian Public Health Network

PMQ

Performance Measurement Questionnaire

SARS

Severe acute respiratory syndrome

STBBIs

Sexually transmitted and blood-borne infections

Executive summary

Background

The National Collaborating Centres for Public Health (NCCPH) Program was created in 2005 alongside the Public Health Agency of Canada (PHAC) and the Pan-Canadian Public Health Network (PHN) in order to support evidence-informed decision making at all levels of Canada's public health system. The aim of the six National Collaborating Centres (NCCs) is to fulfill this mission by identifying knowledge gaps, fostering collaborative networks, and creating and disseminating knowledge translation resources and services.

Each NCC specializes in a different priority area of public health practice, but they share the same model. Each is hosted in a university or provincial public health agency that is supported with contribution agreement funding from PHAC, led and staffed by subject-matter experts, and guided by an external advisory board.

Since 2017, the Program has been managed by the Office of the Chief Science Officer (OCSO). The NCCs each continue to receive $974K annually in core funding from PHAC, an amount set in 2015. Additional short-term project-based funding also supplemented the core budget.

Conclusions and recommendations

Since the 2018 evaluation, the NCCs have continued to perform well in delivering on their three core functions and leveraging their strengths and connections to support a wide audience, including: front-line practitioners and service providers, regional, provincial and territorial public health decision-makers, non-governmental and academic organizations, and PHAC itself. This included pivoting from their established work plans during COVID-19 to leverage their strengths, expertise and networks to support the planning, response, mitigation and recovery of PHAC and other public health actors across Canada.

In addition, the NCCs have continued to produce a wide array of relevant knowledge translation products and activities over the last five years in response to identified knowledge gaps and have continued to collaborate with many public health practitioners and entities across Canada, including PHAC. Many examples of NCC contributions were noted by interviewees, including leadership on issues like health equity, reconciliation with Indigenous peoples and climate change. They have also increasingly collaborated with PHAC units, notably on the Chief Public Health Officer's (CPHO) annual reports. The work of the NCCs has been facilitated by their maturity, expertise, credibility within organizations across jurisdictions, and adaptability to new public health priorities. The NCC model is a key factor in promoting these strengths. This includes the Contribution Agreement mechanism that provides structure to the PHAC-NCC relationship and flexibility to address emerging or changing priorities.

Despite the achievements of the NCCs and strength of the Program model, the NCCs have been facing increasing challenges in meeting the expectations of all key program beneficiaries without supplementary funding from PHAC and external funders. Relying more on short-term supplementary funding affects both annual work plan commitments and the ability to address long-term capacity gaps. As noted in the 2018 evaluation, this situation has led to uncertainty regarding PHAC's expectations for this long-standing Program, i.e., what constitutes success in fulfilling its mission.

The PHAC-NCC relationship, facilitated by the Office of the Chief Science Officer (OCSO), is mutually supportive. The OCSO coordinated meetings between the NCCs and PHAC senior leadership and managers, and organized collaborative work between PHAC units and individual NCCs. However, PHAC could make better use of the information gathered by the six NCCs via their extensive networks from external partners and collaborators. PHAC could also share its own corporate and program-level strategic priorities, and those of the Pan-Canadian Public Health Network, with the NCCs in a systematic manner, contributing to the collaborative development of NCC priorities and work plans. Lastly, despite some streamlining of the NCC performance measurement reporting framework since 2018, there is still room to improve the framework to better tell the story of the Program results and impacts.

The findings discussed in this report have led to the identification of three recommendations.

Recommendation 1: Reassess PHAC expectations of the NCCPH Program in light of funding, PHAC and public health renewal.

Program objectives have remained the same since 2005, although the public health environment has changed, especially following the COVID-19 pandemic. The evaluation findings suggest that PHAC's definition of success for this long-standing Program need to be clarified. At the same time, there are partners and collaborators inside and outside of PHAC who also have expectations for the Program given its design to serve public health practitioners and policy makers across Canada. Therefore, there is a need to review the present-day alignment of the NCCPH Program model and mandate with the current public health environment, as well as PHAC's own mandate and resources, especially in terms of public health renewal.

Recommendation 2: Enhance the two-way information-sharing relationship between PHAC and the NCCs.

Despite previous efforts to re-establish a structure to enhance coordination between PHAC and the NCCs, there remains an opportunity to improve strategic engagement with the NCCs regarding PHAC and PHN priorities. At the same time, PHAC and the PHN could take more advantage of the expertise and extensive networks of the NCCs to inform their own strategic priorities and actions.

Recommendation 3: Improve the NCCPH performance measurement framework to better tell the Program results and impacts story.

The six NCCs continue to meet or exceed PHAC requirements for annual performance measurement reporting. However, it was difficult to use this information to fully portray the results of the six NCCs and the Program's impact as a whole. Although streamlined since 2018, suggestions for improving the performance measurement framework have been made in recent reviews and echoed by interviewees. Enhancements to performance measurement should take into consideration any changes resulting from the first recommendation.

Program description

The mission of the NCCPH Program is to promote the use of scientific and other knowledge for evidence-informed decision making (EIDM) by public health practitioners, program administrators and policy makers across Canada, at all levels of the public health system.^{Footnote 1} It is a cross-cutting program that contributes to the three current core responsibilities of PHAC: Health Promotion and Chronic Disease Prevention, Infectious Disease Prevention and Control, and Health Security.^{Footnote 2}

There are six National Collaborating Centres (NCCs) located across Canada, each focused on a different priority area of public health practice as profiled in Appendix A:

• National Collaborating Centre for Indigenous Health (NCCIH), University of Northern British Columbia, Prince George, British Columbia;
• National Collaborating Centre for Environmental Health (NCCEH), British Columbia Centre for Disease Control, Vancouver, British Columbia;
• National Collaborating Centre for Infectious Diseases (NCCID), University of Manitoba, Winnipeg, Manitoba;
• National Collaborating Centre for Methods and Tools (NCCMT), McMaster University, Hamilton, Ontario;
• National Collaborating Centre for Healthy Public Policy (NCCHPP), Institut national de la santé publique du Québec, Montréal, Québec; and
• National Collaborating Centre for Determinants of Health (NCCDH), St. Francis Xavier University, Antigonish, Nova Scotia.

All six NCCs carry out the mission of the Program using a common model. Each NCC is hosted in a provincial public health institute or university that holds a contribution agreement with PHAC and is headed by a scientific or academic lead. The leads are supported by a manager and small team; all are employees of the host institution. Each Centre has an advisory committee consisting of stakeholder representatives who give input on annual work plan priorities and access to broader networks. All the NCCs perform three key functions to achieve their mission as shown in the original logic model, which can be found in Appendix B:

• Identifying knowledge gaps to encourage research in public health priority areas.
• Networking for collaboration to facilitate knowledge exchange and mobilization among a wide range of stakeholders across the public health system, i.e., practitioners, researchers and policy makers at the local, provincial, territorial and federal levels.
• Translating new and existing knowledge and research into useful and accessible formats tailored to different audiences, drawing on regional, national and international expertise.

Since 2015, the Program budget provides $973,666 per fiscal year to each of the six NCCs. They have received additional funding, mostly through ad hoc projects with various PHAC units and surge funding during the COVID-19 response. See the "Funding mechanism" section for more details.

The Program has been managed since 2017 by the Office of the Chief Science Officer (OCSO) who administers the contribution agreements and reporting requirements. It also serves as a 'single window' for other PHAC units seeking to collaborate with any of the NCCs. This is discussed in the "PHAC-NCC relationship" section. The Program aligns with PHAC's core responsibilities via the knowledge translation support it provides on a variety of priority topics to PHAC and public health-related organizations across Canada.^{Footnote 3} It also supports PHAC's commitment to applying a Sex- and Gender-Based Analysis Plus (SGBA Plus) lens given that each NCC has incorporated health equity and reconciliation with Indigenous peoples into their work.^{Footnote 4}

Origin of the National Collaborating Centres for Public Health Program

The SARS epidemic of 2003 revealed inabilities in Canada to quickly mobilize existing knowledge to inform responses to a public health emergency. It also showed a lack of coordination and collaboration on priorities for existing knowledge development. Expert consultations prior to 2003, and the SARS experience itself, underlined a need to strengthen public health expertise and rapidly transfer knowledge among all levels of government, academia and non-government organizations.^{Footnote 5} Post-2003 reviews, including the Naylor report "Learning from SARS" called for the building of capacity to develop and apply public health evidence in practice.^{Footnote 6 Footnote 7}

Following the SARS epidemic, the federal government committed to investing in the renewal and strengthening of the public health system. Subsequently, the NCCPH Program was announced in 2004 and launched in 2005 alongside the creation of the Public Health Agency of Canada (PHAC) and the Pan-Canadian Public Health Network (PHN). The initial core PHAC funding level was $1.5 million per NCC.

The NCCPH Program was initially designed to help improve joint federal, provincial, and territorial actions on public health, in parallel with the PHN. Prior to the launch of the Program in 2005, the scope was broadened to include local governments, academia, public health practitioners, and non-government organizations, essentially all levels and sectors of Canada's public health system.^{Footnote 8} A timeline of events shaping the Program is provided in Appendix C.

Evaluation scope and approach

This evaluation focused on the performance of the NCCPH Program over the 2018-19 to 2022-23 period, i.e., just prior to the COVID-19 pandemic and then over the course of the national public health response It was also intended to consider the relevance of the Program in the context of the federal public health role and to satisfy Financial Administration Act requirements. The evaluation drew on findings from multiple lines of evidence, listed in Appendix D, to address the following questions on achievements, design and implementation:

Achievement of expected objectives

• How effectively have the NCCs achieved their expected objectives of engaging with and supporting key Program partners and collaborators, including PHAC, over the last five years, including the COVID-19 pandemic response?
  • What challenges, barriers and facilitators exist to achieving Program objectives?
  • Are the objectives reasonable given the Program budget?

NCCPH model

• How well is the NCCPH Program model suited to respond to ongoing and emerging needs for knowledge translation in the Canadian public health system? How have the NCCs evolved to meet demonstrated needs and changing priorities?

Funding mechanism

• To what extent is the funding mechanism appropriate to meet the Program's objectives and partner/collaborator expectations for support from the National Collaborating Centres?

PHAC-NCC relationship

• How well are PHAC supports for the NCCPH Program working, including facilitating NCC engagement with PHAC and measuring Program performance?

Previous evaluations

The NCCPH Program was previously evaluated in 2009, 2014 and 2018.^{Footnote 9} The 2018 evaluation found that the NCCs were continuing to meet a need for knowledge translation services to make evidence-based information accessible and useful and that the NCCs occupied a unique niche. Moreover, the Program had evolved from a front-line focus to supporting all levels of the public health system and the NCCs were seen to be very collaborative, productive, credible and efficient in their use of limited resources. Their ability to foster relationships was seen as a significant strength as were the benefits of the Program model of arm's-length Centres. However, questions were raised about the alignment of NCC work plans with PHAC and working-level priorities, the limited interaction between PHAC and the NCCs and the usefulness of the requirement for all six NCCs to collaborate on joint projects.

The evaluation made recommendations on i) developing a collaborative two-way partnership between PHAC and the NCCs, ii) ensuring that each NCC remains relevant to emerging knowledge needs, and iii) exploring options to maximize efficient resource use.^{Footnote 10}

Evaluation findings

Achievement of expected objectives

This section provides an overview of Program achievements and challenges for the three main functions as well as NCC support for evidence-informed decision making over the last five years.

Knowledge gap identification

The NCCs have continued to gather information regarding knowledge needs in a variety of ways. They all sought input from their respective advisory committees and regularly liaised with OCSO to understand PHAC's priorities and identify opportunities for specific collaborations. Furthermore, the six NCCs held 97 focus groups and undertook beneficiary surveys as well as 64 environmental scans over the evaluation period. In addition, the NCCs logged

674 requests for information over the evaluation period. The NCCs set annual work plan priorities and focused their activities based on this intelligence.

Existing and emerging public health issues prioritized by the NCCs

Table 1, derived from the document review, provides examples of existing and emerging issues addressed by each NCC over the course of the last five years as a result of identifying knowledge gaps.^{Footnote 11}

Many NCC, board, host and user interviewees reported that the NCCs are nimble enough to refocus priorities and produce KT activities rapidly in response to emerging needs, enabled by leveraging relationships/networks to address emerging needs and gaps in knowledge. For example, in the case of NCCID, this flexibility allowed a focus on underserved populations in shelters and experiencing poverty in the urban core, realities and public health dynamics which PHAC would not otherwise access easily.^{Footnote 12} Out of the 69 participants surveyed, 70% strongly agreed that NCC knowledge products and activities have consistently been pertinent and timely over the past five years and that the topics covered by the NCCs are appropriate and relevant given current and emerging public health issues.^{Footnote 13}

Many PHAC, advisory board and user interviewees noted that the NCCs often assume a leadership role in public health, championing emerging areas like work on racism and discrimination, equity, climate change and the healthy built environment, well-being budgeting, population migration, and capacity for critical appraisal of evidence. In one example, the NCCIH is leading a conversation on privileging Indigenous ways of knowing alongside a Western biomedical view, recognizing that health and illness cannot simply be measured empirically but that local Indigenous wisdom, knowledge and history are equally valid tools for inquiry in the health field.^{Footnote 14}

Some NCCPH users, along with other interviewees, highlighted public health priorities which they felt were not being addressed by any of the NCCs. These included the One Health concept^{Footnote 15}, community investment and intersectoral collaboration, populations at risk such as youth, chronic diseases, substance use, antimicrobial resistance (AMR), vector-borne diseases, and mental health promotion. However, a quick sample of the six NCC websites showed that these topics have been previously addressed to lesser or greater extents. For example, the AMR project – which includes One Health approaches – at NCCID^{Footnote 16} is an ongoing priority as are various vector-borne diseases which are also addressed by NCCEH; NCCHPP's intersectoral collaboration work through Health in All Policies and Health Impact Assessment, as well as clarifying the roles of public health in promoting population mental health and wellness^{Footnote 17}; and NCCDH and NCCHPP are supporting a university-based Knowledge Development and Exchange Hub on mental health promotion which is also funded by PHAC through a different program.^{Footnote 18}

Focus on COVID-19

The NCCs picked up early signals of public health sector needs related to the COVID-19 pandemic. The NCCs consulted their advisory committees and continued listening throughout the public health emergency to their diverse networks of partners and collaborators to understand COVID-19 knowledge needs. Early responses included a COVID-related evidence review on school closures (NCCID) and guidance resources on different ethical issues faced by public health practitioners responding to a pandemic (NCCHPP). With the support of OCSO, the NCCs adjusted their annual work plan priorities accordingly, within the scope of their specific areas of practice.^{Footnote 19} The following key examples of COVID-19 knowledge translation needs were identified by the NCCs and added to their work plans:

• NCCDH: equity issues in the public health response to the COVID-19 pandemic, such as measuring differential impacts on diverse communities and highlighting equity-informed public health strategies.
• NCCEH: prevention of indoor and outdoor transmission of the SARS-CoV-2 virus that causes COVID-19.
• NCCHPP: public health ethics during a pandemic response and planning for the public health system post-pandemic.
• NCCID: essential and reliable information on coronavirus disease (COVID-19) for public health practitioners, and mathematical modelling.
• NCCIH: reliable information on the COVID-19 pandemic for Indigenous communities, including countering COVID-19 misinformation, and supporting the use of Indigenous-informed evidence and knowledge in public health decision-making.
• NCCMT: rapid reviews of evidence, coordination to reduce duplication of effort on evidence syntheses, and having an accessible repository of Canadian COVID-19 evidence reviews.

One PHAC interviewee observed that the "NCCs were at the forefront of the work at the right time, because of how well connected they are."

Networking for collaboration

An analysis of NCC reporting data showed that the NCCs combined had 523 organizational partner or collaborator relationships in one fiscal year, 2022-23, with 345 unique organizations across Canada.^{Footnote 20} Figure 1 gives a sense of the geographic distribution of these unique organizations in Canada. There were also 50 unique organizations outside of Canada connected to the NCCs.^{Footnote 21}

Figure 1 - Text description

Figure 1 is a map of Canada that displays the number of unique organizational partners and collaborators for all six of the National Collaborating Centres in each province and territory in 2022-23:

• Yukon: 1
• Northwest Territories: 2
• Nunavut: 1
• British Columbia: 38
• Alberta: 18
• Saskatchewan: 10
• Manitoba: 19
• Ontario: 133
• Quebec: 73
• New Brunswick: 5
• Nova Scotia: 8
• Newfoundland and Labrador: 2
• Prince Edward Island: 3

There were also 32 organizational partners and collaborators in the National Capital Region.

NCCs have continued to use a variety of collaborative mechanisms:

• The members of the advisory boards for each NCC represent a variety of different regions and public health organizations, allowing the NCCs to quickly gather intelligence on emerging needs for knowledge resources and establish annual work plan priorities. These relationships also benefit the participants.^{Footnote 22} For example, NCCMT reported having regular and ongoing contact with 215 senior decision-makers across Canada, such as Medical Officers of Health and senior managers of government public health services, leading to new projects in the last five years. Routine communication with 27 public health programs and schools in academic institutions has resulted in the integration of resources into curricula, and also 14 student practicum placements with the NCCMT.^{Footnote 23}
• The NCCs have a strong relationship with the Office of the Chief Science Officer (OCSO), which administers the Program contribution funding agreements and facilitates collaborations with other units across PHAC^{Footnote 24}, as highlighted in the "Funding mechanism" and "PHAC-NCC relationship" sections. A good example is the Office of the Chief Public Health Officer's annual report team, who have consulted different NCCs each year according to the report theme, including those on a health equity view of COVID-19 in 2020 and on the future of the Canadian public health system in 2021. Notably, all six NCCs were involved in publishing and disseminating four reports building on different aspects of the 2021 CPHO Report "A Vision to Transform Canada's Public Health System". The NCCIH led the creation of one of the reports, Visioning the Future: First Nations, Inuit, and Métis Population and Public Health.^{Footnote 25} In addition, PHAC has commissioned the NCCs to update the national core competencies for public health.^{Footnote 26}
• The NCCs have long-standing relationships with different public health professional associations, including a presence at major conferences, particularly the Canadian Public Health Association's annual conference and the francophone "Journées annuelles de santé publique". NCCs continue to engage with practice-based or professional associations to support the needs of their members, such as the Canadian Paediatric Society (NCCIH), the Canadian Institute for Public Health Inspectors (NCCEH), the Canadian Pharmacists Association (CHNCNCCID), the Community Health Nurses' Initiatives Group (NCCMT), the Ontario Public Health Association (NCCHPP) and the Urban Public Health Network (NCCDH).
• NCCs have also convened topic-focused stakeholder groups on priority topics such as NCCHPP's Canadian Network for Health in All Policies. Furthermore, NCCHPP hosts the secretariat of the Global Network for Health in All Policies, a country-led initiative, whose mission is to work with governments and institutions across different sectors to address the determinants of health, by strengthening the Health in All Policies (HiAP) approach, with the aim of supporting the implementation of the United Nations Sustainable Development Goals and universal health coverage.
• The NCCs continue to collaborate with each other based on their areas of expertise and priorities. The leads and managers of the six NCCs also meet on a regular basis to share priorities and identify opportunities for joint projects. In addition to supporting the 2021 CPHO report mentioned above, examples of recent projects that involved all six NCCs included public health responses for long-term evacuation and recovery^{Footnote 27}, and collaborating to hold a February 2023 Winter Institute conference on the potential for partnerships between shelters and public health organization.^{Footnote 28}

The NCC model strengthens the ability of the NCCs to develop and maintain extensive networks, enabling them to create and mobilize collaborations to address emerging public health issues. Many NCC board and host organization interviewees emphasized that the ability of the NCCs to collaborate is supported by their host universities and provincial public health institutes who are themselves seen as credible and trustworthy. See the section on "Suitability of the NCCPH Program model" for further discussion.

Many interviewees from PHAC, NCCs and the Advisory Boards explained that the NCCPH model of Centres hosted outside of PHAC, as 'arm's length' entities, allows the six NCCs the flexibility to connect with practitioners, researchers and decision-makers across the public health system and jurisdictional lines in a way that federal government organizations may find difficult to replicate. Despite these strengths, several interviewees stated that there are opportunities to expand the reach of the NCCs, with suggestions such as voicing a pan-Canadian perspective on public health priorities, using public health conferences or gatherings to have open exchanges, and engaging more with researchers and PHAC itself. There are examples across the NCCs where these types of activities have happened and are continuing to be implemented.

Focus on COVID-19

The NCCs maintained their networks of public health partners and collaborators throughout the entire COVID-19 emergency despite a rapidly shifting workforce. An NCC host and a PHAC interviewee described how pre-existing networks gave the NCCs the advantage of being simultaneously 'on the ground', while also remaining connected to decision-making and academic organizations. This enabled the NCCs to communicate issues experienced by communities and front-line workers in crisis. Their extensive connections also informed how the NCCs adapted their annual work plans to rapidly engage with and respond to emerging needs during the COVID-19 emergency response and recovery.

The NCCs supported PHAC units and structures, including by conducting rapid evidence reviews, helping to quickly prepare and post information online on their behalf,^{Footnote 29} and presenting insights to the federal, provincial and territorial Special Advisory Committee in 2020 and to the Pan-Canadian Public Health Network in 2022.^{Footnote 30} That said, despite there being a very positive response to their work and contributions, a few interviewees noted that there were difficulties integrating NCCs into important PHAC COVID-19 information-sharing structures.^{Footnote 31} Beyond PHAC, one key example of NCC work on COVID-19 was the role of NCCMT and NCCIH in supporting COVID-END, a global knowledge-translation platform hosted in Canada that was designed to support decision-makers in using evidence while reducing the duplication of effort.^{Footnote 32}

Knowledge translation

A summary of annual performance measurement data from five fiscal years, April 2018 to March 2023, showed that the NCCs:

• produced 137 tools, 1,559 publications, including 195 that were peer reviewed, 218 podcasts, 389 webinars and 405 conference presentations; and
• facilitated 117 workshops and 849 training activities.

The six main NCC websites garnered 4.73 million unique visitors and 1.26 million downloads.^{Footnote 33} The number of unique visitors to all NCC web sites has increased since the last evaluation. For the most part, NCC website traffic in 2022-23 saw more visitors than some PHAC websites with similar themes. A more detailed analysis is presented in Appendix E. The NCCs also reported around 905,000 YouTube views, most of which consisted of communicating recorded webinars and shorter issue spotlights.^{Footnote 34}

Most of the resources produced by the NCCs are online and free of charge in order to make them as accessible as possible. Of note, the NCCIH released a second edition of "Determinants of Indigenous Peoples' Health, Beyond the Social" in 2018, the first book of its kind written primarily by Indigenous authors. Building on this publication, the NCCIH released an Introduction to Determinants of First Nations, Inuit, and Métis Peoples' Health in Canada in 2022, for those new to the idea of decolonization in health.^{Footnote 35} Most documents appear to be available in French and English, however, not all resources are easily translated, such as webinar recordings. A couple of interviewees felt that the use of the full suite of NCC products and activities could be increased by raising public health practitioners' and researchers' awareness of these products, including within PHAC.^{Footnote 36}

Focus on COVID-19

The six NCCs produced a wide array of knowledge translation resources and services related to COVID-19 – some within weeks of a pandemic being declared. In general, they built libraries of reliable and curated information on priority topics, developed new guidance documents, and took advantage of their existing communications and networking capacities to disseminate information in a variety of formats. For instance, the NCCID created a "Quick Links" resource for practitioners in January 2020, updating it over time bi-weekly as new information became available, until the pandemic was declared.^{Footnote 37} As well, the NCCDH launched a COVID conversation series connecting public health practitioners, other NCCs and policy-makers.^{Footnote 38}The NCCs provided knowledge translation support for CIHR's rapid research response grants. Lastly, the NCCMT provided evidence syntheses in response to requests from decision-makers at all levels of public health. It also was a key collaborator in the creation of COVID-END, a network of over 50 Canadian and international research groups specializing in evidence synthesis, health technology assessment, and guideline development.^{Footnote 39} A more detailed inventory of the NCC COVID-19 activities is provided in Appendix F.

Support for evidence-informed decision making in public health

In general, two types of impacts were observed in the evidence gathered for this evaluation. Firstly, public health actors are accessing NCC knowledge products to use within their own organizations. Secondly, many interviewees and survey respondents related how participating in NCC-led training or in collaborative efforts enhanced their own individual or organizational capacity. Examples by core function include:

• Knowledge Gap identification: The 2018 forum "Towards Tuberculosis (TB) Elimination in Northern Indigenous Communities", co-hosted by NCCID, NCCIH, NCCHPP and NCCDH, provided inspiration for prioritizing health as a theme in a Privy Council analysis of First Nations on-reserve housing issues. ^{Footnote 40}
• Networking for collaboration: Global and national-level projects on Health in All Policies (HiAP) led by NCCHPP^{Footnote 41} since 2017 have resulted in the creation of a Canadian Network for HiAP in collaboration with PHAC, supporting an Agency priority to support intersectoral action on the social determinants of health.
• Knowledge translation: The suite of EIDM capacity-building resources and training created and refined over the years by NCCMT has been praised by participants for having enhanced their skills. Those materials have been incorporated into post-secondary curricula and adapted by local public health organizations to build the capacity of their own teams. ^{Footnote 42} NCCEH partnered with the Intact Centre on Climate Adaptation in the development of national guidance to reduce the risk of extreme heat.^{Footnote 43}

Despite a multitude of examples of public health actors supported by the NCCs, there are broader challenges faced by potential users of NCC products and activities in support of EIDM. These include demanding workloads, competing priorities, and the political context of decision making in public health organizations.^{Footnote 44}

Focus on COVID-19

The NCCs refocused their efforts to assist public health actors during the COVID-19 pandemic. Some examples of resources with high uptake include the following:

• NCCIH adapted and updated two vaccine confidence publications in collaboration with NCCID and Indigenous Services Canada. It reached over 800 Indigenous health clinics in rural, remote and urban locations.^{Footnote 45}
• The World Health Organization used the NCCMT risk communication rapid evidence review^{Footnote 46} to inform its work on public health mitigation measures in the aviation sector in 2020-21.
• NCCHPP adapted their previous work on ethical frameworks for public health early in the pandemic to support the focus on COVID-19, including an updated course for practitioners and annotated resources on applying ethics to specific aspects of the COVID-19 pandemic.^{Footnote 47} More than 3,000 people took the course. A majority of participants who gave feedback felt that the course was applicable to their work and that they were now more prepared to apply ethics in support of decision making.^{Footnote 48}
• NCCEH produced COVID-19 guidance on reducing transmission in multi-unit residential buildings which influenced provincial-level and private-sector decision-making on precautionary members and risk communications to residents.^{Footnote 49}
• NCCID and NCCDH collaborated with several public health organizations to implement jointly-produced guidance indicators for public health organizations to assess resilience and understand health equity issues in the face of emergencies.^{Footnote 50}

A 2022 NCCPH network analysis study, based on a survey of 200 public health professionals, showed that the NCCs ranked very high as a trustworthy COVID-19 information source. The analysis showed that stakeholders referred to the NCCs for COVID-19 evidence-based information at a comparable rate to the World Health Organization, PHAC, Health Canada and the US Centers for Disease Control and Prevention (CDC).^{Footnote 51}

Suitability of the NCCPH Program Model

Alongside the Program mission profiled in the "Program description" section, the important features of the NCCPH model are as follows:

• Each NCC is hosted outside of PHAC, in a university or provincial public health organization. The host organizations have largely remained the same, providing a dependable platform from which the NCCs operate.^{Footnote 52}
• Core funding is provided via a contribution agreement between PHAC and each host organization, as discussed in the next section.
• Each NCC is headed by a scientific or academic lead, assisted by a manager and a small staff.
• Each NCC has an advisory committee consisting of partner organization representatives who give input on annual work plan priorities and facilitate access to broader stakeholder networks.
• The NCCs are each focused on a different broad priority area of public health practice, but also have the flexibility to address emerging issues independently or in collaboration with each other. As such, the NCCs meet regularly with each other to share information on priorities and plan joint activities.

The Program mission and model have been stable since its launch in 2005. At the same time, the Program has shown maturity through a gradual turnover of leadership in each NCC and adaptation to several corporate changes to the management of the Program, in particular: the budget reductions noted in the next section, governance and the location of the NCCPH Program in the PHAC corporate branch structure.^{Footnote 53}

A further consideration for the Program is that the context of public health has changed between 2005 and 2023. PHAC and the Pan-Canadian Public Health Network have also matured over the same time span. Core public health competencies were developed in 2007 and there are now more schools of public health in Canada.^{Footnote 54} The number of Master of Public Health programs increased from eight in 2005 to nineteen in 2022 amidst a growing diversity of other post-secondary public health training programs.^{Footnote 55} There have been increased research investments such as the CIHR-PHAC Applied Public Health Chairs Program.^{Footnote 56} New scientific information and communication technologies have become available, including rapid vaccine development and the use of social media channels for communicating public health messages.^{Footnote 57} Lastly, public health priorities have evolved, as noted above in the section on Knowledge gap identification, such as an increasing awareness of Indigenous self-determination, structural and social determinants of health, health equity, and climate change. Despite these changes in context, the contemporary relevance of the NCCPH model is supported by two important similarities between the post-SARS and the present post-COVID-19 eras:

• As with SARS, the COVID-19 public health emergency has tested the capacity of Canada's public health organizations at all levels, including its workforce.^{Footnote 58}
• The need for knowledge translation remains strong and is increasing in the face of multiple complex public health challenges, as highlighted in the 2021 CPHO report.^{Footnote 59} The report calls for closing "the gap between knowledge generation, policy, and practice." Among other actions, it suggests enhancing "rapid and ongoing population health intervention research for prevention and well-being initiatives and strengthen interdisciplinary knowledge synthesis models such as the NCCPH." This recognition of the value of the NCC model was mirrored in a Canadian Institutes of Health Research post-COVID visioning exercise. Advantages of the collaborative knowledge translation concept underlying the NCC model are also reflected in other publications.^{Footnote 60}

Some PHAC and many NCC staff, board, and host organization interviewees related that the model has allowed the NCCs the reach, flexibility and credibility to connect with stakeholders across the public health system and across jurisdictional lines. The stability of the NCCs has also allowed such relationships to mature despite the challenges to the public health workforce outlined in the CPHO's 2021 annual report. According to an advisory board member, the NCCs are valuable because of the relevance, quality, and usefulness of their knowledge translation products and training which are "geared to the right audiences". This is consistent with research on knowledge translation to support EIDM: that it is most effective when customized to suit the varying contexts of individual knowledge users and their organizations.^{Footnote 61}

The NCC model has also supported the NCCs to quickly identify and address emerging priorities. Many PHAC, NCC, board, host and user interviewees agreed that the NCCs were able to leverage their strengths to quickly respond to COVID-19 because of their capacity for knowledge translation, their credibility being built over time, and the reach of their networks connecting community-based and front line public health practitioners, policy makers, and researchers.^{Footnote 62} One interviewee explained that these strengths have allowed the NCCs to be effective in 'peacetime' as well as in times of crisis, providing stakeholders with expertise when needed.

As found in the 2018 evaluation, there appears to be no duplication of roles between the NCCs and other knowledge-producing public health organizations in Canada. For example, the CIHR Institute for Population and Public Health is centred on funding academic research while post-secondary schools of public health are focused on training students for entry into the workforce. Instead, there are many examples of complementarity between the NCCs and other organizations in the public health knowledge 'ecosystem', as discussed in the “Networking for collaboration” section. A few PHAC, NCC, and advisory board interviewees identified that the collaborative niche is enabled by the NCC leads being experts in their field, as well as the advisory board mechanism in which experts and community members provide insight and advice to the NCCs on emerging priorities, key relationships and knowledge gaps in public health.

NCC areas of focus

Many interviewees associated with the NCCs, including users, noted that the six NCC topics are sufficiently broad to address existing priority public health issues and new ones as they emerge. As noted in the “Knowledge gap identification” section, others identified topics that they felt were not adequately addressed by the six NCCs. These included mental health promotion, a stronger focus on other populations at risk such as youth, chronic diseases, substance use and abuse, and the One Health concept of joint action on human and animal health. As previously noted, the NCCs have responded to emerging issues such as these. However, they are limited in their capacity to respond to all emerging issues outside of their annual work plans, as discussed in the following section.

Funding mechanism

Overall, the six NCCs have operated on a combined budget of $43 million over the last five fiscal years. A summary of NCC funding sources is shown in Figure 2. A more detailed table of funding per NCC is provided in Appendix G.

Figure 2 - Text description

Figure 2 is a pie chart showing the amount and proportion of funding, by type for all six of the National Collaborating Centres, from 2018-19 to 2022-23:

• $29.2 million in core contribution agreement funding from PHAC, representing 67% of total funding.
• Additional PHAC funding of $6.8 million as a COVID-related top-up, representing 16% of total funding.
• Additional PHAC funding of $2.5 not related to COVID response and recovery, representing 6% of total funding.
• $5 million from funding sources outside of PHAC, representing 11% of total funding.

Over the past five years, the six NCCs received $29.2 million in contribution agreement funding from PHAC, or $973,666 of core funding annually per NCC. The core funding level has remained the same since 2015, following two consecutive budget reductions since the inception of the Program.^{Footnote 63} The core funding is intended to cover NCC staff salaries and contribute to operational expenses.^{Footnote 64}

PHAC has provided additional short-term funding averaging $1.5M per NCC over the last five years. The funds were flowed to the NCCs via 16 requests for amendments, two MOUs, a contract and a grant. Specific projects were funded by at least 12 different PHAC units, as described in the next section.

Most of the additional PHAC funding was to increase NCC capacity temporarily to help with the COVID-19 pandemic response and recovery. More specifically, an amendment of $120,000 was provided to each NCC in 2020-21. Further amendments of up to $533,333 were provided to each NCC in both 2021-22 and 2022-23. Many interviewees stated that this extra funding during the pandemic was crucial for their response to be effective, enabling a temporary increase in capacity.^{Footnote 65} It should be noted that such funding amendments were rare prior to COVID-19 but have become more frequent. Program staff had already processed eight funding amendments from various PHAC branches by the second quarter of the 2023-24 fiscal year. In contrast, there were seven such amendments in all of 2018-19.^{Footnote 66}

NCCs also received a total of $5.0 million from sources outside of PHAC over the last five years to undertake projects with other partners outside of their PHAC contribution agreement work plans. A majority of this funding was secured by the NCCIH from a number of different federal and provincial government agencies.

NCC host organizations have continued to provide important in-kind supports for the use of the institutional infrastructure such as facilities, utilities, accounting, library, human resources, information technology, and auditing services. It is also understood that some host organizations are making significant contributions to salary costs. Data on host contributions was not complete or consistent enough to report on the total value of host support for each NCC.

Strengths of the contribution agreement mechanism

Many PHAC, NCC and advisory board interviewees commented that the contribution agreements between PHAC and the six host organizations for the NCCs formalize and help to structure the PHAC-NCC relationship. They felt that this mechanism was more suitable than a grant or contract model. A grant would have no flexibility to add supplementary funding for special projects and would remove the ability of PHAC to tie any conditions to the funding, such as the submission of annual workplans or performance reports.^{Footnote 67} Similarly, a contract would only be suitable for providing a service directly to the funder and not for assisting a broad range of public health actors.^{Footnote 68} The eight-year renewal of the current contribution agreements, from 2020 to 2028, has provided more certainty for NCC staffing and planning than the previous cycle of five-year agreements. It has also decreased some administrative demands by allowing NCCs to better concentrate on program delivery and network development over a longer funding period.

The flexibility of the contribution agreement mechanism has allowed PHAC to direct additional funding to specific priorities and to use funds that could otherwise have lapsed. Similarly, it also allows the NCCs to adjust annual work plan commitments. Some PHAC, NCC and host interviewees expressed that additional funding was helpful and appreciated. However, they also described that unexpected amendments that come part-way through a fiscal year exert significant demands on limited NCC capacity to use these funds before the end of that fiscal year. There were suggestions that the amendment process could be improved through better coordination with the NCCs and increased alignment with PHAC and NCC annual administrative funding cycles.^{Footnote 69} In response, Program staff reported working with PHAC units to encourage efficiency in timing and coordination.

Funding limitations

Most interviewees cited the static core funding as an increasingly significant limitation on the capacity of the NCCs to meet the expectations of PHAC and non-PHAC partners/collaborators on fulfilling their role without supplementary funding.^{Footnote 70} A key aspect of NCC capacity to respond to public health knowledge needs concerns staffing. Many NCC interviewees pointed out that it has been harder to retain experienced staff as the static core budget is not adjusted for inflation nor the ensuing increases in operational costs. The short-term and unpredictable nature of supplementary PHAC funding via contribution agreement amendments makes it very difficult to use those additional funds for the salaries of long-term staff. Had the 2015 core funding level of $974 thousand per NCC per year kept pace with inflation, it would have increased to $1.2 million per NCC by 2023.^{Footnote 71}

The NCCs have adapted to this fiscal reality. For example, a few external interviewees highlighted that NCCs are optimizing their budgets by finding efficiencies such as reducing the number of in-person meetings and thus the cost of associated travel through the use of virtual meeting platforms. However, the NCCs are increasingly unable to allow for the salary increases – in some case required by the host institutions – necessary to retain experienced senior staff who are vital to their success, especially subject-matter experts who nurture extensive knowledge-sharing networks. Nor are NCCs able to guarantee employment past the end of the current contribution agreements, recognizing that other organizations are in a similar position. As related in the 2023 NCC collective evaluation report and by some NCC, advisory board and host organization interviewees, experienced NCC staff may leave NCC employment to seek higher-paying positions offering more certainty of longer-term or permanent employment. Many PHAC, NCC, advisory board, and host interviewees expressed concern that budget constraints are making it difficult for NCCs to meet their objective of supporting the knowledge needs of a wide variety of public health organizations and practitioners. The NCCs may soon reach a critical point where they no longer have adequate capacity to serve current partners and collaborators as a primary knowledge source. These challenges were previously noted in the 2018 evaluation.

PHAC-NCC relationship

PHAC is a unique Program partner as it simultaneously acts as a:

• Funder: Through the contribution agreement, PHAC administers the core funding for all six NCCs as well as occasional amendments and associated annual reporting requirements. PHAC does so to achieve the Program mission, i.e., enabling evidence-informed decision making by stakeholders at all levels of the public health.
• Knowledge user: PHAC staff access knowledge resources produced by the NCCs, just like other stakeholders across Canada and internationally.
• Collaborator: Different PHAC programs have undertaken or contributed to specific projects with individual NCCs. Some of these projects have prompted PHAC to send additional funding to each NCC via contribution agreement amendments or, less frequently, contracts.

The relationship is manifested in three key ways: engagement on setting priorities, awareness and intelligence-sharing, and performance measurement reporting. This relationship has adapted to changes in the position of the OCSO and the NCCPH Program within PHAC's corporate structure over time, reflecting larger changes in the Agency.^{Footnote 72}

Engagement on priority setting

Many NCC staff interviewees noted that the OCSO team is very professional and promotes open communication with the NCCs. The OCSO staff administer the contribution agreements, have facilitated regular encounters between the NCCs and various PHAC managers to discuss priorities and often act as a point of contact for other groups within PHAC interested in collaborating with any of the NCCs. A close working relationship between the NCCs and the Program team at PHAC has been maintained over the years. An example of this relationship was the NCC work plan flexibility introduced by the OCSO in April 2020 to prioritize COVID-19 pandemic-related knowledge needs. Almost all NCC staff members mention the appreciation they had for the flexibility PHAC demonstrated during the pandemic, providing surge funding so that they could take on COVID projects as well as the flexibility to amend annual work plans to prioritize COVID-19 work.

Following the 2018 evaluation, the OCSO laid the groundwork for a PHAC-NCC Executive Leads Committee, intended to coordinate PHAC's engagement with the NCCPH and to provide advice to the Chief Science Officer on strategic direction and opportunities in relation to the NCCPH Program. Although this committee was suspended soon after launching due to the demands of the COVID-19 response, the OCSO resumed meetings between the NCCs and PHAC senior managers in October 2022.

Despite the good relationship with PHAC, many NCC staff members interviewed noted that the mandate of the NCCs to serve all levels of the public health system can mean that PHAC's needs must be balanced against those of other partners and collaborators, some of which have also provided funding to various NCCs. Given PHAC's role as the primary funder, there is a risk that PHAC's needs will overwhelm or eclipse those of other partners and collaborators in NCC work plans.

A few PHAC management interviewees noted that there is a need to move beyond ad hoc collaborations with the NCCs to identify priorities more strategically for the NCCs that reflect PHAC's role and priorities. There was also recognition that an intake process could be established for PHAC branches to propose and fund Agency-specific projects for the NCCs as part of operational planning.

Awareness and intelligence sharing

The OCSO has continued to promote awareness of the NCCs across PHAC by facilitating meetings and presentations open to all PHAC and Health Canada staff. However, some NCC staff interviewees noted that PHAC staff awareness of the NCCs was challenged by turnover, the growth of the Agency's workforce, and changes to PHAC's organizational structures. These realities can make it difficult to form critical relationships and maintain awareness of what the NCCs can offer or are already doing.

NCC intelligence on knowledge gaps and the priorities of many different stakeholders across jurisdictions and sectors could help inform PHAC and the PHN.^{Footnote 73} A PHAC manager gave the example of NCCIH becoming a member of a PHN rights and reconciliation working group in recognition of the Centre's capacity to contribute an expert Indigenous public health perspective on critical issues. At the same time, two PHAC managers reflected that a strategic awareness of PHAC and federal, provincial, and territorial PHN priorities should guide NCCs.

Some PHAC interviewees identified topics for which NCCs have become national leaders, becoming knowledge sources for PHAC in the process. These include NCCIH's support for Indigenous self-determination over health and well-being, NCCDH's focus on health equity and NCCHPP's work on Health in All Policies. Despite the growth of interest in the health impacts of climate change and the built environment, a few interviewees related that there seems to be no focused group within PHAC addressing this theme and that NCCEH serves as a natural platform for it.

Performance measurement

PHAC's performance measurement requirements for the NCCPH Program produce rich data for describing the productivity of the NCCs according to their three core functions, as well as their reach. The current tools for Program performance measurement are an annual Performance Measurement Questionnaire (PMQ), an annual work plan update, and an external evaluative requirement. Reporting requirements were streamlined after the 2018 evaluation that found that the NCCPH performance reporting requirements were cumbersome and time-consuming, especially considering limited staff resources to complete them. A new PMQ format was launched in 2019-20. However, the format is still open to interpretation, which has resulted in some inconsistent data between NCCs and fiscal years. This has presented challenges for analysis. In addition, NCC interviewees have stated that the current requirements are very resource intensive despite the effort made to streamline them.

Despite the significant time invested by NCC staff in producing performance information, it provides limited information about the difference that NCCs make in supporting the use of evidence in stakeholder decision making and public health practice. The current Program logic model shown in Appendix B does not portray how the NCCs work or link to current performance measurement requirements. Instead, it focuses on how the Program links to a historic PHAC corporate reporting structure. More broadly, it is challenging to systematically assess results of knowledge translation activities due to the complex set of factors influencing outcomes^{Footnote 74}, many of which are outside the NCC sphere of influence.

PHAC, NCC, and host organization interviewees, along with a recent NCC collective evaluation and separate PHAC assessment, provided suggestions to address these issues.^{Footnote 75} There is an opportunity for NCCs and OCSO staff to collaboratively improve the usefulness and feasibility of the current performance measurement requirements.

Conclusions and recommendations

Conclusions

Since the last evaluation in 2018, the six NCCs have continued to produce a wide array of relevant knowledge products in response to identified knowledge gaps. The NCCs continued to build and maintain extensive networks in order to collaborate with many organizations across Canada, including PHAC. From the beginning of the COVID-19 pandemic, they effectively pivoted from their normal work plans to support the public health response. Many examples of NCC contributions were noted by key informants, including leadership on issues like health equity, reconciliation with Indigenous peoples and climate change. They have also increasingly collaborated with PHAC units, notably on the CPHO's annual reports.

The need for knowledge translation support to public health practitioners and policy makers remains strong despite changes in the sector since 2005. These changes have included emerging public health issues alongside those that persist, new technologies, new players in public health, and evolving public health approaches. That said, like in 2005, the public health workforce has recently been affected by a major public health emergency. The NCCPH Program has matured as shown through the stability of the NCC host organizations, gradual turnover of NCC leadership and adaptation to PHAC's changes in Program management and funding. The work of the NCCs has continued to demonstrate their expertise, credibility across jurisdictions, and responsiveness to new and emerging public health priorities.

The NCC model of knowledge centres, hosted 'at arm's length' outside of PHAC and supported by independent advisory committees, promotes these strengths. This includes the Contribution Agreement mechanism that has provided structure to the PHAC-NCC relationship and flexibility to address changing priorities. In addition, the six NCCs are seen to be operating efficiently within the available funding, as well as seeking alternate sources of funding and in-kind support.

Supplementary funding from PHAC and external sources of funding have enabled the NCCs to generate more knowledge resources and activities, particularly during the COVID-19 emergency response. However, the NCCs have been facing increasing challenges in meeting expectations of supporting evidence-informed decision making for Program partners and collaborators at all levels of Canada's public health system without supplementary funding. The short-term and ad hoc nature of supplementary funding affects the prioritization of annual work plan commitments as well as the ability to address long-term NCC capacity gaps. This situation has led to uncertainty, noted in the 2018 evaluation, regarding PHAC's expectations for this long-standing Program in fulfilling its mission. There is broad agreement among interviewees that the current Program budget does not support sufficient NCC capacity to serve the knowledge translation needs of all key public health actors.

The PHAC-NCC relationship, facilitated by the OCSO, is mutually supportive due to the collaborative nature of the engagement with the leads and managers of the six NCCs. This includes the facilitation of regular meetings and communications between PHAC units and the NCCs, as well as the fulfillment by the NCCs of extensive performance reporting requirements. That said, increasing a strategic two-way sharing of information on public health needs and priorities could help both PHAC and the NCCs to better target their respective activities, meaning that PHAC could make better use of information gathered by the six NCCs via their extensive networks. PHAC could also share its own corporate and program-level strategic priorities, and those of the Pan-Canadian Public Health Network, with the NCCs in a systematic manner, contributing to the development of NCC work plans. There is also room to continue to improve the NCC reporting framework in order to document Program results more efficiently and effectively.

Recommendations

The findings discussed in this report have led to the identification of three recommendations.

Recommendation 1: Reassess PHAC expectations of the NCCPH Program in light of funding, PHAC and public health renewal.

The NCCs were created alongside PHAC and the Pan-Canadian Public Health Network, as part of the original vision for renewing federal public health supports post-SARS. While the six NCCs have continued to perform well, the public health environment and the Program operating context have changed since 2005, especially following the COVID-19 pandemic. However, the NCCPH Program objectives have remained the same since 2005 and the evaluation findings suggest that PHAC's definition of success for this long-standing Program need to be clarified. This may also affect how PHAC and NCC staff understand the alignment of the NCCPH Program model and mandate with PHAC's own mandate. Given that the NCCPH Program was founded alongside PHAC, alignment is an important issue in the context of discussions on PHAC renewal following the COVID-19 pandemic. At the same time, there are partners and collaborators inside and outside of PHAC who have expectations for the Program given its design to serve public health practitioners and policy makers across Canada. Therefore, there is a need to review the present-day alignment of the NCCPH Program model and mandate with the current public health environment, as well as PHAC's own mandate and resources, especially in terms of public health renewal.

Recommendation 2: Enhance the two-way information-sharing relationship between PHAC and the NCCs.

The PHAC-NCC committee that was put in place by OCSO following the 2018 evaluation to enhance coordination was suspended due to the demands of the COVID-19 response. Although the PHAC-NCC relationship continues to be collaborative, the evidence points to an opportunity to improve information sharing between PHAC and the NCCs in two ways:

1. From PHAC to the NCCs: There is an opportunity to build on the OCSO's existing means of sharing of PHAC priorities with the NCCs, such as fielding requests from PHAC units to collaborate, in order to focus more on aligning the work of the NCCs with PHAC and Pan-Canadian Public Health Network priorities at a strategic level.
2. From the NCCs to PHAC: The examples of the NCCs contributing to the annual CPHO reports and presenting to the COVID-19 federal-provincial-territorial Special Advisory Committee suggest that the NCCs could provide useful information to PHAC and the Pan-Canadian Public Health Network, gathered 'from the field' on priority public health issues as they arise.

Recommendation 3: Improve the NCCPH performance measurement framework to better tell the Program results and impacts story.

While the six NCCs continue to meet PHAC's requirements for annual performance measurement reports and supplementary evaluations, it was difficult to use the resulting performance reporting information to fully portray the results of the six NCCs and the impact of the Program as a whole. The performance measurement framework, although streamlined since 2018, could be focused further on demonstrating the achievement of expectations. Suggestions for improving the performance measurement framework have been made in recent reviews and echoed by interviewees. However, the effectiveness of any future changes to the performance measurement framework remains dependent on a renewed understanding of Program expectations; see Recommendation 1.

Management Response and Action Plan - Evaluation of the National Collaborating Centres for Public Health Program, 2018–19 to 2022–23

Recommendation 1

Reassess PHAC expectations of the NCCPH Program in light of funding, PHAC and public health renewal.

Management response

Agree

Action plan

To review the NCC program objectives in light of the shifting public health landscape, future public health workforce needs, available funding, and PHAC renewal.

Deliverables

• Engagement with NCCs and key stakeholders, including a summary report of their perspectives
  • Expected completion date: March 2025
• Review of present public health landscape
  • Expected completion date: March 2025
• Ratification of any changes to NCC program objectives by PHAC Executive Committee
  • Expected completion date: March 2026

Accountability

OCSO in collaboration with Strategic Policy Branch

Resources

Current program support team within PHAC

Recommendation 2

Enhance the two-way information-sharing relationship between PHAC and the NCCs.

Management response

Agree

Action Plan

1) To continue to sustain internal awareness of NCCs annual work, exchange on emerging public health issues arising from NCCs business domains and foster new NCCs collaborations with PHAC programs and committees.

• Deliverable: An NCC engagement strategy will be presented to President and CPHO and include:
  • Annual NCC/PHAC engagement meeting with PHAC senior executives
  • Periodic NCCs presentations to PHAC governance tables (PHN, EC, EG, DG policy)
  • Better coordination and connectivity between PHN and NCCs
  • Periodic information sharing meetings with PHAC programs leads
  • Reporting on NCC and PHAC collaborative projects
    • Expected completion date: March 2025
    • Accountability: OCSO and relevant program VPs

2) To integrate expected collaborations between PHAC and the NCCs into the Integrated Operational Planning and budget requirements.

• Deliverable: Current PHAC corporate planning tools incorporate a subsection to prospectively identify expected collaborations with the NCCs. This deliverable is expected to reduce the need for amendments to the NCC contribution agreements and also create more stability around NCC resourcing for PHAC-related projects.
  • Expected completion date: October 2024
  • Accountability: OCSO/CMB/OCFO

3) To review the NCCs annual workplan submission dates in light of PHAC planning cycle in order to harmonize NCCs and PHAC planning cycles.

• Deliverable: Revised NCCs work plan submission date as indicated in the contribution agreement (Appendix C—Reporting Plan).
  • Expected completion dare: New date set by December 2024
  • Accountability: OCSO

Resources

Current program support team within PHAC

Recommendation 3

Improve the NCCPH performance measurement framework to better tell the Program results and impact story.

Management response

Agree

Action plan

OCSO to review the NCCPH Performance Measurement Framework in light of the findings of both OAE and the assessment exercise. OCSO to engage with the NCCs throughout this process.

Deliverables

Updated NCCPH logic model and performance measurement framework and updated performance reporting tools and templates.

Expected completion date

March 2026 (interdependent with outcomes of Recommendation 1)

Accountability

OCSO

Resources

Current program support team within PHAC

Appendix A: Short profiles of the six National Collaborating Centres

Appendix B: NCCPH logic models

This NCCPH logic model was introduced in 2014-15 and was included in the latest performance measurement strategy dated May 2017. The links to PHAC's corporate program structure and expected results portrayed are out-of-date as of 2023.

Figure 3 - Text description

This diagram depicts the components of the NCCPH program logic model which is a visual illustration of the program's inputs (resources), activities and expected outcomes.

Starting from the bottom of the diagram, the National Collaborating Centres for Public Health's key input is Agency A-Based funding. This input is expected to produce the following activities and outputs: develop and manage contribution agreements with National Collaborating Centres. These activities and outputs are expected to lead to the following immediate, intermediate, and ultimate outcomes; immediate outcome: Knowledge brokers have resources and structures to collaborate across the Public Health system to address knowledge-related needs and gaps required to strengthen evidence-informed decision-making; intermediate outcomes: Products and tools to translate and disseminate public health knowledge are developed and shared through a variety of networks; ultimate outcomes: (i) knowledge is mobilized across networks and sectors to fill information gaps, build evidence-informed decision-making capacity and respond to emerging health issues in the Public Health system, (ii) public health professionals, policy makers and Canadians have access to reliable, actionable Public Health data and information and (iii) Canadians are protected and empowered to improve their health with mechanisms in place to enable the Public Health system to work collaboratively to address existing and emerging public health infrastructure issues.

Figure 4 - Text description

The 2009 NCCPH logic model diagram starts at the top with depicting three activities:

• Program management: NCCPH Program Secretariat, at PHAC, is responsible for managing the contribution agreements established with six host organizations that each house an NCC, promoting and providing advice and guidance to the program, providing secretariat support for the National Advisory Council and supporting the priorities of public health. It also acts as a liaison between PHAC, the Health Portfolio and NCCs. The NCCPH Program Secretariat also monitors and evaluates NCC outputs and activities to ensure alignment with priorities of public health, and facilitates alignment of knowledge translation activities between PHAC and the NCCs
• The National Advisory Council for the NCCPH (provides advice to program management).
• The NCC Collaborative includes representation from each NCC and provides a forum for collective action across all NCCs. It is supported by the Leads Secretariat.

The logic model shows that the NCCPH program activities are intended to reach the following audiences: the NCCs; research and education communities; regional and municipal local health authorities; provincial and territorial governments; public health practitioners; policy makers; NGOs and all Canadians.

The logic model shows the activities leading to the following Program outputs:

• National Advisory Council materials, meetings with NCCs, priority setting documents, promotion tools for NCCPH program; and
• Managed contribution agreements with six host organizations that each house a Centre. At arm's length from PHAC, and within program parameters, each NCC focusses on its own public health priority, develops its own and contributes to a collaborative workplan, disseminates its own knowledge products and builds networks of partners that include regional, national and international subject matter experts, policy makers, practitioners, researchers, academics and others).

The logic model shows the outputs leading to the following immediate level outcomes:

• increased opportunities for enhancing public health through collaboration with the Health Portfolio and NCCs;
• knowledge translation: the synthesis of knowledge in existing and emerging areas of public health, and the dissemination, translation and exchange of knowledge across diverse sectors and disciplines in public health;
• knowledge gap identification: gaps are identified and act as catalysts for applied or new research; and
• networking: increased collaboration with NCCs occurs among and across public health at all levels.

The logic model depicts the following intermediate outcomes:

• increased availability of knowledge and knowledge products for evidence informed decision making in public health;
• increased use of evidence to inform public health programs, policies and practices;
• partnerships developed with external organizations; and
• mechanisms and processes to access knowledge.

These intermediate level outcomes are shown as contributing towards three outcome areas that link to the PHAC Population and Public Health Fund (under which the NCCPH program is funded): Research/Knowledge, Capacity Building and Intersectoral Collaboration.

Lastly, the logic model depicts the Program's long term outcome as "improved public health programs and policies".

* Health Portfolio includes Health Canada, the Public Health Agency of Canada, the Canadian Institutes of Health Research, the Hazardous Materials Information Review Commission, the Patented Medicine Prices Review Board and Assisted Human Reproduction Canada.

** Population and Public Health Fund under which the NCCPH Program is funded

Appendix C: NCCPH Timeline 2005 to 2023

Figure 5 - Text description

This image provides a brief timeline of the development of the NCCPH program. The timeline contains the following dates and descriptions:

• 2005: Launch of the six NCCs alongside the creation of PHAC and the Pan-Canadian Public Health Network. Responsibility for the program is in the PHAC Office of Public Health Practice. Note: The NCCPH program was transferred in a later year to the Public Health Capacity Development Division in the Centre for Public Health Infrastructure, Health Security Infrastructure Branch.
• 2013: NCCPH National Advisory Council disbanded.
• 2015: Reduction in annual core funding per NCC to $973.6 thousand.
• 2018: Public Health Infrastructure removed from PHAC's corporate program structure. Responsibility for the NCCPH program moved from the Health Security Infrastructure Branch, to the Office of the Chief Science Officer in the Infectious Diseases Prevention and Control Branch.
• 2019: Renewal of the NCC contribution agreements for eight years to 2027-28.
• 2020: Start of COVID-19 pandemic.
• From 2020 to 2023: direction from PHAC to focus on supporting stakeholders to respond to COVID-19; temporary increase in annual core funding per NCC.
• 2022: Office of the Chief Science Officer elevated to branch status. Publication of he CPHO Report: A Vision to Transforms Canada's Public Health System; the NCCs commissioned by PHAC to produce supplementary reports.
• 2023: End of COVID-19 pandemic emergency response.

Appendix D: Data Collection and Analysis Methods

This evaluation focused on the performance of the NCCPH Program over the 2018-19 to 2022-23 period. It also considered the relevance of the Program in the context of the federal public health role. The evaluation team collected data using the following methods:

Document and Performance Information Review

Reviewed PHAC and NCC reports focused on NCC activities and needs for knowledge translation in public health, particularly:

• evaluations and evaluative reports produced by PHAC or any of the NCCs;
• Program documentation, including contribution agreements and the performance measurement framework; and
• PHAC corporate reports, including CPHO annual reports and PHAC Departmental Results Reports.

Interviews

Completed 44 interviews with 58 interviewees from the following groups:

• PHAC Program administration staff and senior managers (n=6)
• NCC staff (n=16)
• Host organization representatives (n=9)
• A sample of partners and knowledge users primarily identified by the NCCs:
  • NCC advisory committee representatives representing a variety of stakeholder groups (n=7)
  • Local public health services (n=2)
  • Professional associations (n=3
  • PHAC staff and managers (n=12) and representatives of other federal departments (n=3)

Questionnaire (to supplement the interviews)

Received 43 responses to a questionnaire sent to 68 additional partners and knowledge users identified by the NCCs. These included representatives of Indigenous organizations, federal, provincial, and territorial and municipal governments, professional associations, non-government organizations, and research institutes.

Performance data review

Reviewed annual reporting information from each NCC for five fiscal years 2018-19 to 2022-23.

Financial data review

Reviewed information on core Grants and Contributions funding and additional financial transfers from PHAC to the NCCs.

Literature review

Performed a limited scan of recent literature on knowledge translation in public health.

The evaluation team used triangulation to analyze data collected by these various methods in order to increase the reliability and credibility of the evaluation findings and conclusions. Still, most evaluations face constraints that may affect the validity and reliability of findings. The following list presents limitations encountered during evaluation, their impact, and the mitigation strategies that were put in place.

Limitation: Interviews are retrospective in nature, providing only a recent perspective on past events.
Potential impact: This can affect the validity of assessments of activities or results that may have changed over time.
Mitigation strategy: Triangulation with other lines of evidence substantiated or provided further information on data captured in interviews. Document review also provided corporate knowledge.

Limitation: The potential number of interviewees was very large given the extensive networks and variety of partners, collaborators and knowledge users for each of the six NCCs.
Potential impact: Some potential interviewees were unable to contribute their insight.
Mitigation strategy: Interviewees were selected to achieve representation among all NCCs. A questionnaire was used to extend the opportunity for more respondents to participate than could be interviewed. However, the reach of the questionnaire was very limited compared to NCC networks and cannot be considered to be a significant sample of NCC users.

Limitation: Representation by gender, Indigenous identity or other identities was not considered in the design of the interview or questionnaire methodology.
Potential impact: It is unclear if a balance of voices from key NCCPH Program beneficiary groups was present in the data collected for this evaluation despite the presence of a wide variety of professional, organization and regional representation among those interviewed or reached via the questionnaire.
Mitigation strategy: This continued to be a limitation in the evaluation design, which was mitigated partially by the use of products developed by the NCCs. Future evaluations of the NCCs will enhance representation in the initial design stage.

Limitation: While extensive, some elements of the Program performance measurement data were incomplete or inconsistent between NCCs and between fiscal reporting years.
Potential impact: Assessment of progress towards outcomes was challenging.
Mitigation strategy: Triangulation of other lines of evidence was used to provide further information where there were gaps in performance measurement data.

Limitation: Financial data structure is not linked to outputs or outcomes. Furthermore, data on staffing and in-kind supports to the NCCs was limited.
Potential impact: There is a limited ability to assess efficiency quantitatively.
Mitigation strategy: The evaluation focused more on other outcome areas and used triangulation of other lines of evidence to the extent possible.

The evaluation applied an SGBA Plus lens to its assessment of the Program. Although official languages were not specifically examined, they were not found to be an issue for the Program's activities. An examination of the Sustainable Development Goals was not specified for this evaluation, although the Program (mirroring public health in general) indirectly addresses Goal 3: Good health and well-being, while contributing to others such as Goal 10: Reduced inequalities.^{Footnote 76}

The Office of Audit and Evaluation worked closely throughout the evaluation with a Program contact in the Office of the Chief Science Officer to access documents and performance data, and to identify stakeholder groups. The scope for this evaluation was shared secretarially with the PHAC Performance Measurement and Evaluation Committee in April 2023 to help guide the evaluation questions. The final report and Management Response and Action Plan developed by the Office of the Chief Science Officer were also presented to this committee in January 2024.

Appendix E: NCC web site statistics highlights

Figure 6 - Text description

This figure contains a bar chart showing the number of unique visitors to the main web page for each of the six National Collaborating Centres, between 2018-19 and 2022-23. The data comes from the NCC Performance Measurement Questionnaires. Figures are displayed on the bar chart as follows:

NCCIH:

• 2018-19: 125,762
• 2019-20: 131,170
• 2020-21: 193,072
• 2021-22: 208, 040
• 2022-23: 198,119

NCCEH:

• 2018-19: 65,052
• 2019-20: 76,053
• 2020-21: 286,183
• 2021-22: 428,425
• 2022-23: 269,127

NCCID:

• 2018-19: 93,529
• 2019-20: 141,522
• 2020-21: 138,365
• 2021-22: 164,375
• 2022-23: 126,168

NCCMT:

• 2018-19: 236,969
• 2019-20: 251,130
• 2020-21: 253,552
• 2021-22: 219,450
• 2022-23: 220,518

NCCHPP:

• 2018-19: 38,042
• 2019-20: 44,582
• 2020-21: 53,312
• 2021-22: 53,879
• 2022-23: 52,365

NCCDH:

• 2018-19: 80,781
• 2019-20: 97,887
• 2020-21: 147,448
• 2021-22: 155,552
• 2022-23: 176,700

Data Source: NCC Performance Measurement Questionnaires

Appendix F: Key examples of NCC COVID-19 response activities

Appendix G: NCCPH summary funding 2018-19 to 2022-23

End notes