Evaluation of the National Collaborating Centres for Public Health program 2014-15 to 2018-19

Prepared by
Office of Audit and Evaluation
Health Canada and the Public Health Agency of Canada

March 2019

List of Tables

List of Acronyms

CIHR
Canadian Institute of Health Research
CPHO
Chief Public Health Officer
NCCs
National Collaborating Centres
NCCAH
National Collaborating Centre for Aboriginal Health Canada
NCCDH
National Collaborating Centre for Determinants of Health Canada
NCCEH
National Collaborating Centre for Health Public Policy
NCCID
National Collaborating Centre for Infectious Diseases
NCCMT
National Collaborating Centre for Methods and Tools
NCCPH
National Collaborating Centre for Public Health Canada
NGOs
Non-Governmental Organizations
PHAC
Public Health Agency of Canada
SARS
Severe Acute Respiratory Syndrome

thumbnail

Download the alternative format
(PDF format, 593 KB, 56 pages)

Organization: Health Canada

Published: 2019-06-24

Executive Summary

The purpose of the evaluation was to assess the relevance and performance of the National Collaborating Centres for Public Health (NCCPH) program for the period of April 2014 to September 2018. The evaluation was conducted to fulfill the requirements of the Financial Administration Act and Treasury Board of Canada's 2016 Policy on Results. In the assessment of relevance, the evaluation examined the continued need for the program, understanding of the National Collaborating Centres' (NCCs') role, as well as alignment with government, PHAC, and public health priorities. In assessment of performance, the evaluation examined the achievement of expected outcomes, resource use, mechanisms for coordinating with PHAC, and efficiency related to the program model.

1. Program Description

The creation of the NCCPH program was announced in 2004, along with the PHAC and the Pan-Canadian Public Health Network, as an integral part of the Government of Canada's overall national strategy to strengthen the public health system in Canada.

This program aims to make public health research more relevant and understandable for public health professionals and organizations, so that it can be used in the their day-to-day practices and in policy making (i.e., supporting science from policy to practice).

The NCCPH program distributes a total of $5.8M per year to six National Collaborating Centres (NCCs), located across Canada, that synthesize, translate, and share knowledge on a specific theme to make it useful and accessible to policy makers, program managers, and public health professionals.Footnote [a] Each Centre receives equal funding through a contribution agreement between PHAC and an external organization responsible for hosting the Centre. The six centres and their respective hosts are as follows:

2. Key findings

Relevance

There is a continued need for knowledge translation services to make evidence-based information accessible and useful to public health professionals in support of decision making and policy making. In particular, there is a growing expectation that use of well-sourced and timely evidence will be the standard practice in public health decision making, but challenges remain in doing so (e.g., expanding volume and complexity of data, broadening of public health to include social determinants of health, ongoing resource constraints in the field). As well, in Canada's multijurisdictional public health system, there is an ongoing need to foster networks across jurisdictions in order to share information and best practices.

Although there are other organizations that provide knowledge synthesis, translation, and exchange services in public health, the evaluation found evidence that NCCs occupy a unique niche focused on translating evidence-based knowledge in a very practical manner to support public health professionals across the country. Moreover, NCCs differ from other organizations in that they are non-governmental, which gives them a greater ability to address needs on-the-ground. They are also national in scope, while other organizations tend to have a more provincial or local focus. Some key informants suggested, however, that there is a need to reassess the themes addressed by the Centres based on evolving public health priorities.

Since the last evaluation, there has been an evolution in the mandate of the NCCPH program, from focusing on strengthening public health capacity and targeting local level professionals, to supporting decision making and policy making at all levels of the public health system. Most, but not all, Centres have reflected this shift in program mandate in their collaborations. The intent of the NCCPH program clearly aligns with the Government of Canada and PHAC overall priorities of strengthening public health capacity and science leadership. However, despite NCCs being set up in a way that they can be more responsive to the direct needs of public health professionals, some questions were raised as to whether NCC work aligns well with public health sector priorities at the working level. These questions are partly due to a lack of awareness of how the NCCs develop their work plans, as they do collect input from a range of various sources to inform their development.

Achievement of Outcomes

The NCCs have produced a wide range of high quality knowledge translation products and activities, and are perceived as a credible go-to source on multiple public health issues. There are also many examples of NCC contributions to decision making and policy making in the public health field.

In many instances, the NCCs have responded to emerging needs, including identifying knowledge gaps. However, the current process for developing work plans makes it challenging for them to be nimble and address ad hoc issues as they arise.

The NCCs have undertaken many collaborations with external partners as part of their core business line. Those collaborations have allowed stakeholders to leverage each other's capacity to achieve results, as well as share information across regions and jurisdictions. In many cases, the NCCs have brought together stakeholders that governmental organizations would not have been able to convene in the same manner. Their ability to foster relationships was seen as a significant value-added service provided by the NCCs.

Collaboration happened regularly between two or three NCCs on a variety of initiatives. These collaborations were seen as productive, as each NCC had valuable expertise to contribute. However, the requirement for all six of them to collaborate on signature projects was seen as diverting limited resources that could be better used elsewhere.

Efficiency

Overall, the Centres were seen to be operating efficiently and using their limited resources in a prudent and innovative manner to deliver a significant number of outputs. However, their ability to deliver outputs has been curbed over time, as their funding from PHAC has declined from an annual allocation of $1.5M in the program's initial years, to a flat budget of about $974K per year for the current contribution agreement period (2015 to 2020).

Overall, the NCCPH program model has been seen as beneficial in helping the Centres operate efficiently, allowing them to access the services and expertise of their host organization, leverage their networks, and access funding to advance joint projects. The model of having Centres at arm's length from the Government of Canada has also given the NCCs the ability to remain apolitical and connected on-the-ground. However, while Centres are small in scale, they have to follow the procedures of both their host organization and their contribution agreement with PHAC. Current performance reporting requirements were seen as being cumbersome and time-consuming, especially considering the reduction of the program budget over time.

There was also limited interaction and communication between PHAC and the NCCs on priorities, making it challenging for the NCCs to align annual work plans with areas of interest to PHAC, in order to foster collaboration and avoid duplication. The designated executive lead initiative launched by PHAC in February 2018 has shown promise for creating closer collaborations between the NCCs and PHAC, but it is too early to assess the results.

Recommendations

Recommendation 1

Develop a collaborative two-way partnership between PHAC and the NCCs to promote greater awareness of each other's work and increase collaboration in areas of common interest.

Within PHAC, there has been a general lack of understanding of the role, mandate, and value of NCCs. Also, NCCs have expressed concerns that PHAC priorities are not communicated to them in a timely and systematic manner. This has caused a challenge for NCCs in aligning annual work plans to areas of interest for PHAC and in connecting with appropriate program areas. In light of this, PHAC should build on the initial efforts made by the executive lead initiative and engage more strategically and regularly with the NCCs, in order to leverage each other's knowledge, resources, and networks to advance common goals.

Recommendation 2

Explore opportunities, as part of the contribution agreement renewal, to ensure that each NCC remains relevant to emerging public health sector needs in terms of issues addressed, range of collaborations, and targeted audiences.

The NCCPH program mandate has evolved since the last evaluation, from focusing on strengthening public health capacity and targeting local level professionals, to supporting decision making at all levels of the public health system. Not all Centres have reflected this shift in the program mandate in their target audience and their collaborations, as they continue to focus on local practitioners. As well, NCCPH program topics may have been relevant when selected over 13 years ago, but there was a concern that those themes had never been revised since the program was created, and that knowledge translation efforts should look at other issues that are aligned with emerging global public health priorities. Given that the current funding cycle ends in 2020, PHAC should explore opportunities to ensure that the NCCs remain relevant, including clarifying directions given to them, in order to achieve the current program mandate and fulfill the evolving needs of the public health sector.

Recommendation 3

Explore options for maximizing resource allocation to the NCCs and allow them to use those resources more efficiently to fulfill their core mandate. This includes:

The reduction in funding allocation from PHAC to the NCCs has curbed their capacity to deliver outputs. In this context, PHAC should investigate opportunities to maximize resource allocation and manage expectations. More specifically, PHAC should revisit the requirement for all six NCCs to collaborate on signature projects, given that they are collaborating with each other through other means. PHAC should also provide NCCs more flexibility to address emerging issues as they arise, without having to make significant revisions to their work plans and without affecting accountability for results achieved during the year. PHAC should streamline performance measurement requirements, with a particular emphasis on providing useful data to inform ongoing program management and evaluation, while also keeping the NCCs' reporting burden in line with their current level of funding allocation.

Management Response and Action Plan

NCCPH Program Management Response and Action Plan (2018-2019 to 2019-2020)
Recommendations Response Action Plan Deliverables Expected Completion Date Accountability Resources
Recommendation as stated in the evaluation report Identify whether program management agrees, agrees with conditions, or disagrees with the recommendation, and why Identify what action(s) program management will take to address the recommendation Identify key deliverables Identify timeline for implementation of each deliverable Identify Senior Management and Executive (DG and ADM level) accountable for the implementation of each deliverable Describe the human and/or financial resources required to complete recommendation, including the source of resources (additional vs. existing budget)
Recommendation 1: Develop a collaborative two-way partnership between PHAC and the NCCPH to promote greater awareness of each other's work and increase collaboration on areas of common interest. Agree. The notion of "partnership" is now central to PHAC's business relationship with the NCCs.

1. Formalize the mandate of the PHAC-NCC Executive Leads as key executive liaisons for each of the six NCCs, and with the objective to enable of enabling effective and strategic collaborations between PHAC and the NCCs.

1.1 Establish PHAC-NCC Executive Lead (PEL) Terms of reference

30 September 2019    

2. Raise internal awareness of NCCs annual work, exchange on emerging public health issues arising from NCCs business domains, and foster new NCCs collaborations with PHAC programs and other public health stakeholders.

2.1 A reviewed engagement plan for the NCCs, including the presentation of selected products/issues to PHAC staff and managers through various mechanisms (branch governance, PHACtually speaking special sessions, PHAC/HC broadcast news).

30 September 2019

 

Vice-President, Infectious Diseases Prevention and Control Chief Science Officer Existing resources
 

2.2 The renewal of the contribution agreements for 2020 and beyond will identify a partnership relationship between the NCCs and PHAC and the principles under which the partnership relationship is preserved.

30 September 2019    
Recommendation 2: Explore opportunities, as part of the contribution agreement renewal, to ensure that each NCC remains relevant to emerging needs of the public health sector in terms of issues addressed, range of collaborations and targeted audiences. Agree. 1. Identify and review with PELs and NCCs key priority areas for public health knowledge mobilization. 1.1 A section expressing areas of interest in knowledge mobilization is included in the next solicitation request to NCCs. 30 June 2019    
1.2 A planning meeting with NCCs leads and the CPHO Report Unit 30 September 2019    
2. Work with CGC to refine renewal process and documentation for efficiency and efficacy, yet retain essential TBS requirements. 2.1 Restructured / simplified solicitation process documentation 30 June 2019    
3. NCCPH Program to review NCC annual work plans for their ability to address a broad range of actions (collaborations, networking, and gap identification) that reflect key priorities.

3.1 The annual work plan review process / schedule is documented and updated to include consultations of relevant programs, PELs and PHAC regional directors.

30 September 2019

Vice-President, Infectious Diseases Prevention and Control Chief Science Officer Existing resources

Recommendation 3: Explore options for maximizing resource allocations to the NCCPH and allow them to use those resources more efficiently to fulfill their core mandate. This includes:

  • Revisiting the requirement to collaborate all six of them on signature projects;
  • Providing the NCCs with flexibility to adapt their work plan to address emerging issues (considering that addressing knowledge gaps is part of their core mandate); and,
  • Streamlining performance measurement requirements.
Agree.

1. In the way forward, NCCs will no longer be required to commit to a 'signature' project involving all six Centers. The program will continue to encourage the NCCs to collaborate among themselves, as they see fit, on big projects.

1.1 Removal of NCCPH 'signature' projects from agreement requirements, as documented in the proposed agreements submitted for signatures

30 June 2019

Vice President, Infectious Diseases Prevention and Control Chief Science Officer Existing resources
2. The current work planning cycle allows flexibility by the NCCs to propose and implement changes to their work plans based on emerging priorities. NCCPH Program will pro-actively discuss changes in the work plan and possible in-year additional funding to adapt to changes. 2.1 Provision of a work planning flexibility in the Agreement, as documented in the proposed agreements submitted for signatures. 30 June 2019    
3. NCCPH Program will review current program monitoring and practices to lessen the burden on NCCPH recipients, yet fulfill the Agency's obligations under the Financial Administration Act and CGC standard operating procedure (SOP). 3.1 Updated NCC reporting requirements documentation 30 September 2019    

1.0 Evaluation Purpose

The purpose of this evaluation was to assess the relevance and performance of the National Collaborating Centres for Public Health (NCCPH) program for the period of April 2014 to September 2018. Thee valuation was conducted to fulfill the requirements of the Financial Administration Act and Treasury Board of Canada's 2016 Policy on Results.

2.0 Program Description

2.1 Program Context

Following the 2003 outbreak of Severe Acute Respiratory Syndrome(SARS), a review from the National Advisory Committee on SARS (the Naylor Report) identified challenges facing the public health system in Canada among multiple public health actors at the regional, local, provincial, and territorial levels. The Naylor Report recommended improvements to Canada's public health infrastructure by establishing an arms-length organization to support other levels of government in strengthening public health systems, while fostering an environment of collaboration among public health actors. One "core element" of the public health infrastructure described in the Naylor Report was a central resource for knowledge translation and evidence-based decision making, including the identification of research needs. Footnote [1]

In response to this need, the Government of Canada established the Public Health Agency of Canada (PHAC) and, at the same time, established the Pan-Canadian Public Health Network (PHN) and the NCCPH program. These three initiatives were components of the Government of Canada's efforts to strengthen the national public health system in Canada.

2.2 Program Profile

The National Collaborating Centres (NCCs) Footnote [b] were created to provide a national focal point for sharing knowledge in priority areas for public heath, while building on existing expertise in regions across Canada. In other words, the Centres aim to promote evidence-informed decision making through the following activities:

The NCCPH program established six collaborating Centres hosted at independent organizations across Canada, either a university or provincial public health organization, each specializing in a specific public health priority area. These six Centres are the following (see Appendix B for more details on each centre):

2.3 Previous Evaluations

The NCCPH program was previously evaluated in 2014, covering the period from 2008-09 to 2013-14. Footnote [2]

The previous evaluation found that the NCCPH program broadly aligns with Government of Canada and PHAC roles and priorities, and that there was a continued need for mechanisms to enhance evidence-based decision making in public health. However, there was a lack of clarity regarding alignment and complementarity with other organizations engaged in knowledge translation, synthesis, and exchange. The previous evaluation also found that the program had made progress towards increasing evidence use to inform public health practice, though progress varied across the Centres. It was shown that NCC knowledge products and activities were used by public health professionals to support evidence-based decision making, though the topic-specific design of Centres restricted their ability to respond to emerging public health issues.

The evaluation made recommendations related to clarifying the role of PHAC and the NCCs in knowledge translation and enhancing performance measurement. The program's Management Response and Action Plan was fully implemented.

2.4 Program Narrative

The objective of the NCCPH program is to promote the use of knowledge for evidence-informed decision making by public health professionals across Canada (i.e., supporting "science to policy to practice").Footnote [3] The NCCPH program is built on a theory of change that improved availability, access, and collaboration in regards to knowledge will enable better practices and decisions from public health professionals. This, in turn, is expected to lead to better public health outcomes.

The expected results of the NCCPH program are the following:

Achieving these results supports PHAC's goal of protecting Canadians and empowering them to improve health, with mechanisms in place to enable the public health system to address existing and emerging public health infrastructure issues. The link between the activities, outputs, and outcomes of the NCCPH contribution program is described in the program logic model (Appendix A).

2.5 Program Alignment and Resources

The program's financial data for fiscal years 2014-15 through 2018-19 are presented below (Table 1). The program distributes approximately $5.8 million each year to the six NCCs, with each Centre receiving about $974 thousand annually. The program's total expenditures were $31.6 million over five years.

Table 1: NCCPH Program Planned Budget, 2014-15 to 2018-19 ($)
Year Grants & Contributions Operation & Maintenance SalaryTable A footnote A Total
2014-15 $ 5,842,000 $ 97,000 $ 317,700 $ 6,256,700
2015-16 $ 5,842,000 $ 97,000 $ 339,680 $ 6,278,680
2016-17 $ 5,842,000 $ 107,695 $ 442,956 $ 6,392,651
2017-18 $ 5,842,000 $ 71,900 $448,492 $ 6,362,392
2018-19 $ 5,842,000 $ 40,000 $442,342 $ 6,324,342
Total $ 29,210,000 $ 413,595 $1,991,170 $ 31,614,765

Data Source: Estimated planned budget provided by the Chief Financial Officer Branch at PHAC.

Table footnote 1

Salary includes Employee Benefit Program

Return to table footnote A referrer

3.0 Evaluation Description

3.1 Evaluation Scope, Approach, and Design

The evaluation examined core issues of program relevance and performance, namely the following:

Specific questions on each of these issues were developed based on program considerations and these guided the evaluation process. A detailed description of the evaluation methodology is presented in Appendix C.

Data collection for the evaluation was comprised of the following:

3.2 Limitations and Mitigation Strategies

Most evaluations face constraints that may have implications for the validity and reliability of evaluation findings and conclusions. Table 2 below outlines the limitations encountered during the implementation of the selected methods for this evaluation. Also noted are the mitigation strategies put in place to ensure that the evaluation findings can be used with confidence to guide program planning and decision making. The use of multiple lines of evidence and triangulation were intended to increase the reliability and credibility of the evaluation findings and conclusions.

Table 2: Limitations and Mitigation Strategies
Limitation Impact Mitigation Strategy

There was a lack of consistency in how some of the performance measurement data was reported between NCCs and over several years (e.g. web statistics and numbers of products).

In addition, it was understood that the annual performance measurement questionnaire submitted by the NCCs did not include all outputs produced by the NCCs in a given year (e.g., NCCs may have not reported all their collaborations in a given year).

It was difficult to report valid and reliable totals and trends over time at the level of the whole program and to make comparisons across the NCCs.

Assessments of NCC outputs and collaboration may not be exhaustive.

Performance measurement data was taken at face value. Aggregate numbers derived from performance data in this report have been interpreted with caution and are accompanied with proper caveats to help readers interpret their meaning. Where feasible, data from the performance measurement was triangulated with other sources of information.
The evaluation did not conduct a survey of public health professionals in Canada in order to measure their uptake and awareness of NCC products. The evaluation, for the most part, did not include the viewpoints of public health stakeholders who had no knowledge of, or interaction with, the NCCs. Therefore, the evaluation does not fully assess where there are gaps in terms of knowledge and awareness of the NCCs and their work within the Canadian public health system. Data collected from the interviewees was triangulated with other sources to assess the reach of the NCCs. As well, interviewees were asked to provide an opinion on the relevance of the NCCs, (e.g., gaps in reach, issues, and needs addressed). In addition, an analysis of NCC partnering was performed to show reach by region and type of organization (allowing gaps to be inferred).
The selection of key informants for this evaluation was based on suggestions provided by PHAC regional offices, the program secretariat and the NCCs. Key informants who completed an interview had previous knowledge of the NCCs. Since the evaluation did not include the viewpoints of public health stakeholders who had no knowledge of, or interaction with, the NCCs, it is possible the key informants had bias toward the NCCs. A vast number of individuals were invited to participate in the interviews and the sampling strategy was designed to ensure a representation of all regions of Canada and of a variety of public health stakeholder groups (frontline personnel, regional and national policy and program representatives, academics, etc.). Key informants generally had a balanced perspective on the NCCs raising both positive points and areas for improvement. Information collected from the key informants was validated with other sources of data where possible.

4.0 Findings

4.1 Relevance

4.1.1 Continued Need for Knowledge Translation

There is a continued need to support efforts to make evidence-based information more useful and accessible to public health professionals, in order to support decision making and policy making. There is also an ongoing need to foster networks and collaboration across the public health system in order to share information and best practices.

Since its launch in 2004, the NCCPH program has aimed to address the need for knowledge mobilization in public health and facilitate the application of evidence and emerging research in policy development and practice implementation. In the context of this evaluation, findings from all lines of inquiry have indicated that there is a continued requirement for public health professionals to access and use well-sourced evidence in a collaborative manner to support decision making and policy development.

A review of recent literature indicates that, to achieve objectives of better population health, more widespread adoption of evidence-based strategies across the public health landscape, in innovative formats continues to be a necessity. Footnote [5] To support this, access to timely, reliable, and actionable information through cross-sector partnerships is a necessary priority for public health in the 21st Century. Footnote [6] Additionally, studies found that, despite many accomplishments in public health over the last 30 years, practitioners and policy makers alike must pay greater attention to evidence-based approaches to public health, as there are numerous direct and indirect benefits,including the following: Footnote [7]

In order to achieve these benefits, researchers noted that wide-scale dissemination of practical information on effective public health interventions must occur more consistently at all levels of government. Footnote [8]

Key informants, including NCC staff and external stakeholders, echoed findings from the literature and document reviews that there is both a growing expectation to implement evidence-based decision making and, more specifically, that there has been an evolution in practice related to its acceptance as the standard over the last five years. In light of this, the majority of internal and external key informants noted a need for ongoing support to address challenges in the capacity to use evidence-based information effectively. Challenges identified by key informants include the following:

4.1.2 Understanding the Role of the NCCs

The role of the NCCs is fairly well understood by public health professionals outside of PHAC; however, internal key informants noted a general lack of understanding of the NCCs' role within PHAC.

The majority of external key informants were able to articulate the mission and vision of the NCCs. Many external key informants understood that the broader role of NCCs was to support the public health system by making related research more relevant, accessible, and understandable for practice and in policy making, and this was seen as well aligned with PHAC's fundamental responsibilities. A few external key informants noted that NCCPH had a role to play in sparking innovation and applying new methods and tools to address public health problems.

Based on interviews conducted with internal key informants, there appears to be a widespread lack of awareness within PHAC as to the ongoing role and mission of NCCs, and how they relate to the work PHAC undertakes. A majority of internal key informants noted a lack of clarity on whether it is the NCCs' mandate to support PHAC's work and on how PHAC could engage with the NCCs to advance areas of common interest. (See section 4.3.3 for more details).

Based on internal and external key informant interviews, it seems that this lack of clarity on the role of NCCs is, at least, partly explained by perceptions from both the NCCs and PHAC staff. On one hand, as explained by NCC key informants, the Centres typically do not see PHAC as part of their primary targeted audience. On the other hand, the majority of internal key informants have reported not knowing how to engage with the NCCs but also not being able to engage with the NCCs because they are perceived to be at arm's length from PHAC. Overall, it has been a challenge to find the right balance between supporting PHAC while also maintaining the independence of NCCs and their ability to develop work plans based on needs identified on-the-ground.

4.1.3 Adapting to an Evolving Program Mandate

The NCCPH mandate has evolved since the last evaluation. Its goal has shifted from strengthening public health capacity and targeting local level professionals to focusing on supporting decision making at all levels of the public health system. However, not all NCCs target all levels of the0020public health system and not all have adjusted the scope of their collaboration to reflect the shift in program mandate.

As noted by program authorities and in public documents, such as PHAC's Report on Plans and Priorities, for years previous to 2014-15, the NCCPH program mandate was to strengthen public health capacity, translate health knowledge, and promote and support the use of knowledge and evidence by public health professionals in Canada, in collaboration with provincial, territorial, and local governments, academia, public health professionals, and non-governmental organizations.

The program mandate statement shifted with the renewal of contribution agreements in April 2015. As noted in public documents, such as the 2015-16 PHAC Report on Plans and Priorities, the program mandate statement was expanded from a specific focus on strengthening public health capacity to promoting the use of knowledge for evidence-informed decision making by policy makers, program managers, and professionals.

The shift in program focus was also accompanied by a change in target audience and collaborators. The mandate statement prior to the contribution agreement renewal clearly outlined an objective to collaborate with local governments. However, the current mandate statement targets public health professionals in different roles across the public health system, from frontline practitioners to managers and policy makers.

Most NCCs have reflected this shift in program mandate in their target audience. As confirmed by most NCC key informants, the Centres' target audience is generally public health professionals across the system. However, the NCCEH tends to have a particular focus on public health inspectors, who represent a narrower group of Canadian public health professionals. The NCCAH is different from other NCCs since it is the only population-based NCC and has a commitment and responsibility to Indigenous individuals, families, and communities. However, according to NCC key informants, NCCAH's targeted audience includes Indigenous and non-Indigenous organizations, networks, practitioners, educators, and government organizations that have a vested interest in Indigenous health and wellness.

In practice, as further discussed in section 4.2.1, according to surveys and assessments conducted by the NCCs, users of their products are generally individuals or organizations working at the local and regional levels of the public health system.

Furthermore, an assessment of collaborators and partners reported in the NCC performance data shows that not all NCCs have a national scope to their collaboration. Specifically, most of NCCEH, NCCHPP, and NCCMT reported collaborators and partners were located in the region where each of those centres are located, or were located outside of Canada in the case of NCCMT Footnote [c]. Other NCCs appear to have a more national scope when it comes to working with collaborators and partners (see section 4.2.1 and Appendix B for more details). Additionally, the same performance data shows that all NCCs reported having collaborators and partners mainly within provincial, territorial, or local governments. The only exception is NCCAH, which has a broader range of collaborators from the Government of Canada and national-level Indigenous organizations.

4.1.4 Overlap, Duplication, and Complementarity

There are other organizations involved in the field of knowledge synthesis, translation, and exchange, but evidence suggests that NCCs have found a unique niche focused on translating evidence-based knowledge into a very accessible and practical manner that supports public health professionals across Canada.

As indicated in the previous 2014 NCCPH Evaluation Report, there are number of other organizations and initiatives that support knowledge, synthesis, translation, and exchange activities in the public health area. This has not changed significantly in the last five years. Table 3 below presents an overview of some of the key players currently in the field Footnote [d].

The previous evaluation was not able to conclude on the extent to which knowledge, synthesis, translation, and exchange efforts that existed at that time complemented the NCCPH program. In the context of the current evaluation, most internal and external key informants noted that, despite any overlap, most NCCs have found their niche and target different audiences, or focus on different issues than organizations included in their field. Other external knowledge users noted that, even if there is some duplication, this can be a positive factor, as organizations can share knowledge and expertise and build on common areas of interest. In general, it was seen that that NCCs differ from other organizations in the sense that their focus is national, they are non-governmental, and their scope is to translate evidence-based information in a very practical way.

As shown in Table 3, the evaluation team did not find organizations dedicated to knowledge translation and synthesis on Indigenous health issues.

Many external key informants also noted that the NCCs complement others in the field and ensure a level of consistency across the country regarding access to the best public health evidence. Some examples of complementary activities include the following:

Table 3: Other Knowledge, Synthesis, Translation and Exchange Initiatives in Public Health
KSTE Initiative Scope Services/Activities/Tools Provided
Geographic Reach Disease Specific vs Broad Public Health Knowledge Creation Knowledge Translation Networking/ Collaboration Education/ Training
Canadian AIDS Treatment Information Exchange (CATIE) National AIDS/HIV Yes Yes Yes Yes
Canadian Public Health Association (CPHA) National Public Health   Yes Yes Yes
Institute of Population and Public Health, Canadian Institutes of Health Research (IPPH- CIHR) National Public Health Yes Yes Yes  
Pan-Canadian Public Health Network National Public Health     Yes  
Cochrane Database of Systematic Reviews International All Health   Yes    
Public Health Ontario Provincial Public Health   Yes Yes Yes
Institut national de sante publique du Québec (INSPQ) Provincial Public Health Yes Yes Yes Yes
British Columbia Centre for Disease Control Provincial Communicable and chronic disease, preventable injury and environmental health risks Yes Yes Yes Yes
Regional Health Authorities (approx. 73 across Canada almost half in Ontario) Footnote [10] Local level Public Health   Yes Yes Yes
Wellesley Institute Local (Greater Toronto Area) Social Determinants of Health/Health Equity   Yes Yes Yes

Data Source: Website review for each initiative; accessed October 2018.

4.1.5 Alignment with Government and Public Health Priorities

The overall mandate of the program aligns with Government of Canada and PHAC overall priorities around strengthening public health capacity and science leadership. However, at the working level, it is not clear if the activities of the NCCs align well with PHAC and public health sector priorities. As well, some key informants noted a need to re-assess the themes addressed by the Centres based on evolving public health priorities.

Over the last five years, improving knowledge transfer and evidence-based decision making in the public health workforce and subsequently protecting the health of Canadians have been identified as priorities for the Government of Canada. As such, the NCCPH program's broad mandate in this area continues to align with priorities set forth in the following sources:

Findings on the level of alignment between NCC and PHAC priorities at the working level are mixed and uneven across the NCCs. As reported by both internal and NCC key informants, discussion is generally limited between PHAC and the NCCs on how to align their priorities. See section 4.3.3 for more details.

External key informants had mixed perspectives on whether or not the work of NCCs was well aligned with public health sector priorities. On one hand, a number of external key informants felt that the NCCs were well aligned with sector priorities, and were even responsible for pushing agendas forward in emerging areas. These areas include health equity considerations for public health inspectors (NCCDH and NCCEH), training in Indigenous child and youth health to reflect the Truth and Reconciliation call to action (NCCAH), online health impact assessment training modules (NCCHPP), and refugees and public health intervention strategies (NCCID), to name a few. See Section 4.2.4 for more detail on NCCs' work in these areas.

On the other hand, many other external key informants generally agreed that the work of the NCCs was useful, but raised questions on whether it was focusing on the right issues. Many external key informants commented on not knowing what input is considered in work plan development and not being consulted on it.

That being said, as part of their regular working process, the NCCs use different means to identify the knowledge needs of the public health sector. These mechanisms include conducting periodic environmental scans and targeted surveys, and obtaining feedback from various public health professionals through their advisory board and networks of collaborators. Other important mechanisms for the NCCs to stay abreast of emerging issues in public health are external requests for information and invitations to collaborate that they receive on a continuing basis. These serve as sources of information on stakeholder needs for NCCs to consider in their work planning. As reported by NCC key informants, this information is used to define priorities in their draft work plans and drive their activities.

Some internal and external key informants noted concerns about the program renewal along current themes and host organizations. They noted that NCCPH program topics may have been relevant when selected over 13 years ago, but identified a need to re-assess the themes addressed by the Centres based on evolving public health priorities.

New themes suggested by one or more external key informants included the following:

4.2 Effectiveness: Achievement of Outcomes

4.2.1 Access, Reach, and Use of Evidence-Based Resources

Evaluation evidence indicates that, overall, the six NCCs have successfully mobilized evidence-based information through a variety of channels, thus reaching professionals across Canada.

Products and activities generated by the NCCs

NCCs work together to promote the use of their scientific research and knowledge to strengthen public health practice, programs, and policies in Canada. To accomplish this, the Centres are expected to identify knowledge gaps, foster networks, and provide a range of evidence-based resources, including knowledge translation services. Footnote [15]

According to performance data provided by the NCCs, they developed a variety of knowledge translation products and activities each year, including peer-reviewed and non-peer reviewed publications, such as blog posts, commentaries, position papers, fact sheets, guidance documents, evidence reviews, videos, podcasts, infographics, book chapters, articles, and reports. Footnote [16]

Overall, NCCs distributed consistent levels of knowledge translation products and activities during the period covered by the evaluation. From 2015-16 to 2017-18, they collectively produced a total of 250 to 450 knowledge translation products each year, with an average of 260 knowledge translation activities each year. They offered a wide range of knowledge translation activities designed to meet the unique needs of their target audience, including in-person workshops and programs, web-based training and webinars, conference presentations, and knowledge exchange events.

Reach of NCC products and activities

The evaluation examined the reach of NCC products and activities using web metrics and survey of users conducted by the NCCs. Readers should keep in mind that this data may not provide a complete assessment, since each Centre uses a variety of channels to distribute their products, including mailing lists, workshops, webinars, online courses, presentations, courses, national gatherings, etc.

Web metrics in Table 4 indicate that the NCCs have generated a relatively high level of online engagement with their products. In 2017-18, NCCMT and NCCAH in particular had levels of unique visitors two to three times higher than the highest ranking comparable web content from PHAC. NCCID's web content was visited nearly twice as often as PHAC's Infectious Disease web content.

Table 4: Number of Visits to NCC Websites compared to Selected PHAC Web pages in 2017-18 (ranked from Highest to Lowest)
Websites Unique Visitors Average Time on Site
NCCMT 195,687 5:30
NCCAH 127,142 2:59
PHAC Vaccines and Immunization 94,542 0:57
PHAC Social Determinants of Health and Health Inequalities 45,088 2:34
NCCID 44,636 2:27
NCCDH 44,598 1:44
NCCEH 40,264 2:00
NCCHPP 34,566 2:27
PHAC Infectious Diseases 25,393 1:01
PHAC CPHO Report 5,928 1:00

Data Source: Self-reported data from NCC (as shown in 2017-18 Performance Measurement Questionnaire) and PHAC web analytics

An analysis of survey data reported in evaluations conducted by the NCCs show that, in general, frontline public health providers made up between 20 to 30% of knowledge users. Other types of knowledge users included public health program managers, community and non-governmental organizations, researchers and students, medical officers of health, and health care providers. While all NCCs reached public health professionals in general, some Centres reached more specialized audiences. The NCCAH, for example, most commonly reached community-based organizations and organizations serving Indigenous populations, while the largest group of knowledge users for NCCEH was public health inspectors.

Each Centre was able to demonstrate that their training and products reached users from across Canada. Based on survey data from the Centres, individuals from Ontario made up between 30-60% of knowledge users, followed by users from Quebec, who accounted for 10-15% of knowledge users. This is generally aligned with the distribution of the Canadian population. Footnote [17] Some Centres demonstrated a higher proportion than average of knowledge users in the region where they were located, relative to other Centres. This included NCCHPP where approximately 30% of users were from Quebec, NCCDH in Nova Scotia where users from their province were 10% compared to approximately 1-3% among other Centres, and NCCAH and NCCEH in British Columbia, where users from that province made up between 17-20% compared to under 10% in other regions. Since the evaluation did not conduct an assessment of the awareness of NCCs among public health professionals in Canada, it is not possible to determine whether those Centres have a higher concentration of users within their own region because users are more aware of the Centre located closer to them. In addition, two NCC key informants commented that regional distribution of their users reflects the geographic distribution of public health professionals, especially whose practice and expertise relates to the work of a given Centre across Canada. The evaluation team has not been able to confirm this statement since there were no statistics available on the number and distribution of public health professionals in Canada.

4.2.2 Usefulness of Evidence-Based Resources

In general, NCC products are perceived as a credible go-to source by knowledge users. They described these products as timely and relevant, and shared the view that they were useful to a broad public health audience.

When asked about the relevance and timeliness of NCC resources, most key informants, in both PHAC and external organizations, shared the opinion that NCCs produce highly credible work that is pertinent to their needs. Some knowledge users highlighted that this was particularly important where local or regional public health units operate in a situation of resource scarcity.

Survey evidence collected by the NCCs confirms that knowledge users have a positive view of the quality and relevance of their products. Surveys conducted by NCCAH, NCCMT, and NCCHPP showed that 80 to 100% of survey respondents had high levels of satisfaction with training and knowledge synthesis products, and had access to up-to-date information.

In particular, a few external knowledge users specifically noted that NCCAH was an excellent source for culturally-relevant knowledge products that reflect Indigenous cultural and historical realities. This is validated by survey responses from the NCCAH that indicate that over 90% of knowledge users found their products and services to be culturally relevant. One key informant noted that this Centre allows students in universities to have access to Indigenous knowledge as it pertains to public health issues, knowledge that may not otherwise be available.

While there was a general level of agreement that NCCs produced timely and relevant information products and services, a small proportion of knowledge users, mostly working within PHAC, noted a concern in regards to the ability of NCC products to meet local needs or the needs of specialized knowledge users. Specifically, a few external key informants noted that, due to the broad scope of issues addressed by NCCs, coupled with the fact that their work addresses public health issues from a national perspective, knowledge translation products are not always relevant to local and regional realities or the needs of specialized knowledge users. In general, these concerns were considered an acceptable limitation on the usefulness of the NCCs knowledge translation products, given an understanding of the Centres' mandate and resource levels.

4.2.3 Contribution to Policy and Decision Making

There are many examples of NCC products being used to inform and support policy and decision making.

Knowledge users in the field of public health across Canada, and within PHAC, described a variety of ways in which they referred to NCC knowledge products and activities. The most common way knowledge users reported using these products was by accessing training provided by the Centres for professional development, while educators working in academia and in community programming used NCC materials as resources to support learning for students or community members. As well, several internal and external knowledge users also indicated that they consulted NCC knowledge translation products to receive an overview of available evidence, which often informed their work. The NCCs also addressed requests from stakeholders at all levels of the public health system, such as requests for information and consultations on specific issues, for letters of support (e.g., for research funding proposals), to participate in peer reviews, to submit articles for publication, to partner on a project, or to present at an event. NCCs have also helped foster relationships to support knowledge exchange on key public health issues by leveraging their existing networks with key stakeholders on specific policy and practice issues, including frontline health and public health professionals, partners working within other provinces and territories, not-for-profit organizations (NGOs), and Indigenous organizations.

Furthermore, anecdotal evidence suggests a variety of ways in which NCC products have been used to support evidence-informed decision making by a broad range of knowledge users. Specific examples heard from key informants and validated with evidence from internal and public documents for each NCC include the following (See Appendix B for more detail):

Further to those examples, NCCs have informed policy making by the Government of Canada in general: some NCCs have appeared before Senate and House of Commons Standing Committee hearings to provide information on issues such as mental health of Indigenous peoples (NCCAH), antimicrobial resistance (NCCID), the impacts of radon on health (NCCEH), and the impacts of extreme heat (NCCEH).

4.2.4 Addressing Emerging Issues and Knowledge Gaps

There are many examples of the contributions NCCs have made to address emerging public health issues. However, the current process for developing work plans and their limited resources make it challenging for them to be nimble. In addition, NCCs are only able to address emerging issues where there is already a great deal of existing data.

External key informants and performance data have identified many examples of NCC products and resources that address emerging public health issues, including:

The evaluation found numerous examples of how the NCCs have addressed emerging issues, and some external key informants praised the NCCs as being responsive 'go-to' sources of pertinent and timely information on hot-button issues. However, many others, both internal and external, were often not confident that NCCs had been nimble or actively contributed to providing information on emerging issues in a timely manner. In this regard, internal NCC staff explained that their annual work plan process is set over the months leading up to a new fiscal year and is difficult to change once established. A few external key informants noted that the NCCs' scope of action pertains to issues where there is already a great deal of data and research available, but NCC key informants reported being active in convening actors across the country on emerging issues where data and research is lacking.

4.2.5 Fostering Collaboration with Public Health Partners

The NCCs undertake collaborations with a range of partners as part of their core line of business. These collaborations aim to achieve greater results than participants could achieve alone and to build networks across the system. Overall, NCCs' ability to collaborate on different initiatives and to network with different partners across the public health system is seen as one of the most valued capabilities of the Centres.

Mechanisms for collaboration, purpose, and reach

Collaboration between the six NCCs and public health stakeholders is a core aspect of the NCCPH program and it is systematically recorded in NCC annual work plans. External key informants and NCC staff described collaborations as a strategic means of addressing knowledge needs related to specific issues. Collaborations enable the NCCs to broaden their reach beyond what they could achieve by acting alone.

As shown in the performance data and key informant interviews, the NCCs have each developed networks of partners over time, through a range of different mechanisms. Typically, NCC collaborations consist of the following type of activities:

NCCs are reporting to PHAC on the collaborators and partners they had over a given year as well as the names of the projects where these collaborations/partnerships occurred. An analysis of collaborations and partnership data reported for 2015-16 to 2017-18Footnote [e] show that approximately 570 individual collaborations were documented, giving an average of 95 collaborations per Centre over three years. The greatest level of reported collaboration was with governmental organizations at all levels of the public health system, with a focus on provincial and territorial governments or local governments. These were followed by academic groups, NGOs and Indigenous organizations. While most NCCs reported having most of their collaboration with governmental organizations, NCCMT reported having slightly more collaborators and partners from the academic domain than from governmental organizations. NCCAH also reported the highest number of collaborators from national-level Indigenous organizations and academic groups.

In terms of geographical distribution, the highest concentration was with national-level organizations; however some NCCs tended to focus more on their own geographic area. The data shows that NCCHPP and NCCEH reported having collaboration and partnerships mostly with organizations located in their own regions but reported national-level collaborations as their second highest area of focus. In contrast, data reported by other centres show that NCCDH highest concentration of collaborations was in Ontario, NCCAH had the most national-level collaborations and NCCMT was most active internationally. Of note, although the program scope is national, about 14% of NCC reported collaborations were with organizations located outside of Canada. For more details on the collaborative activities for each NCC, see the profiles in Appendix B.

There were many examples of collaboration between PHAC and the NCCs found in the performance data. The purpose of these activities varied widely and pertained to the following:

As shown in performance data, the level of collaboration between PHAC and the NCCs varied across the Centres. NCCID, NCCAH and NCCDH have had ongoing collaborations and involvement with PHAC on common initiatives or to share knowledge. Of note, among all the Centres, NCCID reported the highest level of interaction with PHAC, but internal key informants often mentioned not knowing what value NCCID added to PHAC's activities. The level of reported involvement with PHAC is much lower for NCCMT and NCCHPP, and very limited for NCCEH. Consistently, both the NCCEH and internal key informants acknowledged that there is very little alignment between the work undertaken by this Centre and PHAC. Anecdotal evidence showed that NCCEH interacted more with Health Canada than PHAC on environmental public health issues (e.g., the management of crude oil spill incidents from a public health point of view).

Challenges that hampered collaboration between NCCs and PHAC were reported by key informants and are further discussed in section 4.3.3 of this report.

Benefits of collaborations

External key informants confirmed that collaborations and networks put in place by the NCCs have allowed partners to leverage each other's capacity to achieve results, to share information across regional, professional, and jurisdictional silos, and to bring a diversity of perspectives to a given issue. They explained that the collaborative and trustworthy nature of the NCCs encourages engagement from stakeholders and the development of long-term relationships, helps the NCCs better identify and validate knowledge gaps and needs, and facilitates the sharing of information.

In addition, key informants noted that the NCCs have brought together stakeholders that governmental organizations would not have been able to in the same way. This is particularly noteworthy for NCCAH, which has been successful in engaging with national Indigenous organizations. One reason given by key informants for this strength of the NCCs is that they are seen as entities that do not have a political agenda, and this allows them to be trusted by a wide variety of stakeholders to convene discussions and transfer information between jurisdictions more easily than PHAC could. One key example identified by key informants was the forum Towards TB Elimination in Northern Indigenous Communities, organized in January 2018 by NCCID, with the support of NCCAH, NCCHPP, and NCCDH. This forum brought together Indigenous and non-Indigenous stakeholders from across seven provinces and territories to share experiences on addressing tuberculosis at the community level and discuss intersecting factors, such as the history of colonization policies and social inequities affecting population health.

While key informants generally agreed on the value achieved by NCC collaboration, some external key informants said that the NCCs could have been more strategic and systematic in their engagements to maximize information sharing and resource use for knowledge translation, as well as avoid duplication of effort. Suggestions included increasing their liaisons with existing networks of public health professionals and the three existing provincial public health organizations, as well as taking part in more regional and national conferences, with the idea that personal interactions with the NCCs will grow their audience. That being said, NCC key informants indicated that reductions to their funding allocation over time have reduced their capacity to participate in various events (see section 4.3.1 for further details on resources allocation).

4.2.6 Fostering Collaboration among the NCCs

Collaboration and coordination happen regularly between the NCCs on a variety of initiatives. While collaboration is seen as beneficial and productive when it occurs organically on projects where NCCs have expertise to contribute, the requirement for all six of them on to collaborate on signature projects is seen as diverting limited resources that could be more effectively spent elsewhere.

The contribution agreements require the NCCs to collaborate together on different initiatives. As reported in both interviews and program documents, the NCCs have put in place various mechanisms to foster collaboration among them, including the following:

In addition, some NCCs have adopted additional practices to promote information sharing and collaboration with other centres:

As part of their coordinated approach, all six Centres contribute to an annual NCCPH Knowledge Translation Award and to the Knowledge Translation in Public Health Medicine webinar series, delivered in partnership with the Public Health Physicians of Canada, where each NCC provides content for webinars designed to meet knowledge gaps on specific topics and continuing education needs.

In addition to these initiatives, PHAC directed the NCCs in 2015 to undertake signature collaboration projects involving every Centre, in order to show the value that they could bring to a topic when working together. PHAC directed the NCCs to reserve 6% of their annual budget (funds and in-kind salaries for staff time) to contribute to these six-way collaborations. Since 2015, the following signature projects have been completed:

At the time of the evaluation, the NCCs were working on public health matters related to long-term public health responses to evacuation due to natural disasters in Canada.

The Population Mental Health Promotion signature project resulted in the publication of resources on population mental health promotion for children and youth. A forum was held in 2018, in partnership with PHAC, the Canadian Mental Health Association, the Centre for Addiction and Mental Health, and the Mental Health Commission of Canada, with a wide representation of stakeholders to clarify the roles of public health in addressing mental health issues.

Although the collaborative projects are seen as generating positive benefits by some NCC key informants, all NCC key informants raised concerns about these projects, indicating that it has been very challenging to find themes where every NCC could meaningfully contribute. Some Centres have only had limited participation on a given signature project. There was a widespread sense among the NCCs that collaboration on six-way signature projects is somewhat artificial, and consumes significant time and resources that could be dedicated to other work where they could contribute their expertise. A couple of NCC staff suggested that PHAC could replace its directive on signature collaborations with advice on high-priority topics that two or more NCCs could address together, and even contribute staff resources to assist with collaborative projects of specific interest to PHAC.

Beyond the signature projects and collaborative initiatives involving every NCC, there is evidence in documents and many key informant interviews that each NCC has regularly collaborated with one or two other Centres.Footnote [f] According to most NCC key informants, those collaborations have usually happened in an organic manner, based on initiatives where each NCC could contribute expertise in a meaningful way. Examples include:

NCC key informants generally agreed that when NCCs collaborate, especially in an organic way, they can realize a number of advantages, such as the ability to address a complex public health issue from a variety of perspectives, and to reach a larger audience of stakeholders to confirm knowledge needs and exchange information. That being said, some external key informants encouraged the NCCs to continue to strengthen collaborative ties with each other and to communicate more clearly with their audiences on how NCCs coordinate to foster effective collaboration and avoid duplication among themselves.

4.3 Efficiency

4.3.1 Use of Resources Available

The NCCs are seen to be operating efficiently and using their limited resources in a prudent and innovative manner. Reductions in funding allocation from PHAC to the NCCs have curbed their capacity over time to deliver outputs, but there are opportunities to maximize resource allocation and manage expectations.

The budget allocation from PHAC to the NCCs has declined since the program creation; from an annual allocation of $1.5M in the program's initial years, to a flat budget of $974K per year during the current contribution agreement period (2015 to 2020).

A breakdown of the NCCs' budget, as reported in the contribution agreements and Table 5 below, shows that, on average, about 12% of NCC funding is dedicated to overhead costs (e.g. rent, material and utilities, other payments to host, and performance measurement). As such, the NCCs carry out their core activities with a budget of approximately $857K per year (i.e. 88% of $974K).

As described in section 4.2.1 of this report, and as noted by the majority of external users and in performance data, NCCs have delivered a significant number outputs, both products (published papers, tools, DVDs, and podcasts) and activities (workshops, webinars, conference presentations, and online training) with in their relatively limited budgetFootnote [g].

Table 5: Distribution of NCCs Annual Budget
NCCs Budgeted Operating Expenses Percentage Allocated
Salary, external consultants, contractors 76%
NCC Collaborative Work 6%
Travel 5%
Conference registration, meetings, events 1%
Total Operating Expenses 88%
NCCs Budgeted Overhead Expenses Percentage Allocated
Rent and utilities 5%
Other (material; equipment, other contractors) 4%
Performance Measurement 2%
Other Payments to Host 1%
Total Overhead Expenses 12%

Source: Appendix B: Budget, NCCPH Contribution Agreements 2015-2020

Many NCC-related key informants from the Centres, host organizations, as well as board members, commented that budget reductions have created challenges for them in maintaining consistent staffing levels over the five-year term of their contribution agreement, given that the human resources policies and collective agreements of their hosts require NCCs to index salaries, despite a flat annual budget. Given this constraint, NCC key informants and several external users noted that Centres often struggle with managing expectations while trying to maintain capacity to deliver on their ambitious mandate. As such, it is challenging for the NCCs to take on new work beyond what is anticipated in their work plans without compromising the achievement of work they are accountable for delivering. Some NCC-related key informants noted that further reductions in the budget would likely result in the NCCs not having the necessary resources to fulfil their mandate, as currently defined.

Although each Centre can attract additional resources through contracts with other organizations, data from the performance measurement questionnaires shows that PHAC's contribution remains the largest source of NCC funding Footnote [h]. Four of the NCCs (NCCAH, NCCMT, NCCID, NCCDH) received amendments to their original 2015-2020 agreements to provide them additional funding to carry out specific projects for PHAC. The additional funds ranged from $90K (2% additional funds) for NCCDH to expand the North-West Regional Health Equity Forum and create a training program to build knowledge in emerging areas of health equity, to $381K (7% additional funds) for NCCMT to take on additional work to develop an interactive online resource to deliver tools and information on Ebola. The amendments resulted in three of six NCC contribution agreements (NCCAH, NCCID, NCCMT) exceeding a $5 million allocation over the five-year period of 2015-2020.

In terms of leveraging from external sources, the NCCs have leveraged, on average, about only 3% of their funding through outside contracts or resources provided by their host institutions. The amount of funds leveraged has varied widely over the years and between the NCCs. For example, NCCAH obtained the most additional funds from contracts of all the Centres (e.g., 10% of its funding in 2017-18) while NCCEH reported no additional funds or in-kind support in both 2016-17 and 2017-18. Although all NCCs indicated they receive various types of in-kind support from their respective hosts (e.g., legal services, human resources support), the dollar value of in-kind support was not consistently reported across all NCCs, therefore it was not feasible to report on value or leveraging of in-kind support from host organizations.

As noted by many NCC and external key informants and program documents, NCCs have implemented a variety of innovative approaches or efficient practices to maximise their outputs. These have included using student placements to support capacity, experimenting with the use of artificial intelligence to determine relevance of information, using social media, including video production, to expand reach, and launching online meeting platforms to support networking. These tools have been especially important, given budget limitations on travel, although it is recognized that internet accessibility limits persist in some regions. That being said, the NCCs highlighted that face-to-face meetings are still key to building relationships and recognition with stakeholders (e.g., participation at national conferences, NCCMT's outreach meetings with Masters of Public Health programs across Canada).

While many external and NCC-related key informants have called for an increase in funding allocation to the NCCs, a few other internal and external key informants have suggested alternative options. One such approach is a revision to the formula for funding allocation, in order to base the level of funding for each NCC on different criteria, such as level of relevance, level of performance, scope of the work, or level of maturity.

4.3.2 Benefits and Challenges of NCCPH Program Model

The NCC program model has proven to be beneficial in numerous ways, such as supporting the NCCs' efficient operations while also providing advantages to the host organization, and giving the NCCs the ability to remain apolitical and connected on-the-ground. However, it has not been without challenges, as the NCCs have had to follow the procedures and policies of two organizations: those of their host organization, and the rules of the contribution agreement.

Host model

NCCPH program contribution agreements are established between PHAC and a host institution, rather than the NCC per se, as they are not legal entities. In the case of NCCAH, NCCID, NCCMT, and NCCDH, the contribution agreement is with a university, while for NCCEH and NCCHPP it is with a provincial public health organization. Since the inception of the program, the host institution has remained the same for all Centres, with the exception of NCCID, which moved from the International Centre for Infectious Diseases to the University of Manitoba in July 2015.

According to the majority of key informants from both the NCCs and the host organizations, the hosting model is beneficial in many ways. For example, it gives the NCCs access to useful infrastructure and support services, such as office space, internet, reference services through open and free access to university library resources, legal and human resources services, links to researchers with relevant expertise, as well as access to students to assist via placements or volunteer work. In addition, some contribute financial resources to advance joint projects with the NCCs (e.g., the host and the NCC may partner to organize conference on campus).

Moreover, the host model offers the ability to leverage the networks and expertise of the host organization. The Centres can also benefit from the host's reputation and national exposure to enhance both their profile and reach.

There are, however, challenges related to the current model. As explained by NCC key informants, the six NCCs operate as a "business within a business", meaning that the Centres have to follow the administrative procedures of their host organization, while also following the requirements of the contribution agreement with PHAC. For example, some Centres need to follow the approval processes and policies of their host organization for travel authorization, or for obtaining contracts and funding from external sources.

Contribution agreement model

The NCCs were created to be a focal point at the federal, provincial, and territorial levels for collaboration with local governments, academia, public health professionals, and non-governmental organizations. A contribution agreement model was chosen to support the program mandate.

NCCs and most external key informants noted that the NCC model of having Centres at arm's length from the Government of Canada is beneficial, as their work plans are not subject to a political agenda and it puts the Centre in a good position to act as a third party between different levels of government, communities, and researchers, thus breaking the silos of the Canadian health system. It was also noted by some external key informants that the NCC model has allowed the Centres to remain more nimble than governments and more connected to developments at the front line of public health, given their regular interactions with professionals and through guidance from their external advisory boards, which have a broad perspective on strategic directions in the field.

Reporting requirements

NCCs key informants viewed the current reporting requirements for the contribution agreements as cumbersome and time-consuming. Currently, the NCCs are required to submit a work plan at the beginning of the year and report annually on the fulfillment of their work plan by completing a performance measurement questionnaire and a year-end progress report. NCCs are also required to provide an oral mid-year progress report via teleconference with PHAC. This was initiated in 2015-16 as a written requirement, then changed in 2017-18 to a teleconference. NCCs are also required to undertake periodic evaluation work, either internally or performed by external consultants, in addition to a program evaluation undertaken by PHAC to fulfill requirements of the Financial Administration Act.

Many NCC key informants indicated that the performance measurement questionnaire (implemented in 2015-16) was a positive initiative, as it allows them to report on their achievements in a systematic manner. However, they noted that completing both the performance measurement questionnaire and the year-end progress report, as well as submitting annual updated work plans, is time-consuming, in part because the reports use very different formats, but also because of some overlap in terms of information to include.

On average, NCCs dedicate 2% of their budget to performance measurement, which does not account for time spent by staff to complete the different reports. It was noted that, although the funding granted to the NCCs has been reduced since their creation, the reporting requirements have not changed, with the exception of the Mid-Year Progress Report, which has been provided orally since 2017-18, as opposed to in writing.

Several NCC key informants noted they were unaware of what PHAC does with the performance data submitted by the NCCs and to date, they have not received feedback on the value or use of this information.

The review of performance data for this evaluation revealed that the NCCs had provided a significant amount of useful data to support program accountability. In particular, the evaluation team found useful data in the performance measurement questionnaires and evaluations led by the NCCs. The end-of-year progress reports were of limited use for the evaluation, as the performance measurement questionnaires already provided essential information to support assessment of NCCPH program relevance, performance and efficiency. The review also identified areas for improvement in regards to the quality of data reported. Mainly, there was a lack of clarity and consistency on how NCCs outputs were measured and reported across Centres and over time. It was difficult to roll up data from every NCC into a program-level assessment due to inconsistent information reported across Centres.

A few key informants from PHAC suggested a review of reporting requirements to make them more in line with the maturity of the NCCs and their level of funding.

4.3.3 Coordination with PHAC

NCCs expressed concerns that PHAC priorities are not communicated to them in a timely and systematic manner. This has caused a significant challenge regarding the ability of NCCs to align annual work plan activities with PHAC's and connect with appropriate program areas.

Although the PHAC program office communicates with NCCs on work plan development, both NCCs and PHAC key informants indicated that there is room for improvement on information sharing. For example, interviews with PHAC key informants revealed a lack of awareness of the NCCs' work and potential value they could bring to the Agency. However, it was noted that PHAC has continuously faced challenges in finding a way to engage and collaborate with the NCCs. Likewise, NCCs have reported that navigating the Agency is difficult for them. They have faced challenges in knowing who they should contact in program areas and in managing relationships with PHAC contacts given staff turnover.

All NCCs expressed concerns that PHAC input in their work plan was brief, lacking in strategic focus, and often provided late in its development, thus making it difficult to incorporate into activities for the upcoming fiscal year. They also noted that they did not necessarily receive clear and systematic feedback on PHAC's priorities. Some Centres took it upon themselves to proactively undertake research on PHAC activities from publicly available sources (e.g., Report on Departmental Plans and Priories, the PHAC web site). This was viewed as affecting their ability to be proactive in that it was difficult to be ahead of the game if they heard about PHAC's priorities at the same time as the general public.

NCCs and most internal key informants noted that the relationship with PHAC is principally that of funder-recipient and that it would be beneficial to develop a two-way partnership and better communication going forward, in order to maximize the benefits of the program investment and adequately align levels of effort. The majority of internal and NCC key informants noted that more could be done to ensure that PHAC program areas engage with NCCs to leverage their capacity where it has been judged relevant to do so, especially where work areas overlap. For example, numerous external key informants noted that PHAC could benefit from the NCCs' ability to effectively convene a broad variety of key stakeholders and their understanding of the public health system as a whole.

PHAC has recently put several initiatives in place to foster information sharing with the NCCs. For example, the program secretariat has occasionally hosted Health Portfolio knowledge mobilization sessions in Ottawa to help connect the NCCs with potential federal knowledge users. More recently, PHAC has mandated six Director Generals from various program areas within the Agency to liaise with a specific NCC. In most cases, the Director General is responsible for programs that closely align with the public health issue covered by the NCC with whom they are liaising. Launched in February 2018, this initiative aims to reinforce horizontal relationships, provide strategic assessment on NCC activities at PHAC's senior level, and actively share information with NCCs on bilateral priorities, issues, and opportunities.

Although it is too early to assess results from this initiative, most internal and NCC key informants mentioned that this approach has shown promise for creating closer collaboration between NCCs and PHAC. Some executive leads have noted that developing a better relationship with the NCCs will take some time and that the end goal of this initiative is still not clear.

5.0 Conclusions

Overall, there is a continued need for knowledge translation services to make evidence-based information accessible and useful to public health professional in support of decision making and policy making. In addition, in Canada's multijurisdictional health system, there continues to be a need to foster networks across the system. NCCs tend to occupy a unique niche focused on translating evidence-based knowledge in a very practical manner to support public health professionals across the countries.

NCC activities clearly align with the Government of Canada and PHAC overall priorities of strengthening public health capacity and science leadership. However, some key informants raised concerns about the program renewal along the six themes selected in 2004 and suggested a need to re-assess the themes, in accordance with evolving public health priorities. Also, although there has been an evolution in the mandate of NCCPH program, from focusing on strengthening public health capacity and targeting local level professions, to supporting decision making and policy making at all levels of the public health system, not all NCCs have reflected this change in their collaboration.

In terms of outcomes, the evaluation shows that the NCCs have made practice-relevant knowledge resources useful and accessible for a broad range of public health professionals across Canada. Their knowledge synthesis products and activities are valued and used to support evidence-informed decision making and policy making. In many instances, the NCCs have responded to emerging needs, even if the current process for developing work plans and the NCCs' limited budget often made it challenging for them to be nimble.

The ability of NCCs to foster relationships and collaboration with a range of stakeholders across the system was seen as a significant source of added value. Collaboration happened regularly between two or three NCCs on a variety of initiatives where each NCC had valuable expertise to contribute. While those collaborations are seen as beneficial, the requirement for all six NCCs to collaborate on signature projects is seen as diverting limited resources.

In terms of efficiency, the Centres have delivered a significant number of outputs with a relatively limited budget. However, their ability to deliver these outputs has been curbed as their funding from PHAC has declined over time. The NCCPH program model has been beneficial in helping the Centres operate efficiently, but is not without challenges as the NCCs have had to follow the procedures and policies of both their horst organizations and their contribution agreement. The model of having Centres at arm's length from the Government of Canada is seen as giving the NCCs the ability to remain apolitical and connected on-the-ground. Concerns were raised over reporting requirements that are seen as burdensome, considering the level of funding allocated to the NCCs.

Finally, there were limited interactions between PHAC and the NCCs, making it challenging for the NCCs to align their annual work plans to areas of interest for PHAC. The designated executive lead initiative launched by PHAC in February 2018 has shown promise for creating closer collaborations between NCCs and PHAC, although it is too early to assess the results.

6.0 Recommendations

Recommendation 1

Develop a collaborative two-way partnership between PHAC and the NCCs to promote greater awareness of each other's work and increase collaboration in areas of common interest.

Within PHAC, there has been a general lack of understanding of the role, mandate, and value of NCCs. Also, NCCs have expressed concerns that PHAC priorities are not communicated to them in a timely and systematic manner. This has caused a challenge for NCCs in aligning annual work plans to areas of interest for PHAC and in connecting with appropriate program areas. In light of this, PHAC should build on the initial efforts made by the executive lead initiative and engage more strategically and regularly with the NCCs, in order to leverage each other's knowledge, resources, and networks to advance common goals.

Recommendation 2

Explore opportunities, as part of the contribution agreement renewal, to ensure that each NCC remains relevant to emerging public health sector needs in terms of issues addressed, range of collaborations, and targeted audiences.

The NCCPH program mandate has evolved since the last evaluation, from focusing on strengthening public health capacity and targeting local level professionals, to supporting decision making at all levels of the public health system. Not all Centres have reflected this shift in the program mandate in their target audience and their collaborations, as they continue to focus on local practitioners. As well, NCCPH program topics may have been relevant when selected over 13 years ago, but there was a concern that those themes had never been revised since the program was created, and that knowledge translation efforts should look at other issues that are aligned with emerging global public health priorities. Given that the current funding cycle ends in 2020, PHAC should explore opportunities to ensure that the NCCs remain relevant, including clarifying directions given to them, in order to achieve the current program mandate and fulfill the evolving needs of the public health sector.

Recommendation 3

Explore options for maximizing resource allocation to the NCCs and allow them to use those resources more efficiently to fulfill their core mandate. This includes:

The reduction in funding allocation from PHAC to the NCCs has curbed their capacity to deliver outputs. In this context, PHAC should investigate opportunities to maximize resource allocation and manage expectations. More specifically, PHAC should revisit the requirement for all six NCCs to collaborate on signature projects, given that they are collaborating with each other through other means. PHAC should also provide NCCs more flexibility to address emerging issues as they arise, without having to make significant revisions to their work plans and without affecting accountability for results achieved during the year. PHAC should streamline performance measurement requirements, with a particular emphasis on providing useful data to inform ongoing program management and evaluation, while also keeping the NCCs' reporting burden in line with their current level of funding allocation.

Appendix A - NCCPH Program Logic Model

Figure 1 - Appendix A - NCCPH Program Logic Model
Figure 1- Text Description

This figure depicts the components of the NCCPH program logic model which is a visual illustration of the program's inputs (resources), activities and expected outcomes.

The National Collaborating Centres for Public Health's key input is Agency A-Based funding. This input is expected to produce the following activity/output: develop and manage contribution agreements with National Collaborating Centres. This activity/output is expected to lead to the follow immediate, intermediate, and ultimate outcomes; immediate outcome: Public Health Organizations have resources and structures to collaborate across the Public Health system to address knowledge-related needs and gaps required to strengthen evidence-informed decision-making; intermediate outcomes: Products and tools to translate and disseminate public health knowledge are developed and shared through a variety of networks; ultimate outcomes: (i) knowledge is mobilized across networks and sectors to fill information gaps, build evidence-informed decision-making capacity and respond to emerging health issues in the Public Health system, (ii) public health professionals, policy makers and Canadians have access to reliable, actionable Public Health data, and (iii) information and Canadians are protected and empowered to improve their health with mechanisms in place to enable the Public Health system to work collaboratively to address existing and emerging public health infrastructure issues.

Appendix B - National Collaborating Centre Profiles

1. NCC for Environmental Health (NCCEH)

2. NCC for Aboriginal Health (NCCAH)

3. NCC for Infectious Diseases (NCCID)

4. NCC for Methods and Tools (NCCMT)

5. NCC for Healthy Public Policy (NCCHPP)

6. NCC for Determinants of Health (NCCDH)

Appendix C - Evaluation Description

The scope of the evaluation of the National Collaborating Centres for Public Health (NCCPH) covered the period from April 2014 to September 2018. The evaluation reviewed the relevance, performance, and efficiency of the NCCPH program during this time, aligning with requirements described in the Treasury Board of Canada's 2016 Policy on Results.

The evaluation was designed to address the intended outcomes of PHAC's NCCPH program, and provides insight into the issues and questions presented in the following table.
Core Issue Evaluation Question
Relevance
Issue #1. Continued Need.
  • 1. What program model and institutional hosts are required to strengthen the public health capacity in 2018?
    • Which priorities should be tackled?
    • To what extent does the NCCPH meet those needs?
    • What are the possible areas for improvement?
  • 1.1 To what extent has the level of public health capacity in Canada changed in the last five years?
  • 1.2 Considering these changes, to what extent is the NCCPH still appropriate in terms of:
    • roles and objectives
    • program model
    • scope of work
    • issues addressed and activities
Issue #2: Alignment with Federal Roles and Responsibilities.
  • 2. To what extent is the role of the NCCs understood internally and externally?
  • 3. Do the scope and activities of the NCCs duplicate, overlap, or complement activities of other stakeholders, including PHAC?
Issue #3: Alignment with Government of Canada Priorities

4. To what extent do NCC priorities and the NCCPH align with PHAC priorities and objectives?

  • To what extent do NCC priorities align with public health sector priorities?
Performance
Issue #4: Effectiveness
  • 5. To what extent is there evidence of contribution to achieving expected outcomes?
  • 6. To what extent have the NCCs achieved their objective of promoting the use of knowledge for evidence-informed decision making?
    • 6.1 To what extent have knowledge products and activities reached public health practitioners, program managers, and policy makers?
    • 6.2 What evidence is there of use of NCC knowledge translation products to support decision making?
    • 6.3 To what extent is there evidence of the usefulness of NCC knowledge products and activities? (i.e., timeliness and relevance for supporting decision making)
    • 6.4 To what extent have knowledge translation, synthesis, and exchange training activities resulted in improved knowledge?
  • 7. To what extent are NCCs effective at addressing emerging public health priorities?
Issue #5: Efficiency and Economy
  • 8. What are the mechanisms in place to allow collaboration and alignment of priorities between PHAC and NCCs? How effective are these mechanisms and what are the potential areas for improvement?
  • 9. To what extent is NCCPH efficient?
    • 9.1 To what extent are the NCCs leveraging other resources (financial, collaboration, and expertise and knowledge of other partners);
    • 9.2 What is the level of administrative cost of the program, when compared to total program expenditures?
    • 9.3 Are there alternative models for public health mobilization?
      • How do they compare?
      • Can they enhance the NCCPH approach?
    • 9.4 What best practices have been put in place to achieve expected outcomes?
  • 10. Overall, to what extent does the NCCPH represent value for money?

Data Collection and Analysis Methods

Sources of information used in this evaluation included a document review, financial data review, and key informant interviews. Data was analyzed by triangulating information gathered from the different sources and methods described below.

The use of multiple lines of evidence and triangulation were intended to increase the reliability and credibility of the evaluation findings and conclusions.

Footnotes

Footnotes 1

When referring to the NCCPH program, this encompasses all aspects of PHAC management of the program, including the NCCPH Program Secretariat. When referring to the NCCs, this includes the Centres themselves, their activities, and their personnel.

Return to footnote [a] referrer

Footnotes 2

When referring to the NCCPH program, this includes all aspects of PHAC's management of the program, including the NCCPH Program Secretariat. When referring to the NCCs, this includes the Centres themselves, their activities, and their personnel.

Return to footnote [b] referrer

Footnotes 3

Based on data reported by each NCC in their performance measurement questionnaires from 2015-16 to 2017-18. The questionnaire requests that each Centre report on the organization they collaborated or partnered with during the year, as well as the name of the project where the collaboration occurred.

Return to footnote [c] referrer

Footnotes 4

This is not an exhaustive list and does not include an assessment of all organizations involved in knowledge, synthesis, translation, and exchange related to the specific topic areas of the Centres.

Return to footnote [d] referrer

Footnotes 5

As the performance measurement questionnaires were established part-way through the period covered by this evaluation, figures are provided here to give a sense of the importance of collaborations, but do not provide a full picture. Collaborations in 2014-15 have not been included, and some NCCs may not have reported all their collaborative activities in their self-reported questionnaires. It should also be noted that the count of collaborations does not reflect the variations in their characteristics; e.g., their length of time or outputs.

Return to footnote [e] referrer

Footnotes 6

Collaborations involving less than the six NCCs are not included in the 6% of the budget set aside for collaborative work involving all NCCs.

Return to footnote [f] referrer

Footnotes 7

It is not feasible to draw conclusions on cost per output or undertake a comparison of outputs between NCCs. This is due to the broad variance in the type, nature, and scope of outputs produced by each NCC and the potential differences in how outputs are reported by each NCC. For instance, some NCCs report every oral talk given at a particular conference, but not all NCCs use the same methods to report on conference participation.

Return to footnote [g] referrer

Footnotes 8

It is possible that the performance measurement questionnaire did not fully reflect all additional source of funding the NCCs may have been able to attract over and above PHAC's funding allocation.

Return to footnote [h] referrer

Footnotes 9

National Collaborating Centre for Methods and Tools. (2018). Build your CV with McMaster University-certified Learning Modules. Retrieved from: https://www.nccmt.ca/capacity-development/certified-modules

Return to footnote [i] referrer

Endnotes

Endnotes 1

Health Canada. (2004). Learning from SARS: Renewal of Public Health in Canada. Retrieved from: https://www.canada.ca/content/dam/phac-aspc/migration/phac-aspc/publicat/sars-sras/pdf/sars-e.pdf

Return to footnote [1] referrer

Endnotes 2

Public Health Agency of Canada. (2014). Evaluation of the National Collaborating Centres for Public Health Program 2008-2009 to 2013-2014. Retrieved from: https://www.canada.ca/en/public-health/corporate/mandate/about-agency/office-evaluation/evaluation-reports/evaluation-national-collaborating-Centres-public-health-program-2008-2009-2013-2014.html

Return to footnote [2] referrer

Endnotes 3

Public Health Agency of Canada. (2017). Supplementary Information Tables 2017-18 Departmental Plan: Public Health Agency of Canada. Retrieved from: https://www.canada.ca/en/public-health/corporate/transparency/corporate-management-reporting/reports-plans-priorities/2017-2018-supplemental-information-tables.html

Return to footnote [3] referrer

Endnotes 4

Public Health Agency of Canada. (2017). Supplementary Information Tables 2017-18 Departmental Plan: Public Health Agency of Canada. Retrieved from: https://www.canada.ca/en/public-health/corporate/transparency/corporate-management-reporting/reports-plans-priorities/2017-2018-supplemental-information-tables.html

Return to footnote [4] referrer

Endnotes 5

Jansen et al. (2015). Health Research Policy and Systems, Conditions for Sustainability of Academic Collaborative Centres for Public Health in the Netherlands: a mixed methods design. Retrieved from: https://health-policy-systems.biomedcentral.com/articles/10.1186/s12961-015-0026-7

Return to footnote [5] referrer

Endnotes 6

DeSalvo KB, Wang YC, Harris A, Auerbach J, and Koo D, O'Carroll P. (2017). Public Health 3.0: A Call to Action for Public Health to Meet the Challenges of the 21st Century. Retrieved from: https://www.cdc.gov/pcd/issues/2017/17_0017.htm

Return to footnote [6] referrer

Endnotes 7

Stefan K. Lhachimi, Malgorzata M. Bala, Giedrius Vanagas. (2016). Evidence-Based Public Health Retrieved from: https://www.hindawi.com/journals/bmri/2016/5681409/

Return to footnote [7] referrer

Endnotes 8

Brownson RC, Fielding JE, Maylahn CM. (2014). Evidence-based Decision Making to Improve Public Health Practice. Front Public Health. Retrieved from: https://uknowledge.uky.edu/frontiersinphssr/vol2/iss2/2/

Return to footnote [8] referrer

Endnotes 9

Canadian Institute of Health Research. (2013). Evaluation of Canadian Institute of Health Research Knowledge Translation Funding Program. Retrieved from: http://www.cihr-irsc.gc.ca/e/47332.html

Return to footnote [9] referrer

Endnotes 10

National Collaborating Centres for Healthy Public Policy. (2013). Structural Profile of Public Health in Canada Retrieved from: http://www.ncchpp.ca/710/Structural_Profile_of_Public_Health_in_Canada.ccnpps

Return to footnote [10] referrer

Endnotes 11

Public Health Agency of Canada. (2013). Strategic Horizons 2013-2018. Retrieved from: http://publications.gc.ca/collections/collection_2014/aspc-phac/HP55-1-2013-eng.pdf

Return to footnote [11] referrer

Endnotes 12

Office of Chief Science Officer. (2018). Canada's Chief Science Officer Letter to Prime Minister and Minister of Health on First 100 days in Office. Retrieve from: https://www.ic.gc.ca/eic/site/052.nsf/eng/00007.html

Return to footnote [12] referrer

Endnotes 13

Innovation, Science and Economic Development Canada. (2018). Canada's Science Vision. Retrieved from : http://www.ic.gc.ca/eic/site/131.nsf/eng/h_00000.html

Return to footnote [13] referrer

Endnotes 14

Office of the Minister of Science. (2017). Canada's Fundamental Science Review: Investing in Canada's Future, Strengthening the Foundations of Canadian Research. Retrieved from: http://www.sciencereview.ca/eic/site/059.nsf/eng/home

Return to footnote [14] referrer

Endnotes 15

National Collaborating Centres for Public Health. (no date). About Us. Retrieved from: http://nccph.ca/about-us/

Return to footnote [15] referrer

Endnotes 16

For more information, see Publications retrieved from:

  • National Collaborating Centre for Infectious Diseases: https://nccid.ca/publications/
  • National Collaborating Centre for Aboriginal Health: https://www.nccah-cnsa.ca/34/Publications.nccah
  • National Collaborating Centre for Environmental Health: http://www.ncceh.ca/content/publications-and-products;
  • National Collaborating Centre for Methods and Tools: https://www.nccmt.ca/impact/publications;
  • National Collaborating Centre for Determinants of Heath: http://nccdh.ca/resources/nccdh/;
  • National Collaborating Centre for Healthy Public Policy: http://ncchpp.ca/73/Publications.ccnpps.

Return to footnote [16] referrer

Endnotes 17

Statistics Canada. (2018). Annual Demographic Estimates: Canada, Provinces and Territories (Total Population only). Retrieved from: https://www150.statcan.gc.ca/n1/en/pub/91-215-x/91-215-x2018001-eng.pdf?st=qval7n9R

Return to footnote [17] referrer

Endnotes 18

Public Health Agency of Canada. (2016). The Chief Public Health Officer's Report on the State of Public Health in Canada 2016 - A Focus on Family Violence in Canada. Retrieved from: https://www.canada.ca/en/public-health/services/publications/chief-public-health-officer-reports-state-public-health-canada/2016-focus-family-violence-canada.html

Return to footnote [18] referrer

Endnotes 19

Public Health Agency of Canada. (2017). The Chief Public Health Officer's Report on the State of Public Health in Canada 2017 - Designing Healthy Living. Retrieved from https://www.canada.ca/en/public-health/services/publications/chief-public-health-officer-reports-state-public-health-canada/2017-designing-healthy-living.html

Return to footnote [19] referrer

Endnotes 20

Public Health Agency of Canada. (2016). The Chief Public Health Officer's Report on the State of Public Health in Canada 2016 - A Focus on Family Violence in Canada. Retrieved from: https://www.canada.ca/en/public-health/services/publications/chief-public-health-officer-reports-state-public-health-canada/2016-focus-family-violence-canada.html

Return to footnote [20] referrer

Endnotes 21

Public Health Agency of Canada. (2017). The Chief Public Health Officer's Report on the State of Public Health in Canada 2017 - Designing Healthy Living. Retrieved from: https://www.canada.ca/en/public-health/services/publications/chief-public-health-officer-reports-state-public-health-canada/2017-designing-healthy-living.html

Return to footnote [21] referrer

Endnotes 22

Employment and Social Development Canada. (2018). Indigenous Early Learning and Child Care Framework. Retrieved from: https://www.canada.ca/en/employment-social-development/programs/indigenous-early-learning/2018-framework.html

Return to footnote [22] referrer

Endnotes 23

Ontario Ministry of Health and Long-Term Care. (2018). Relationship with Indigenous Communities Guideline, 2018. Retrieved from: http://health.gov.on.ca/en/pro/programs/publichealth/oph_standards/docs/protocols_guidelines
/Relationship_with_Indigenous_Communities_Guideline_en.pdf

Return to footnote [23] referrer

Endnotes 24

National Collaborating Centre for Aboriginal Health. (2015). Review of Core Competencies for Public Health: An Aboriginal Public Health Perspective. Retrieved from: https://www.nccah-ccnsa.ca/495/Review_of_Core_Competencies_for_Public_Health__An_Aboriginal_Public_Health_Perspective.nccah?id=145

Return to footnote [24] referrer

Endnotes 25

National Collaborating Centre for Infectious Diseases (2018). Notifiable Diseases Database: Factsheet. Retrieved from: https://nccid.ca/publications/notifiable-diseases-database-factsheet/

Return to footnote [25] referrer

Endnotes 26

National Collaborating Centre for Infectious Diseases. (2018). Featured Antimicrobial Stewardship Tools. Retrieved from: https://nccid.ca/wp-content/uploads/sites/2/2018/03/Featured-AMS-Tools.pdf

Return to footnote [26] referrer

Endnotes 27

National Collaborating Centre for Infectious Diseases (2018). Antibiotic Awareness Week; For Providers; Viral Prescription Pads. Retrieved from: https://antibioticawareness.ca/

Return to footnote [27] referrer

Endnotes 28

National Collaborating Centre for Infectious Diseases. (2018). Featured Antimicrobial Stewardship Tools. Retrieved from: https://nccid.ca/wp-content/uploads/sites/2/2018/03/Featured-AMS-Tools.pdf

Return to footnote [28] referrer

Endnotes 29

National Collaborating Centre for Methods and Tools. (2018). Build your CV with McMaster University-certified Learning Modules. Retrieved from: https://www.nccmt.ca/capacity-development/certified-modules

Return to footnote [29] referrer

Endnotes 30

Public Health Agency of Canada. (2017). The Chief Public Health Officer's Report on the State of Public Health in Canada 2017- Designing Healthy Living. Retrieved from: https://www.canada.ca/en/public-health/services/publications/chief-public-health-officer-reports-state-public-health-canada/2017-designing-healthy-living.html

Return to footnote [30] referrer

Endnotes 31

National Collaborating Centre for Healthy Public Policy. (no date). Health Impact Assessment. Retrieved from: http://www.ncchpp.ca/553/Publications.ccnpps?id_article=1711

Return to footnote [31] referrer

Endnotes 32

National Collaborating Centre for Healthy Public Policy. (2017). Online Continuing Training Course - A Framework for Analyzing Public Policies. Retrieved from: http://nccdh.ca/resources/entry/lets-talk-public-health-roles

Return to footnote [32] referrer

Endnotes 33

National Collaborating Centre for Determinants of Health. (2013). Let's Talk: Public Health Roles for improving Health Equity. Retrieved from: http://nccdh.ca/resources/entry/lets-talk-public-health-roles

Return to footnote [33] referrer

Endnotes 34

National Collaborating Centre for Health Public Policy. (2017). A Framework for Supporting Action in Population Mental Health. Retrieved from: http://www.ncchpp.ca/553/Publications.ccnpps?id_article=1711

Return to footnote [34] referrer

Endnotes 35

Public Health Ontario. (2016). Health Equity Indicators for Ontario Local Public Health Agencies, User Guide. Retrieved from: https://www.publichealthontario.ca/en/ServicesAndTools/Documents/LDCP/LDCP_user%20guide.pdf

Return to footnote [35] referrer

Endnotes 36

Ontario Ministry of Health and Long Term Care. (2018). Health Equity Guideline. Retrieved from http://health.gov.on.ca/en/pro/programs/publichealth/oph_standards/docs/protocols_guidelines/
Health_Equity_Guideline_2018_en.pdf

Return to footnote [36] referrer

Endnotes 37

Nova Scotia Department of Health and Wellness. (no date). Nova Scotia Public Health Health Equity Protocol. Retrieved from: https://novascotia.ca/dhw/publichealth/documents/05-Health-Equity-Protocol.pdf

Return to footnote [37] referrer

Endnotes 38

National Collaborating Centre for Determinants of Health. (2012). Improving health equity in Saskatoon: From data to action. Retrieved from: http://nccdh.ca/resources/entry/casestudy-SK

Return to footnote [38] referrer

Endnotes 39

National Collaborating Centre for Determinants of Health. (2015). Learning from Practice: Equity in Influenza Prevention in Saskatoon. Retrieved form" http://nccdh.ca/resources/entry/learning-from-practice-equity-in-influenza-prevention-in-saskatoon

Return to footnote [39] referrer

Endnotes 40

Manitoba Chief Public Health Officer. (2018). Chief Provincial Public Health Officer Position Statement on Health Equity. Retrieved from: https://www.gov.mb.ca/health/cppho/docs/ps/health_equity.pdf

Return to footnote [40] referrer

Report a problem or mistake on this page
Please select all that apply:

Thank you for your help!

You will not receive a reply. For enquiries, contact us.

Date modified: