ARCHIVED - Formative Evaluation of the National Collaborating Centres for Public Health Program (NCCPH)

Appendix E: Detailed Results by Evaluation Issue

Issue 4:  Relevance

Issue #4 relates to the evaluation question – to what extent does the NCCPH Program continue to be consistent with federal government and PHAC priorities?  Findings are presented under the following section headings:

• Alignment with Government of Canada, PHAC and public health priorities; and
• Continued relevance of current model.

Current Government of Canada, PHAC and Public Health Priorities

Description

The objective of the NCCPH Program supports the Program Activity Architecture of: “Healthier population by promoting health and preventing disease and injury”. ^{Footnote 40} The goal of the Promotion of Population Health Contribution is to increase the capacity of individuals and communities to maintain and improve their health by:

• Building community capacity;
• Stimulating knowledge development and dissemination; and
• Partnership building/intersectoral collaboration.^{Footnote 41}

These three Promotion and Population Health outcomes are in part linked to the intermediate outcomes outlined in the NCCPH logic model, including:

• Increased availability of knowledge for evidence-based decision making in public health;
• Increased use of evidence to inform public health programs, policies and practices;
• Partnerships developed with external organizations; and
• Mechanisms and processes to access knowledge.

The evaluators could find no reference to the NCCPH Program in PHAC’s 2008-2009 Report on Plans and Priorities^{Footnote 42}, perhaps because the focus of this document is on the direct activities of PHAC.  However, the document identifies a range of public health priorities that are clearly consistent with NCC content areas, such as infectious diseases prevention and control, and strengthened public health capacity.  The importance of knowledge development and exchange, as well as evidence informed policies and interventions are highlighted in relation to Healthy Living, Overweight and Obesity Prevention, and Mental Health.

The NCCPH Program is referred to in the 2005 Conference of Deputy Ministers of Health report, Improving Public Health System Infrastructure in Canada: The Report of the Task Group on Strengthening Public Health System Infrastructure.^{Footnote 43}  In this report, the need to make public health a top priority, including improving infrastructure and increasing capacity at all levels, is reiterated.  As described in the report, it is anticipated the focus of the National Collaborating Centres on knowledge translation, applied research and training will encourage collaboration, integration, and a system-wide perspective to public health infrastructure development.

The F/P/T governments have articulated health goals for Canada, including an overarching goal of aspiration in which every person in Canada is as healthy as they can be – physically, mentally, emotionally, and spirituallyFootnote 44.  A population and determinants of health approach is evident in the specific goals related to: basic needs (social and physical environments); belonging and engagement; healthy living; and a strong system for health that is coordinated across the country, responsive to disparities in health status and offers timely, appropriate care.

The importance of strengthening Canada’s knowledge base is advocated beyond the health sector.  In May 2007, Canada’s Prime Minister unveiled the science and technology strategy, Mobilizing Science and Technology to Canada's Advantage.^{Footnote 45}  In this strategy, it is recognized that Canada must continue to strengthen its knowledge base to be on the cutting edge of developments that lead to benefits at the level of health, environment, society and economy.

Feedback

Perceived alignment

Only PHAC and other national representatives were asked to consider alignment of the NCCPH Program with the Government of Canada’s current priorities and mandate, and PHAC strategic outcomes and goals.  A national representative stated the program is absolutely in alignment particularly in relation to the priorities of science and research and the integration between government and academic communities. 

A representative indicated that knowledge generation is being strengthened for public health, both within and external to PHAC, involving greater engagement with academics across the country.  However, there is a need [that NCCs fulfill] for content experts to be engaged with public health programs.  

National and NCC representatives were asked to comment on alignment of the NCCPH Program with population and public health indicators.  The response was mixed.  Those who agree the Program is aligned with population and public health indicators thought NCCs are a relevant, proactive and appropriate way to address the issues through much need KT and knowledge management.  Those informants who disagreed with Program alignment either questioned what population and public health indicators were meant or suggested that not all public health priorities were being addressed.

Respondents were asked if the six NCC content areas address current public health priorities.  Five informant groups answered in the affirmative.  Specifically, one national representative said the content areas are important and one Advisory Board member said Aboriginal Health is a major priority for Health Canada.

Negative responses related to a lack of understanding about the rationale or logic for choosing the content areas of some or all of the NCCs, e.g., NCCDH is too broad a topic, and isn’t environmental health one of the determinants of health? Was a NCCID needed when this topic is already well covered in Canada?  Why was Aboriginal Health chosen and immigrant health not considered?

Individual comments of concern regarding the current NCCs were:

• NCCs and the products are not useful to front line practitioners;
• NCCs are not in sync with local priorities;
• NCCs are too topic-specific;
• A few NCCs are in place due to political reasons;
• The mandate of one NCC is unclear; and
• Key public health determinants are missing.

Perceived need for additional NCCs

Informant groups were asked if additional NCCs were needed and, if so, what topic areas and locations should be considered.  Respondents from the majority of informant groups (8/10), including from four NCCs, indicated no, uncertainty, or other options for addressing content areas.   Of those indicating uncertainty, the major theme was a need to wait and see, or for existing NCCs to become stabilized and demonstrate their success before additional NCCs were added.  Additional themes related to a need for an organizing framework prior to adding more NCCs, and that additional topics (e.g., chronic disease, injury prevention or emergency preparedness) could be accommodated within the existing structure (although additional funding may be needed). 

While not a prevalent theme, individual respondents representing four informant groups voiced concerns regarding the relevance or focus of one of the six NCCs, and some suggested this NCC could be eliminated.

Respondents from three informant groups suggested the needed for additional NCCs; of these, three respondents mentioned the need for more dollars and/or more support if this were to occur.

Respondents from two informant groups suggested the location of additional NCCs, should they be considered, is irrelevant given the national mandate.  Others who commented on location stated that if additional NCCs were contemplated, then:

• A competition should be held to determine the location, and different models – such as consortiums or virtual coalitions;
• Location should be based on where the best expertise may be accessed; and
• Housing within Canada’s Schools of Public should be considered as a logical fit.

Suggested additional topics that might be appropriate for NCCs, were: 

• Injury prevention,
• Chronic disease prevention;
• Mental health and/or addiction;
• Other population based content areas (e.g., vulnerable populations, elderly, children, ethno-cultural) (x 3 informant groups);
• Emergency preparedness;
• Health economics;
• Healthy childhood development;
• Rural health;
• Urban reform; and
• Workplace health.

Suggestions for enhancement

Representatives from five informant groups suggested revisions to the current six NCCs, including either expansion of NCC topic area/focus or elimination/absorption of centres.  In relation to expanding the focus of NCCs, three informant groups suggested:

• Refocus existing NCCs based on current issues;
• Integrate chronic diseases into existing NCCs;
• Expand current NCCs to include injury prevention and risk assessment tools;
• Integrate early childhood development into NCCAH and NCCDH; and
• Expand the focus of NCCID beyond HIV to include other infectious disease areas (e.g., zoonoses).

In relation to integrating chronic diseases into existing NCCs, two informant groups identified the siloing of chronic diseases as a challenge.

Regarding elimination or absorption of NCCs, two informant groups suggested:

• Eliminate NCCID (x 2 informant groups);
• Incorporate NCCDH into other NCCs; and
• Question NCCMT cutting across all the NCCs.

Two informant groups suggested an organizational framework be selected to determine NCC content areas.  Participants offered the following frameworks for consideration:

• Five core functions of public health (x 2 informant groups);
• Ottawa Charter;
• Based on populations; and
• Determinants of health.

Continued Relevance of Current Model

The NCCPH model may be defined by the following elements:

• Geographically dispersed Centres of Excellence across Canada;
• Located within the umbrella of a host organization with a strong linkage to academia;
• Representing independent legal entities operating at arm’s length from government;
• Funded and managed through the federal government’s Contribution Agreement mechanism;
• Each given a national mandate involving knowledge synthesis, translation and exchange;
• Related to a specific, relevant, priority public health topic area; and
• With an expectation for collaborative effort.

No major changes to the defining elements of this model were suggested by representatives in any informant group.  The KSTE mandate was perceived to be appropriate as it fills a unique niche and is needed for public health; the concept of dispersed centres with a pan-Canadian mandate and expectation for collaboration was supported; and linkage with academic oriented host organizations was supported.  While concerns related to the restrictions evident through the Government of Canada’s processes, no suggestions for changes to the Contribution Agreement mechanism or arm’s length relationship were offered. 

While no changes to the current NCCPH design were recommended, NCC informants did offer some comments and suggestions related to the way the NCCPH is represented in the existing logic model. 
Representatives from three NCCs thought the logic model accurately reflected the NCCPH.  On the other hand, numerous issues and suggestions for enhancement were received from representatives from four NCCs.  Informants identified issues of: organization and logic, understanding, content and language, attribution, and presentation.  Comments received from individuals or NCC staff groups are presented below.

Organization / logic

• The logic model doesn’t have organizations or structures anywhere.  Activities are carried out by organizations and it is not clear who is responsible for what;
• The activities are at the program management level but the outcomes are at the NCC level – there are no clear activities to achieve the outcomes;
• The Advisory Council is not represented – it could be put in as a box with a dotted line to the Agency;
• The logic model does not speak to / include the work of the collective group;
• Activities are not directly linked to outcomes;
• The jump is too great between program and individual NCC – the program seems to be operating at a really high and broad level;
• The logic model is a linear framework and KSTE involves circles and feedback loops; and
• The logic model needs to be integrative, not siloed.

Understanding / definition

• We don’t know the difference between increased opportunities for collaboration and the networking function;
• The definition of decision-making is elusive.  It can refer to who decides on how resources are spent.  The level in which decision-making occurs is dependent on organizational hierarchy.  Decision-making can be at a variety of levels, across the spectrum.  There is formal and informal decision making – the informal refers to people who have influence on the decision maker;
• What is meant by Intersectoral collaboration and how does it apply?; and
• A clearer definition of ‘health portfolio’ would be helpful – who is included in the health portfolio?

Content / placement / wording

• Capacity development is an activity as well as an outcome;
• The model does not go far enough in terms of longer term outcomes – better and more effective programs and services at the local level – work through determinants of health and ultimately improve the health status of the Canadian population and reduce disparities;
• The logic model does not capture all of the outputs or activities; it seems like it does not match what is happening at the individual NCC level;
• Networking can fit in more than one place;
• The most proximate outcomes for NCCs is making information available;
• The outputs don’t reflect the time and energy required to build the network of NCCs;
• Program management fits in more than one place;
• Public health organizations and decision-makers should be included as part of the target audience;
• We’re uncomfortable with the concept that this is focused on public health practitioners only – while we try to influence them, we are also trying to influence organizations, so we’re uncomfortable with separating the concept of practitioners and health units or health clinics as we are not dealing with practitioners in the clinic;
• Our client group includes Medical Health Officers – under reach, suggest public health practitioners and policy makers at the local and regional level in Canada;
• Doing research is an issue; we do not believe that the NCCs are mandated to do research;
• In the second box of immediate outcomes, we have replaced the word ‘application de connaissances (knowledge application)’ with the term ‘transfert de connaissances (knowledge transfer)’; and
• Our language has evolved – under immediate results, instead of ‘data’ it is now ‘strategies’.

Attribution

• Intermediate outcomes are outside of our control; and
• The intermediate outcome of uptake by stakeholders of the knowledge we produce does not depend solely on the NCC’s work.

Presentation

• This may just be aesthetics but is there a reason why the KT box is larger with larger font than the others under immediate outcome?  There are different sized boxes and fonts across the logic model – is this to highlight some areas?
• Why are there single, double and tripling of arrows?  Why are some boxes shaded differently than others?