ARCHIVED - Formative Evaluation of the National Collaborating Centres for Public Health Program (NCCPH)
Appendix E: Detailed Results by Evaluation Issue
Issue 1: NCCPH Design and Delivery
Issue #1 relates to this evaluation question – to what extent is the design and delivery of the NCCPH Program appropriate? Findings are presented under the following section headings:
- NCCPH design and implementation;
- Governance and accountability; and
- Strengths and suggested areas for improvement.
Each section starts with a description of the topic area, followed by feedback received on these topics during the informant interviews and focus groups. In the feedback components within the sections that follow, unless identified as individual comments, themes are presented in general descending order of the frequency with which they were mentioned across 10 informant groups. Each of the six NCCs comprise an informant group, as do Advisory Council (AC) members, NCC Advisory Board members, PHAC and other NCCPH representatives at the national level, and interested or involved parties. In some cases, quotes are provided for illustrative purposes.
NCCPH Design and Implementation
Description of the NCCPH Model
Based on the logic model and other NCCPH documentation, the evaluators determined that the NCCPH Program is defined by the following features that collectively represent a unique niche within Canada’s health system:
- Geographically dispersed Centres of Excellence across Canada;
- Located within the umbrella of a host organization that is research based or has strong linkages with a research centre;
- Representing independent legal entities operating at arm’s length from government;
- Funded and managed through the federal government’s Contribution Agreement mechanism;
- Each given a pan-Canadian mandate involving knowledge synthesis, translation and exchange;
- Related to a specific, relevant, priority public health topic area; and
- With an expectation for collaborative effort.
The conceptualization and early implementation activities of the NCCPH Program are articulated in the Summary Report 2005-2006.Footnote 22 Information from this document was supplemented with further detail obtained through informant interviews.
According to informants, the need to address knowledge synthesis, translation and exchange (KSTE) in public health was identified as early as 2000 by senior administrators working within Health Canada and researchers working in the area of public health. Connections with senior academics in the United Kingdom (UK) served to influence the concept of regionally-based centres that would act as resources for public health, including support with knowledge translation.
In 2001, the Canadian Institutes of Health Research – Institute of Population and Public Health (CIHR-IPPH), in partnership with the Canadian Institute for Health Information – Canadian Population Health Initiative (CIHI-CPHI), undertook a pan-Canadian consultation on population and public health priorities – Charting the Course – resulting in recommendations for three priority areas of action in relation to KSTE for public health.Footnote 23 Subsequent to this, the CIHR-IPPH initiated a literature review in 2002 on the issues and barriers related to KSTE in public health and identified four categories of critical issues that related to the incorporation of research evidence into policy and public health practice:
- Issues of the evidence base – including evidence synthesis;
- Issues of knowledge transfer and exchange;
- Issues of knowledge uptake and utilization; and
- Issues of evaluation – e.g., effectiveness and efficiency of KSTE strategies, and impact on population health outcomes. Footnote 24
To address these issues, Kiefer et al. (2005) proposed a Centres of Excellence model “with specific research units/nodes focussing on particular topic areas…and with each health department in the country being linked formally with a research unit”.Footnote 25 Essentially, the proposed structure would function as a “network of networks”.
With the profile given to public health post-SARS, the opportunity arose to incorporate the early thinking into tangible action when the establishment of the NCCPH Program was announced. The NCCPH concept was based on the vision for a “national agency to be comprised of a series of regional hubs, which would be partnered with local academic centres, local/provincial public health systems (including the existing and proposed provincial public health agencies), and other stakeholders”, as proposed in the Naylor Report.Footnote 26
Two collaborating centre approaches were reported to have been considered in the organizational design for the NCCPH: the national collaborating centres established by the National Institute for Health and Clinical Excellence (NICE) in the UK, and PHAC’s Centres of Excellence for Children’s Wellbeing. Although the NCCPH does not mirror either of these models directly, these models were reviewed with a view to informing the integration requirements and organizational capacity for the NCCs.
Little documented information was available to the evaluators on the decision making process related to topic selection and geographical siting of the NCCs. There is indication that these decisions were made by the then Minister of State for Public Health, in consultation and negotiation with provincial Health Ministers.Footnote 27 Key informants suggested that the designers considered expertise available in the particular locations, regional representation and mix of urban and rural settings as factors, although most acknowledged the political nature of the decision making process.
Based on limited information, it appears that the NCCEH was assigned to the British Columbia Centre for Disease Control (BCCDC) as it had a pre-existing emphasis on environmental health. The placement of the NCCID within the newly established International Centre for Infectious Diseases (ICID) in Winnipeg was likely considered to be a complementary fit. Negotiations were held with Quebec before the topic of healthy public policy was agreed upon. There was some latitude in particular provinces and regions regarding NCC siting. In particular, Ontario and the Atlantic provinces are mentioned with respect to the time it took for the provincial governments to determine the site. In the end, PHAC determined the best siting for Methods and Tools was McMaster University. The Atlantic provinces chose not to place NCCDH in Halifax but to site this centre in the more rural location of Antigonish, Nova Scotia. The decision to include a centre for Aboriginal Health was reported to be an afterthought, and there was no mention made of the rationale for siting this centre in Prince George.
The NCCs are located in two types of host organizations. Three centres are located in universities, specifically McMaster University (NCCMT), St. Francis Xavier (NCCDH), and University of Northern British Columbia (NCCAH); and three are located within public health centres or institutes that have strong linkages with one or more academic institutions – BCCDC (NCCEH in Vancouver), ICID (NCCID in Winnipeg) and Institut national de santé publique du Québec (NCCHPP in Montréal and Québec City).
The Government of Canada’s Contribution Agreement mechanism was determined to be the best available funding alternative to support the implementation of the NCCPH Program. The Contribution Agreement mechanism involves an arm’s length relationship whereby the organization does not act as an agent of the Crown nor can the Crown directly provide services or direct the activities of the organization. Based on this mechanism, six independent Contribution Agreements were negotiated and established with the selected NCCs’ host organizations.
The design features of the NCCPH Program were further articulated with the development of the evaluation framework and logic model, Risk-based Management and Accountability Framework (RMAF) and Risk Assessment for the NCCs, and the release of the Request for Proposal (RFP) for the establishment of the Centres. Of note is that the NCCPH intermediate outcomes, as stated in the logic model, are linked to the Population and Public Health logic model. The goal of the Promotion of Population Health Contributions is to increase the capacity of individuals and communities to maintain and improve their health by:
- Building community capacity;
- Stimulating knowledge development and dissemination; and
- Partnership building/intersectoral collaboration.Footnote 28
According to the Progress Evaluation Framework (February 2008) Footnote 29, the primary beneficiaries of the NCCPH Program are public health practitioners, policy makers and non-government organizations who are expected to gain from improved access, availability and quality of knowledge translation products and from the improved processes for collaboration and cooperation. Others who are expected to benefit are:
- PHAC which is expected to benefit from collaborative processes and improved readiness capacity of public health practitioners across Canada in delivering on its mandate;
- Regional and municipal local health authorities who are also expected to benefit from enhanced knowledge and opportunities with respect to public health;
- Research, science and academic communities through improved connectivity with front line public health practitioners and their issues;
- Federal, provincial and territorial governments; and
- Canadians who are expected to benefit from the improved collaboration among different levels of government and increased use of evidence/knowledge applied to public health policy and practices.Footnote 30
Guiding principles, as reported by PHACFootnote 31 are:
- Relevance – reflecting the needs of practitioners, and be aligned with Canada’s overall public health strategy, and the work of PHAC;
- Interconnected – strengthening linkages between public health researchers and practitioners at regional, provincial, national and international levels;
- Coordinated – setting priorities and operate as members of a network, and focus on national public health priorities;
- Complementary – to the work of PHAC and other federal partners, provincial governments, academic institutions, and other key stakeholders;
- Interdisciplinary – support and engagement of all major public health disciplines operating at the municipal, provincial, regional and national levels across Canada; and
- Participatory – adopting a participatory approach and involving all relevant stakeholders in planning and implementing initiatives.
Description of NCCPH Implementation
The information provided in this section on NCCPH implementation was taken from the Summary Report 2005-2006.Footnote 32
The PHAC Strategic Policy Directorate was responsible for developing the NCCPH implementation plan in consultation with PHAC senior management, Health Canada, and the lead jurisdictions. This included developing a clear understanding of Contribution Agreements and defining the arm’s length relationship of the NCCs to PHAC, especially in relation to reporting and information sharing.
Using a project management approach, an internal project team was developed and included: a project lead; a representative from each NCC content area; regions, policy, grant and contribution funding representatives; legal from Health Canada; and administrative support. An Advisory Team representative of the PHAC Branches (i.e., consisting of the Chief Medical Health Officer of Canada, Deputy Chief Public Health Officer Branch Director Generals, and an external public health expert) was developed to support the project team. The Advisory Team approved the project team’s project plan.
The National Collaborating Centre Working Group consisting of a project lead from each participating region was established to further develop the implementation scope and concept with the project team. In January 2005, the project team and working group met to discuss the following topics:
- Concept of the NCCPH, based on other collaborating centres;
- Focus on KT;
- Developing a networking capacity; and
- Implication of the Government of Canada and Contribution Agreement requirements.
There was reportedly a freeze on release of the dollars which resulted in a time delay for receipt of dollars for the NCCPH Program. Interim funding in the amount of $450,000 per NCC was obtained through the Population Health Fund for the year 2005-2006 to enable NCC to initiate planning activities. Five out of six NCCs were able to take advantage of the interim dollars to hire staff or consultants to begin development. NCCMT did not request these funds as it was restricted by Government of Ontario policies of financial management.
Between April and June 2005, PHAC held meetings with the NCCs and Advisory Council (AC) with the goals of developing a common understanding of KSTE and assisting the NCC leads in developing work plans and activities relevant to knowledge transfer and public health.
In the spring of 2005, PHAC developed through internal and external consultations critical success factors and a logic model as part the Results-Based Management and Accountability Framework (RMAF) and in August 2005, $200,000 was repaid to the Population Health Fund, once funds were released.
A directed Request for Proposal (RFP) was provided to six pre-selected NCCs in September 2005. As a stipulation of the RFP, NCCs were required to outline an approach and methodology to KSTE using the definition outlined by the Canadian Institutes of Health Research (CIHR). The NCCPH Program Secretariat and Advisory Council reviewed and evaluated the NCC proposals using established criteria: relevance of subject matter; and approach and methodology to KSTE. As a result of the review process, four proposals were approved while two required revision.
PHAC entered into Contributions Agreement negotiations with each NCC with advice from Grants and Contributions Division and Health Canada Legal division. Substantial delays were experienced with two NCCs aligned with provincial governments. As of December 2006, all six Contribution Agreements were signed.
Several implementation challenges were outlined in the 2005-2006 Summary Report, including:
- Developing a national perspective;
- Understanding what KSTE means in public health;
- Limited human resource capacity in knowledge translation (KT);
- Lack of leadership at all levels;
- Provincial governments as host organizations for two NCCs;
- Appropriateness of funding mechanism; and
- Ability of dispersed NCCs to act as a collective.
The implementation of the NCCPH also built on lessons learned regarding collaborative centre work from other centres and organizations (e.g., Centres for Disease Control and World Health Organization).
Informant Feedback on Design and Early Implementation
Informants at the national level who were involved in the design and/or early implementation phase perceived that the NCCPH Program unfolded very closely to what was envisioned at the outset. They observed that some people at the senior Federal/Provincial/Territorial (F/P/T) and practitioner level may have expected it to roll out faster than had the original planners. It was noted that the original planners had not expected results to be evident for at least 10 to 15 years. There was a perception that there is a groundswell of support building for the NCCPH among Chief Medical Officers of Health, the Public Health Network, and key champions in the field.
When asked about the successful aspects of the design and implementation process, the most prevalent theme revolved around leadership. This included the diversity, interest and engagement of senior players at the F/P/T and non-government organization levels, PHAC and Advisory Council members towards the task of putting structure to a unique concept.
Below are sample comments on the theme of leadership in response to the question, what worked well?
It was inclusive – there were conversations among key players. There were high level discussions with senior players at the federal, provincial and NGO levels; people putting their heads around the problem. They were pretty engaged.
The Advisory Council had the role of elaborating one program with six centres – not directing them too much, but providing a big picture view.
The commitment of a competent, involved Advisory Council that took to heart the success of the program.
The interest and enthusiasm of the Council was positive. They have tremendous stature in the world, and their input was extremely positive and useful.
The Chair of the Advisory Council is excellent. He has gone above and beyond what he was mandated to do.
The design committee was an interesting experience. It was made up of people from various segments of the Agency [PHAC] regional areas, strategic policy and some representing the science and content side. It is one of the few times in the Agency where you get people from a variety of different disciplines together and they are able to work together and contribute to the process and content.
Those responsible for the program at PHAC believed in the program, were committed to it and were helpful vis-à-vis the Advisory Council and the NCCs.
Other positive comments about the design and implementation of the NCCPH Program related to the concept of centres distributed in regions across Canada but having a pan-Canadian mandate, the arm’s length relationship between PHAC and the NCCs, diversity of host organizations, and clarity about the role and functions of the NCCs. The fact that some NCCs chose to further define (and limit) their mandate was viewed positively.
When asked about the design barriers and challenges, a number of themes emerged listed in general descending order of how frequently the theme was mentioned:
The NCC concept was breaking new ground
A number of informants described the challenges with designing and implementing a program that is charting new territory. These included issues associated with developing a common understanding of what knowledge translation is about, finding and keeping people with KSTE expertise, differences in styles and approaches, and lack of clarity regarding the KSTE mandate. Several mentioned expectations for NCCs to roll out more quickly, but suggested that those holding such expectations may not have understood the challenges with implementing such a new concept.
We engage five key players as per the WHO pentagram framework – academia, community, institutions, government, etc. This program is not putting government in the middle, and that’s new.
There were specific challenges getting rolling because this is new and has never been done before. It’s the nature of the business. Three centres had problems [relating to finding and/or keeping a Scientific Director, finding a suitable host and addressing host expectations that were inconsistent with the KT mandate].
Before, when the federal government does [sic] something, it used one of three options: a demonstration project, a grant with a finite end or capacity building. Now they’re looking at the landscape, identifying partners and gaps, and looking at how to help front line folks. It was hard for people to engage in this different thinking. Some thought we were a granting agency.
There is no “road map” for the development of any of the NCCs, including this one.
The way the program was created was not clear and involved political issues. The mandate of each centre was not clear. The Centres had to accept this but maybe there was a more logical way of creating them.
There were differences in approaches and styles and disagreement about what KT is. People had to find their roles and niches, but we are starting to sort ourselves out.
[There were] expectations of the normal way of doing business in academia. Money goes to the provinces and people wanted access to this money, so we had trouble getting the money to the NCCs. There were ridiculous expectations, especially in places where there was a ‘everybody must get a prize’ mentality.
KT wasn’t out there. We had to build a bridge between academia and practice. KT doesn’t fit with academia or an academic career. Academics found this was not a winner for them. This gave rise to lots of our problems, including finding and keeping Scientific Directors.
When you design a program, usually there is already some infrastructure and people in place, but in this case there was nobody, especially in relation to KT. There wasn’t anybody who does this work. It [finding KT expertise] is bad enough in clinical work but it was even worse for us. This is slowly changing. Now there is a lot more happening as a result of the NCCs.
There is a shortage of people with appropriate backgrounds so they have to learn on the job. People graduating at the Masters level in public health didn’t get any courses on summarizing and critically appraising the literature. People don’t have that skill in the job market. We have had to explain to our colleagues in academia that they are not to do primary research; this isn’t curiosity driven research. [So, we] don’t fit well with academia and with practice people who are unevenly trained.
Government of Canada/PHAC accountability structures and processes
Among those who could speak to the early design and implementation phase of NCCPH development, several commented on the challenges associated with attempting to fit the NCCPH model into well established government accountability requirements and PHAC business processes. There was a general perception that this program was different, and that government rules presented as barriers to what people would have considered the normal way of doing business, and that PHAC was on a learning curve with respect to knowledge translation.
The accountability structure is difficult. Grants and Contributions formalized the Agreements but they didn’t help in terms of addressing administrative obstacles. Administrative tools were not adapted. For example, NCCs got the same budget each year. There are very heavy mechanisms in place.
The format of the Contribution Agreement has been difficult to work with. First, it is a little bit too directive for academics. There is no sense of trust. It is designed for community groups. It is too onerous for skilled academics. That needs rethinking. It shouldn’t be so micro-managed if the people are already established to do this type of work.
There are no honoraria for Council. PHAC lawyers said they couldn’t pay anyone for this role. They [PHAC] shoehorned what was possible to fit their rules.
There is a need for expertise in understanding the relationship as arm’s length. People find it hard to understand a Contribution Agreement with an arm’s length design. It took time to explain and build relationships. There is a need for guidance and support versus being directive. It was hard getting senior staff [within PHAC] to understand the need to develop [these] relationships and develop KT expertise.
There is a lack of clear understanding of KT within PHAC. PHAC is making in-roads in knowledge translation but in silos. They are all working in the absence of a larger effort. The Agency is still on a learning curve [but] it has to be a presence and needs to be a player.
Insufficient centralized infrastructure
Informants noted issues related to having sufficient infrastructure available for necessary technical and administrative support for this program. They expressed concerns related to PHAC Secretariat turnover and the limitations to their role, given the arm’s length relationship; NCC time devoted to coordination detracting from NCC core functions; and limited time availability of Advisory Council members to provide technical support.
Interaction between the centres has been a challenge. There can be duplication in methods and tools and so collaboration needs to be encouraged. But centres can spend all their time collaborating versus fulfilling their mandate.
Each of the six NCCs contribute money toward central, common services, but the proper structure and the finances are not there. Other national programs would have finances dedicated to a centralized structure all its own. The current system is not efficient or economical. For example, all NCCs are busy doing coordination strategies that need to be coordinated. Another example is the two months of staff time that we contributed toward the Summer Institute.
There was a challenge in trying to figure out where the NCCs should start collaborating. It took some time to identify common priorities, e.g., communications, common services, evaluation. There are not enough people at the Secretariat to support this. More resources are required for NCCs to collaborate with each other.
Staffing and turnover within PHAC are issues. There is too much work and not enough people. [They] need more staff. So much time is spent on process and bureaucracy that there is not time to develop their own KT expertise. NCCs need support in capacity building in KT but we can’t build our own capacity to pass it on.
Council tries to fulfil this [necessary technical support] function but everyone is busy with a full time job. They have been generous to provide time outside of Council time.
Differences among NCCs and host organizations were noted to present as challenges, and resulted in variations in approaches used and ability to progress with the mandate.
The siting of each of the six centres was heterogeneous. Some set up in universities. Some thought it [the siting approach used] was egalitarian but the setting didn’t have any track record. Some have undergone cataclysmic change, with the work of the NCCPH Secretariat and the Advisory Council.
The reality was the heterogeneity in this structure. Some topics required a lot of community involvement; other topics have almost no involvement and I don’t know that there should be. It means they don’t carry a heavy burden of this community involvement.
There were different starting points in terms of field, mandate and capacity of host organizations which have had an impact on progress or lack thereof of the NCCs.
Pan-Canadian initiatives present challenges. There are different reforms and realities across the country.
When asked what they thought should have been done differently in the design or implementation phase, four distinct themes arose:
Competitive process for the selection of the host organizations:
There should have been a call put out that invited applications from host institutions that would commit to provide the people to fulfill the mandate yet still be linked to researchers.
There should have been consultations and a competitive process to determine topic areas and locations to drive the program from the start.
[There should have been] a peer reviewed competition. We could have said to the selection committee that they must pick places in each of the six regions of Canada. Without this, some organizations got the wrong idea of what this was about and, even worse, some saw it as a pocket of money.
Sufficient mechanism for centralized coordination and technical support:
PHAC should have funded a full secretariat to provide technical support to the NCCs. The staff have a great skillset and have given it their all [but] they could not also provide the technical support. PHAC should have taken a more hands on approach and that the Secretariat should have taken more responsibility for the common website portal. I’m not confident that PHAC can provide support without being more directive than they need be. They need a technical secretariat that is supportive and helpful. It may have to be arm’s length.
All NCCs should be interlinked and interacting on the website.
The structure and separate resources (i.e., not derived from the budgets of the six NCCs), to ensure support for centralized coordination and support functions needs to be there for the Secretariat.
More resources were required for NCCs to collaborate with each other.
Greater clarity in several areas:
Better communication around why centres were allocated as they were. I know it was political.
Be more explicit in terms of roles and responsibilities. Establish preconditions. Clarify relationships.
Criteria needed to be better defined. Should we have pulled the plug sooner on NCCXX?
We should have been clearer to NCCs that their work was to support public health practitioners.
More flexibility in funding amounts and considerations:
There should have been a ramp up where funding in year one isn’t the same as in subsequent years.
We would like to advocate for a different funding formula and a different set of considerations, including the function of consultation, the wide variety of mandates and the different stages that each NCC is at.
The model needs to understand the resources that it takes to bring so many players from different perspectives into the fold. For example, one of our Committee Co-Chairs is from Labrador. To have a face-to-face meeting together costs $5,000.00.
Having the money follow the announcement in order to match public expectations for our Centre.
Governance and Accountability
The governance structure and accountabilities are documented in the February 2008 Progress Evaluation Framework and the 2005 Results-Based Management and Accountability Framework and Risk Assessment for the National Collaborating Centres (RMAF). Highlights are excerpted below. The governance structure for the NCCPH is represented in Figure 2 in the main body of this document.
The accountabilities for each of the key players, taken from the 2005 RMAF documentFootnote 33 were specified as follows:
- Establish the Program goals and objectives;
- Be responsible for audit and evaluation strategies and other mechanisms to ensure Program accountability;
- Provide a single point of contact for the management of the Contribution Agreements;
- Develop mechanisms to ensure there is no duplication of the work of the NCCs; and
- Play complementary knowledge translation function to ensure the mechanisms are in place within the Agency to promote the sharing and uptake of information for its Program and policy development.
NCCPH Program Secretariat – was to be responsible for overall Program accountability, specifically to:
- Solicit and review proposals;
- Negotiate and manage Contribution Agreements;
- Monitor the Program to ensure compliance with the Program’s terms and conditions;
- Establish Program-related policies and guidelines;
- Undertake performance measurement activities as set out in this RMAF;
- Act as the central point for communication for the Program – preparing briefing materials for the Minister and for the Chief Public Health Officer;
- Develop Program materials for NCCs, which may include materials for public consumption such as newsletters, web site content, media kits, etc.;
- Have a shared role in showcasing and promoting the NCC Program, both within and outside the federal government;
- Provide secretariat support to the Advisory Council (e.g. organize meetings, develop meeting materials, etc.);
- Promote collaboration among NCCs by facilitating the development of its network;
- Promote the development of linkages within PHAC and between NCCs and the Agency;
- Connect the work of the NCCs to other public health initiatives such as the Public Health Network and the Pan Canadian Public Health Strategy;
- Identify recipients for audit using a risk-based approach;
- Ensure evaluation is conducted; and
- Provide required information to report on results and feed into the PPH RMAF.
Advisory Council (AC) – to provide ongoing advice and guidance to PHAC on certain issues pertaining to the NCCs, specifically to advise on:
- Priorities for the National Collaborating Program and the role of each Centre in addressing them;
- Approaches to promote the co-ordination of the NCCs’ work plans and their alignment with national public health goals and strategies;
- Program evaluation strategies and reports;
- The underlying science of NCC proposals and work plans; and
- The relevance of their proposed work to national priorities.
NCCs, which consist of a host organization that will provide secretariat support and an administrative office to:
- Establish a multi-discipline consortium of partners that will include regional, national and international subject matter experts, policy makers and other groups;
- In collaboration with the consortium, develop, monitor and evaluate the progress of the NCC’s work plan;
- Prepare a proposal for Ministerial approval that includes a work plan, evaluation requirements, and clarity of roles and responsibilities;
- Identify a single point of contact within the NCC who will manage the Contribution Agreement with PHAC;
- Play a role in supporting and influencing the work of the Public Health Network through networking and providing supportive knowledge products;
- Showcase and promote their Centres and any products developed;
- Collaborate and consult with other FPT and PT structures; and
- Be accountable to PHAC through the Contribution Agreement(s).
Accountability requirements for the NCCs were articulated in the RFP (August 2005) upon which their proposals were based and the subsequent Contribution Agreements signed. The expectations can be categorized into two groups: (a) reporting requirements, (b) evaluation requirements. These are described below.
The reporting requirements for the NCCs include the submission of annual work plans each January and semi-annual progress/narrative reports each May and October. The NCCs are also expected to provide updates on projected spending each January.
In the NCC RFPs, it is stated that an evaluation report is to be completed by each NCC by January 2009, and in the Contribution Agreement, it is stated that the recipient shall:
- Carry out an evaluation of the Project funded through this Contribution Agreement in accordance with the National Collaborating Centre Program’s Evaluation Guide;
- Submit the results of the evaluation to the Minister; and
- Agree to participate in any evaluation on a regional, provincial, territorial and/or national scale.
Appendix C of the Contribution Agreement states that NCCs are to submit an annual evaluation and audit each May.
The typical required reporting pattern for the NCCs is presented in Table 2.
Update on projected spending
Narrative work plan on activities/deliverables, including Cashflow forecast
Progress/narrative work plan on activities/deliverables completed, including Year end Record of activities and expenditures (actual spending)
Annual Evaluation & Audit report (for the fiscal year ending March)
Progress/narrative report on activities/deliverables, including Cash flow forecast (projected spending) and record of expenditures (actual spending)
Feedback on Governance and Accountability
Informant feedback related to the role of the NCCPH Program Secretariat, Advisory Council, NCC Advisory Boards, and host organizations is presented below.
NCCPH Program Secretariat
Informant groups were asked what is working well regarding the NCCPH Program Secretariat. Themes that emerged include: the Program Secretariat provides timely feedback, advice, and follow up or support; the Secretariat is approachable and available; and the Secretariat contributes knowledge about the program so the NCCs can collaborate as partners (e.g., horizontal collaboration).
Informant groups were also asked to identify any challenges or barriers in relation to the NCCPH Program Secretariat. Several informant groups questioned the Secretariat’s expertise or quality of advice regarding knowledge translation or implementation of common projects as this is the domain of the Scientific Directors and the Advisory Boards. They offered the following suggestions for enhancement at the Program Secretariat level:
- Continuation of face-to-face feedback from the NCCPH Program Secretariat at the Summer Institute; and
- Clarification regarding the role of the NCCPH Program Secretariat (i.e., administrative or directive).
NCC representatives were generally positive about the Advisory Council’s (AC) ability to provide feedback and support; overall guidance/national vision; and content expertise. One informant group noted that Council members represent a good intermediary between PHAC and international work.
In relation to challenges or barriers at the Advisory Council level two minor themes were noted related to representation of people with limited field experience, especially during early implementation, and the lack of direct connection/dialogue between the AC and the NCCs.
Respondents in two informant groups wondered about the ongoing role of the AC now that the NCCPH Program is established. Sample suggestions included potential roles in broad, collaborative, integrated planning (e.g., collective view of KT); serving as a sounding board regarding perception of NCCs across Canada; gap identification; determination of topic areas, and consideration of new NCCs.
Representatives from two informant groups suggested a more direct connection between the AC and the NCCs (e.g., face to face meetings).
Several Advisory Board (AB) members and two NCC informant groups indicated there is a good mix of people on the boards (e.g., representatives from different regions, public health communities). A few Advisory Board members and two NCC groups said the ABs provide feedback on work plans and progress. Three informant groups said the ABs are engaged and involved. Several NCC informant groups indicated that having cross membership between the Advisory Council and Advisory Boards is useful.
Several themes emerged in relation to challenges or barriers at the AB level:
- Challenges associated with AB members’ attendance at meetings or in reaching them because of their busy schedules;
- AB representation, e.g., some suggested ABs or some members of ABs do not have sufficient public health and/or KT expertise, or do not have an understanding of the national picture; and
- Some ABs have had difficulties or have not functioned well.
The most prevalent suggestion was the need for a spectrum of representation or primary representation from the field.
Other suggestions offered for consideration by individual informants included:
- Establish a process for selecting AB members;
- Develop clearer guidelines on the composition and responsibility of ABs;
- Have a meeting of ABs across the NCCs;
- AB role should be more proactive (e.g., provide strategic direction); and
- The AB role should be to focus on individual NCCs.
NCC informant groups, including host organization representatives, were asked to describe the benefits of having the NCCs affiliated with the host institutions. The majority of NCC informant groups listed knowledge and capacity building in both directions, including the sharing of expertise. Most of the NCC informant groups identified increased opportunities for networking for the NCCs and the host organizations. Half of the NCC groups said affiliation of an NCC with the host organization enhanced the credibility or reputation of both groups. A few NCC informant groups identified recruitment of quality staff as another benefit. Finally, some groups stated that affiliation of the NCC with the host increased the visibility or profile of the host.
In relation to challenges or barriers, the majority of NCC informant groups identified issues around accountability, including dual reporting to PHAC and the host, as well as different reporting systems and timelines. Representatives from most of the NCCs informant groups also said the relationship between the host and PHAC represents a challenge. Half of the groups cited challenges around human resources, such as having to follow the host organization’s rules for hiring.
NCC informant groups offered the following suggestions in relation to the current arrangement with host institutions:
- Promote greater involvement of the host institution with the NCCs (e.g., collaborate on projects); and
- Clarify accountability roles and responsibilities (e.g., clarify who the NCC reports to).
Current Design and Delivery – Strengths, Challenges and Suggested Areas for Improvement
What is Working Well?
Informant groups identified several strengths related to the current design and delivery of the NCCPH Program. Three major themes and a number of minor themes were identified.
Collaboration among NCCs
The majority of groups highlighted collaboration among the NCCs. Examples provided by informants of collaborative activities include networking, information and resource sharing, identifying and collaborating on joint projects, mentorship provided by more established NCCs to newer NCCs, cross-assignment of NCC representatives on Advisory Boards, and the ability to leverage from each other (e.g., leveraging existing regional and provincial networks, working together to present to CPHA, Summer Institute).
The Advisory Council was also identified as a positive feature in the design and delivery of the Program. In particular, informant groups described the following positive features:
- Composition and characteristics – Council members were described as knowledgeable, experienced, and credible, with high profiles at the national and/or international level; and considered to represent a good mix of members from diverse geographical regions.
- Role – informant groups highlighted the ability of Advisory Council members to provide technical/peer review feedback on NCC work plans, being aligned with public health trends, acting as a vehicle for achieving commonality, and functioning as a ‘common touch point’ for the NCCs;
- Approach and style – respondents appreciated Council members’ flexibility and collegial style of interaction.
NCCPH Program Secretariat
Informant groups acknowledged the role of the NCCPH Program Secretariat as a benefit. Participants said the NCCPH Program Secretariat has demonstrated leadership and commitment, offer support, and are trustworthy, open, and transparent.
The NCCPH mandate was described as being necessary and appropriate as well as clear, consistent, and complementary across the NCCs. A few groups commented on the emerging capacity of staff and infrastructure over time. The Summer Institute was highlighted as a particular success story of the NCCPH. Finally, some NCC representative groups noted the relationship with their host organization, specifically mentioning the ability to leverage their hosts’ existing infrastructure.
Current Challenges and Barriers
Feedback from informant groups suggested several challenges or barriers related to the design and delivery of the NCCPH Program:
NCCPH model and mandate
Feedback from informants suggested issues associated with the definition or interpretation of the KSTE mandate as well as the organizing framework for the six NCCs. These issues were expressed in questions about the KT form, perceived overlap and duplication across NCCs, and the suggested need to define specific niches.
Some informants questioned whether the NCC role should not be expanded to include knowledge generation (e.g., what does an NCC do if there is no research evidence on a particular topic?), direct work in building KT capacity in the filed (e.g., advocating for or involvement in KT in Masters of Public Health programs), and advocacy for specific evidence-based policy.
NCC diversity / lack of common strategies and tools
Informants identified issues associated with the heterogeneity of the NCCs. In particular, informants discussed the different stages of individual NCC development and its impact on the ability to demonstrate the success of the overall Program. Other issues related to NCC diversity include a lack of common strategy and KSTE tools (e.g., single website), and a tendency of NCCs to operate in silos rather than as a collective. Each NCC is noted to be evolving on its own.
A few informant groups identified NCC collaboration and communication as an issue. Participants said it takes a long time to achieve consensus among the NCCs and that is has been a challenge achieving cohesion and a common vision across the NCCs.
Arm’s length relationship
Comments suggested some confusion regarding the relationships among the various players – NCCPH Program Secretariat, Advisory Council, Advisory Boards and host organizations. This included questions about who establishes NCC priorities, what is the role of the Program Secretariat in directing versus facilitating NCC work, and NCCs’ ability to suggest or advocate for policies that are in direct conflict with Government of Canada’s stated policies (e.g., harm reduction). Not representing a theme but a potentially critical question was whether it was possible to have six legally independent centres operate as a single NCCPH Program:
It has been a struggle achieving a common program when you have six arm’s length independent agencies with different mandates, target audiences and topic areas. We have to get our heads around the fact that you do not have a normal PHAC program. You are funding at arm’s length six independent organizations, each subject to their own host organization’s structures and processes, [and you cannot] get them to act as a single unit. That does not negate collaborative effort but we are not a single Program.
Funding was cited as another challenge. In particular, informant groups were acutely aware of the cutback in annual funding. Groups also acknowledged the three year funding cycle represents a challenge to long term planning, security and staff recruitment.
Reporting and paperwork bureaucracy
Informants identified the amount of reporting as a challenge. Some NCC representatives questioned the necessity to report on some items (e.g., activity logs and networking section).
Staff recruitment and retention
Informants stated that it was difficult to find staff with the right skill set in KSTE and public health. Staff turnover at the NCCPH Program Secretariat level was also identified as an ongoing issue.
Host organization issues
Issues related to the host organization were also identified as a challenge by a few informant groups. The most common issues raised included dual reporting to PHAC and the host organization; the struggle to balance the mandates between the NCCPH Program and the host organization; differences among the NCCs related to the type of host; and a disconnection between PHAC and the host organization. For example, a few host organization representatives expressed dissatisfaction with their role in rubber stamping or being “treated like an empty vessel”.
Lack of NCCPH program awareness/visibility/profile
A minor theme was lack of awareness of the NCCPH Program and the NCCs both among the fields or content areas of the individual Centres and within PHAC, including the various branches and at a senior level.
Suggestions to Improve Current Operations
Informant groups suggested the following areas for improvement:
Strengthen national focus/NCC collaborative effort
Informant groups suggested the development of a formal strategy through planning and priority setting across the NCCs. They also recommended more centralized support and coordination. For example, a central database would be a useful tool allowing NCCs to compile and share information. Participants suggested more fluidity and an integrated effort to further promote NCC collaboration. Staff sharing was identified as a concrete means to achieve this goal. Informant groups also recommended a centralized body with a sufficient budget to coordinate and promote NCC collaboration. It was posited that this central body could be assigned to an existing NCC or could be established as an independent NCC.
Informant groups recommended considering a broader mandate of KSTE to include KSTE training, competency building in public health, professional supports, and knowledge generation. Another suggestion was to consider NCC consolidation.
Address funding issues
Informant groups suggested considering funding beyond a three-year cycle, examine different funding formulas, develop a contingency plan in the event of cutbacks in funding, and address sustainability.
Informant groups proposed clarification in several areas including terminology, expectations regarding NCC revenue generation, governance roles and responsibilities, and target audiences.
Reduce accountability burden/streamline reporting
Suggestions offered included:
- Make reporting more efficient/streamlined/less labour intensive by determining what is really needed;
- Develop a common system for reporting; with some suggesting an online tool;
- Reporting requirements need to be more results-based versus process monitoring;
- Provide clarification around rationale of reporting requirements including how the information will be used and how much detail is needed; and
- Reduce reporting to once annually.
Informant groups suggested increasing networks and lineages with organizations, particularly at the provincial and national level. Specific examples of organizations listed by participants include: Health Canada/PHAC, First Nations and Inuit Health (FNIH), the Public Health Network (PHN), and provincial public health organizations.
Increase alignment with the public health community
Representatives from the informant groups suggested an increased alliance with the public health community, including the PHN and front line practitioners.
Report a problem or mistake on this page
- Date modified: