ARCHIVED - Formative Evaluation of the National Collaborating Centres for Public Health Program (NCCPH)

Executive Summary

Background

Recent public health issues have highlighted the need to strengthen Canada’s public health expertise, capacity, infrastructure, and knowledge transfer abilities.  The National Collaborating Centres for Public Health (NCCPH) initiative was announced in 2004 with an objective to foster evidence based decision making within Canada’s public health system.  Between 2005 and 2006, six National Collaborating Centres (NCCs) were established across Canada with topic specific mandates: Aboriginal health, determinants of health, environmental health, infectious diseases, healthy public policy, and methods and tools.

Evaluation design and methods

The purpose of the evaluation was to assess achievement of results against stated objectives, goals and immediate outcomes.  Overarching evaluation questions were posed:

1. Program design and delivery:  To what extent is the design and delivery of the NCCPH Program appropriate?
2. Achievements / successes:  What is the status of the NCCPH Program’s progress toward achieving its immediate goals and objectives?
3. Resource adequacy:  To what extent is the right level of financial and human resources available to effectively deliver the NCCPH Program?
4. Relevance:  To what extent does the NCCPH Program continue to be consistent with federal government and PHAC priorities?

Data sources included a document review; key informant interviews and/or on site focus groups; and observation and validation of findings at a NCCPH strategic planning retreat.

Conclusions

Based on the evaluation findings, the fundamental design of the NCCPH model is appropriate and no major changes to the defining elements are suggested.  The knowledge synthesis, translation and exchange (KSTE) mandate fulfills a unique niche within Canada’s public health context.  The NCCPH Program continues to be relevant and aligned with Government of Canada, Public Health Agency of Canada, and public health priorities.  The emerging KSTE capacity among NCC staff, made possible through dedicated and nationally dispersed funding, is promising for future impact of the NCCPH Program.

Placing NCCs in host organizations that are well positioned to bridge the gap between research and policy/practice was an important design consideration, and the extent to which the NCCs are in an environment where KSTE is understood appears to have had an impact on successful implementation of the mandate.  There is mutual benefit of NCC affiliation with the present host organizations. 

Governance, accountability and performance measurement structures and processes have been implemented and proven effective in identifying and addressing risks.  The technical advice and guidance of the Advisory Council, and work of the NCCPH Program Secretariat, have been particularly critical to the success of the NCCPH Program.

Despite operating as independent entities, the NCCs appear committed to collaboration and report this as one of the Program’s greatest successes.  A collegial, open and respectful relationship is evident among NCCs and, generally, among NCCs, the NCCPH Program Secretariat and the Advisory Council.

Given the short, four year history of the NCCPH Program, a tremendous amount has been achieved in taking a relatively vague and untried concept through design, implementation and output phases.  The NCCs have demonstrated varying degrees of achievement in knowledge translation and gap identification, depending on their stage of implementation and trajectory on the KSTE learning curve.  Collectively, they have achieved an impressive connection with the broader public health community at provincial, national and, even, international levels.  Although it is too early for impact to be evident, the NCCPH Program is well positioned to support public health practitioners and policy makers in applying best available evidence. 

A number of challenges and vulnerabilities were identified.  These predominantly relate to the organizing framework of the NCC topic areas, mandate boundaries, centralized leadership and support, host organization concerns, reporting requirements, and long term sustainability.  The need to revisit and revise the logic model was also identified.

Findings suggest a need for additional funding to strengthen centralized leadership and support.  However, with the exception of the restoration of funding cutbacks and annual cost adjustments, findings do not suggest a need for an increase to the original total annual $9.15M NCC base funding envelope at this time.  Most informants perceived a need for the NCCs to stabilize and demonstrate achievement before additional dollars are allocated.  Opportunities for potential efficiencies to be gained were identified, most notably review and reconciliation of mandates and, possibly, budgets across NCCs.  Similarly, while additional topics were suggested for consideration as priority areas, the general feedback was that it is premature to add new NCCs. 

Recommendations

Recommendations represent those the evaluators believe are the most critical to position the NCCPH Program for continued success into the future.  It is recommended that:

1. PHAC ensures the NCCPH Program governance structures remain sound and effective. Specifically:
   • 1.1 The Advisory Council and NCC Advisory Board structure be maintained.
   • 1.2 The NCCPH Program Secretariat and Advisory Council review the role of the Advisory Council in view of the support needed by NCCs as they move beyond the implementation phase into stable organizations.
   • 1.3 PHAC Senior Management investigates and addresses underlying NCCPH Program Secretariat recruitment and retention issues.
2. PHAC, in collaboration with the Advisory Council and NCCs, review the existing NCCPH mandate, specifically to:
   • 2.1 Review and clarify the parameters for NCC involvement in knowledge generation, capacity building, advocacy and professional supports.
   • 2.2 Review the organizing framework for the NCC areas of focus with a view to enhancing the logic, reconciling redundancies, clarifying target audiences, and increasing the flexibility to address new and emerging priority areas.
   • 2.3 Revisit NCC funding allocations in view of the decisions arising from the reviews proposed in 2.1 and 2.2 above.
3. PHAC, in consultation with the Advisory Council and NCCs, establish an independent mechanism to strengthen leadership, support and coordination.
   • 3.1 The mechanism be designed to strengthen the following essential functions:
     • Strategic direction and leadership for the NCCPH Program, including enhanced presence and linkages with common partners, collaborators, and interested parties;
     • Technical KSTE support to NCCs, including further development of a common KSTE framework, enhancing collaborative effort towards KSTE strategies, processes and tools, reducing duplication of effort, and continued KSTE knowledge and capacity building for NCC staff; and

     • NCC coordination activities, including but not limited to the existing Leads Secretariat functions, hosting and maintaining the common web portal, organizing the Summer Institute and other common forums, and further developing centralized systems that support all NCCs.

       While the last of the three functions may be accommodated through assignment to one or more of the existing NCCs (as per the Leads Secretariat), the nature of the first two functions suggest a distinct overarching rather than peer-level structure and mandate. The potential for accomplishing this through an additional centre should be explored.

   • 3.2 PHAC provide sufficient additional resources for the implementation of Recommendation
     • 3.1.  Existing NCC and NCCPH Program Secretariat budgets would not cover the range of functions and activities envisioned. 
4. The NCCPH Program Secretariat meets with host organization representative(s) to determine and resolve any concerns identified.
5. The NCCPH Program Secretariat, in consultation with the Advisory Council and NCCs, determines a long term strategy for sustainability of the NCCs.  This strategy should include considering  the appropriate approach to and parameters associated with accessing funding from other sources.
6. PHAC review the NCC reporting requirements and processes with a view to reducing the paper work burden.
   • 6.1 The amount of reporting required of NCCs should be in proportion to assessed risk.
   • 6.2 For well established NCCs with low assessed risk, reporting should be reduced to once annually and be results rather than activity based.
7. PHAC clarify and communicate to PHAC programs and NCCs the expectations of and appropriate mechanisms for the NCCs to collaborate and link with, and provide KSTE products and services to PHAC programs and services. 
8. The NCCPH Program logic model be reviewed and revised prior to initiating a summative evaluation of the NCCPH Program.