ARCHIVED - Formative Evaluation of the National Collaborating Centres for Public Health Program (NCCPH)
Appendix A: Data Matrix
National Collaborating Centres for Public Health Program
| EVALUATION QUESTIONS | INDICATORS/MEASURES | METHODS | DATA SOURCES |
|---|---|---|---|
| Issue #1 - Design and Delivery: To what extent is the design and delivery of the NCCPH Program appropriate? | |||
| How are the NCCPH Program design and delivery mechanisms (i.e., governance structure, resources, processes and support mechanisms) contributing to the achievement of the program’s goals? | Design component: Program model Description of the NCCPH Program model and any updates, including rationale for initiating and selection of the program model and program priorities Design component: Governance and accountabilities Description and perceptions of the governance structures Description and perceptions of accountability and decision-making structures and processes Design component: NCC specific Description and perception of the program model design related to the NCCs (regional and topic specific differences) Description of how NCC content areas were decided Design component: Resources Identification of resources (cash and in-kind) |
Document review Interviews and/or focus groups Literature search |
- NCCPH documents - NCCPH Program Secretariat/PHAC staff - Identification of best practices and comparison of program model with other organizations with similar structure (geographically dispersed centres of excellence), mandate (KSTE), and/or content (public health). |
| What are the challenges, constraints and/or systemic barriers in the implementation of the NCCPH Program? | Delivery component: Implementation Barriers to implementing the program by type # and type of operational constraints |
Document review Interviews and/or focus groups |
- NCCPH and NCC documents - NCCPH Program Secretariat/PHAC staff |
| Are different NCCs facing different challenges? | Barriers specific to each NCC in implementing the program | Document review Interviews and/or focus groups |
- NCCPH and NCC documents - NCCPH Program Secretariat/PHAC staff |
| How are these challenges being addressed? | Types of solutions to identified challenges undertaken at: (a) Program level; and (b) NCC level | Document review Interviews and/or focus groups |
- NCCPH and NCC documents - NCCPH Program Secretariat/PHAC staff |
| What is working well in the NCCPH program? | Delivery component: Satisfaction Perceived satisfaction |
Document review Interviews and/or focus groups |
- NCCPH and NCC documents - NCCPH Program Secretariat/PHAC staff |
| Does the NCCPH Program have appropriate performance measurement and reporting strategies? | Accountability Evidence of performance measurement and reporting strategies: (a) NCC program secretariat/PHAC; (b) Advisory Council; (c) Advisory Board; (d) NCCs; and (e) NCC Collaborative |
Document review Interviews and/or focus groups |
- NCCPH and NCC documents - NCCPH Program Secretariat/PHAC staff |
| Are NCCs reporting on planned results? | Reporting requirements on results for NCCs are identified Evidence NCCs are reporting on planned results |
Document Review | - NCCPH and NCC documents |
| To what extent are NCCs meeting the requirements of the Contribution Agreements? | Evidence the NCCs are meeting the requirements of the Contribution Agreements | Document review | - NCCPH and NCC documents |
| Are the NCCs collecting data on immediate outcomes? | Evaluation Evidence the NCCs are collecting data on immediate outcomes |
Document Review | - NCCPH and NCC documents |
| Are options/alternatives to improve program design/delivery being considered or recommended? | Continuous Improvement Evidence of continuous improvement mechanisms at the program level Recommendations to improve program design/delivery |
Document review Interviews and/or focus groups |
- NCCPH and NCC documents - NCCPH Program Secretariat/PHAC staff |
| Issue #2 – Achievements / Successes: What is the status of the NCCPH Program’s progress toward achieving its immediate goals and objectives? | |||
| In what manner and to what extent have the NCCPH Program’s planned activities been implemented? | Activities Evidence planned activities have been implemented, e.g., Terms of Reference for NCCPH/PHAC; establish logic model; establish Advisory Council; provide secretarial support for Advisory Council; establish NCCs; conduct risk assessment of NCCs; completed funding approval forms; provide plans & priority-setting documents; # and type of meetings/year |
Document review Interviews and/or focus groups |
- NCCPH documents - NCCPH Program Secretariat/PHAC staff |
| Are the planned activities producing the expected (a) management level outputs; and (b) program (NCC) level outputs? | Outputs # and type of management level outputs, e.g.,
# and type of NCC level outputs, e.g.,
|
Document review Interviews and/or focus groups |
- NCCPH documents - NCCPH Program Secretariat/PHAC staff |
| To what extent is the NCCPH Program on target toward achieving its immediate outcomes related to increased opportunity for collaboration? | Immediate outcome: Collaboration Evidence of increased collaboration among NCCs, e.g., #, type, and variety of connections among NCCs; NCC Collaborative is established; # of NCC Collaborative meetings/year; Summer Institute Retreat is established Evidence of increased collaboration with health portfolio, e.g., #, type, and variety of connections among health portfolio NCCs have developed a plan/strategy for increasing collaboration with identified interested and involved parties. Specific activities are identified NCCs have identified key interested and involved parties for collaboration |
Document review Interviews and/or focus groups |
- NCCPH and NCC documents - NCCPH leads and NCC staff |
| To what extent is the NCCPH Program on target toward achieving its immediate outcome related to knowledge translation? | Immediate outcome: Knowledge translation # and types of knowledge products/resources (a) under development; (b) developed; and (c) disseminated/delivered # and types of activities/mechanisms/pathways being used for knowledge translation # and types of knowledge exchange activities # of organizations participating in collaborative product development |
Document review Interviews and/or focus groups |
- NCCPH and NCC documents - NCC leads and NCC staff |
| To what extent is the NCCPH Program on target toward achieving its immediate outcome related to identifying knowledge gaps? | Immediate outcome: Knowledge gap identification # of scans used to identify knowledge gaps Unexpected knowledge gaps are described |
Document review Interviews and/or focus groups |
- NCCPH and NCC documents - NCC leads and NCC staff |
| How are those knowledge gaps being addressed? Are lessons learned being identified and applied to ongoing activities? |
# and types of initiatives/plans to address knowledge gaps Evidence lessons learned are applied to ongoing activities |
Document review Interviews and/or focus groups |
- NCCPH and NCC documents - NCC leads and NCC staff |
| To what extent have important partnerships and linkages been established? | Immediate outcome: Networking # and type of partnerships of NCCs with external organizations NCCs have developed a communication plan to promote awareness of the Centre among key partners, interested parties and potential beneficiaries. Key partners, interested parties and potential beneficiaries are aware of relevant NCC Key partners, interested parties and potential beneficiaries indicate acceptability and credibility of NCC |
Document review Interviews and/or focus groups |
- NCCPH and NCC documents - NCC leads and NCC staff |
| Are best practices being identified and shared among the NCCs? | # and types of mechanisms developed for sharing information among NCCs # and types of collaboration/partnerships/ coordination efforts among NCCs # and types of communications/networking activities, e.g., Summer Institute |
Document review Interviews and/or focus groups |
- NCCPH and NCC documents - NCC leads and NCC staff |
| Is the program on target toward achieving its (a) intermediate outcomes; and (b) longer-term outcomes? | Intermediate & long term outcomes Evidence of increased availability of knowledge for evidence-based decision making Examples of evidence-based decisions informing public health programs, policies, and practices Evidence of mechanisms and processes to access knowledge # and types of requests for information |
Document review Interviews and/or focus groups |
- NCCPH and NCC documents - NCCPH Program Secretariat/PHAC staff |
| Issue #3 – Perceived Cost-Effectiveness: To what extent is the right level of resources available to effectively deliver the NCCPH Program? | |||
| Are the allocated resources (i.e., human, financial, etc.) sufficient and/or available to achieve the NCCPH Program’s intended immediate outcomes given the planned reach? | Cost-effectiveness: Adequacy of resources Type and level of resources contributed Description and perceived adequacy of type and level of infrastructure created (e.g., facilities, systems, etc.) Perception of adequacy of resources |
Document review Interviews and/or focus groups |
- NCCPH and NCC documents - NCCPH Program Secretariat/PHAC staff |
| How could the efficiency and effectiveness of the NCCPH Program be improved? | Cost-effectiveness: Efficiency & effectiveness Examples of successes / best practices and lessons learned regarding cost-effectiveness Perception of possible efficiency/ effectiveness enhancements Examples of cash and in-kind contributions leveraged by NCCs |
Document review Interviews and/or focus groups Literature search |
- NCCPH and NCC documents - NCCPH Program Secretariat/PHAC staff - Comparison of program model with other organizations with similar structure (geographically dispersed centres of excellence), mandate (KSTE), and content (public health) |
| Issue #4 – Relevance: To what extent does the NCCPH Program continue to be consistent with federal government and PHAC priorities? | |||
| To what extent does the NCCPH Program continue to be consistent with (a) Government of Canada; (b) PHAC priorities; and (c) the needs of Canadians? | Relevance Evidence and/or perception of alignment of Program with GoC current priorities and mandate Evidence and/or perception of alignment of program with PHAC ‘s strategic outcomes and goals Evidence and/or perception of alignment of Program with PHAC’s Program activity architecture Evidence and/or perception of alignment of Program with population and public health indicators |
Document review Interviews and/or focus groups |
- NCCPH and NCC documents - NCCPH Program Secretariat/PHAC staff |
| Is the NCCPH Program still consistent with the program’s original logic model? | Evidence of alignment of the current program with the original rationale for the creation of the NCCPH Program as provided through the formative evaluation | Document review Interviews and/or focus groups |
- Logic model - NCCPH Program Secretariat/PHAC staff |
| Is the NCCPH Program meeting the broader public health goals? | Examples and perception of alignment of Program with broader public health goals | Interviews and/or focus groups | - PHAC representatives - NCCPH Program Secretariat/PHAC staff - NCC leads and NCC staff - Advisory Council members |
| Is this program model still relevant? | Perception of relevance | Interviews and/or focus groups Literature search |
- PHAC representatives - Comparison of program model with other organizations with similar structure (geographically dispersed centres of excellence), mandate (KSTE), and content (public health) |
| Are the public health priorities still the same? Should there be other National Collaborating Centres (NCCs)? |
Perception that NCCPH Program is addressing current public health priorities Perception regarding need for other NCCs |
Document review Interviews and/or focus groups |
- NCCPH and NCC documents - PHAC representatives |
| What changes, if any, would make the NCCPH Program more relevant? | Evidence suggesting enhancements based on best practice | Interviews and/or focus groups Literature search |
- PHAC representatives - Comparison of program model with other organizations with similar structure (geographically dispersed centres of excellence), mandate (KSTE), and content (public health) |
Page details
- Date modified: