ARCHIVED - Formative Evaluation of the National Collaborating Centres for Public Health Program (NCCPH)

 

Appendix A: Data Matrix

National Collaborating Centres for Public Health Program


Formative Evaluation Data Matrix
EVALUATION QUESTIONS INDICATORS/MEASURES METHODS DATA SOURCES
Issue #1 - Design and Delivery: To what extent is the design and delivery of the NCCPH Program appropriate? 
How are the NCCPH Program design and delivery mechanisms (i.e., governance structure, resources, processes and support mechanisms) contributing to the achievement of the program’s goals?

Design component: Program model

Description of the NCCPH Program model and any updates, including rationale for initiating and selection of the program model and program priorities

Design component: Governance and accountabilities

Description and perceptions of the governance structures

Description and perceptions of accountability and decision-making structures and processes

Design component: NCC specific

Description and perception of the program model design related to the NCCs (regional and topic specific differences)

Description of how NCC content areas were decided

Design component:  Resources

Identification of resources (cash and in-kind)

Document review

Interviews and/or focus groups



Literature search

- NCCPH documents

- NCCPH Program Secretariat/PHAC staff
- NCC leads and NCC staff
- Advisory Council members
- Advisory Board members
- Sponsor site representatives

- Identification of best practices and comparison of program model with other organizations with similar structure (geographically dispersed centres of excellence), mandate (KSTE), and/or content (public health).

What are the challenges, constraints and/or systemic barriers in the implementation of the NCCPH Program? 

Delivery component: Implementation

Barriers to implementing the program by type

# and type of operational constraints

Document review

Interviews and/or focus groups

- NCCPH and NCC documents

- NCCPH Program Secretariat/PHAC staff
- NCC leads and NCC staff
- Advisory Council members
- Advisory Board members
- Interested/involved parties
- Sponsor site representatives

Are different NCCs facing different challenges? Barriers specific to each NCC in implementing the program

Document review

Interviews and/or focus groups

- NCCPH and NCC documents

- NCCPH Program Secretariat/PHAC staff
- NCC leads and NCC staff
- Advisory Council members
- Advisory Board members

How are these challenges being addressed? Types of solutions to identified challenges undertaken at: (a) Program level; and (b) NCC level

Document review

Interviews and/or focus groups

- NCCPH and NCC documents

- NCCPH Program Secretariat/PHAC staff
- NCC leads and NCC staff
- Advisory Council members
- Advisory Board members

What is working well in the NCCPH program?

Delivery component: Satisfaction

Perceived satisfaction

Document review

Interviews and/or focus groups

- NCCPH and NCC documents

- NCCPH Program Secretariat/PHAC staff
- NCC leads and NCC staff
- Advisory Council members
- Advisory Board members
- Interested/involved parties
- Sponsor site representatives

Does the NCCPH Program have appropriate performance measurement and reporting strategies?

Accountability

Evidence of performance measurement and reporting strategies:  (a) NCC program secretariat/PHAC; (b) Advisory Council; (c) Advisory Board; (d) NCCs; and (e) NCC Collaborative

Document review

Interviews and/or focus groups

 - NCCPH and NCC documents

- NCCPH Program Secretariat/PHAC staff
- NCC leads and NCC staff
- Advisory Council members
- Advisory Board members

Are NCCs reporting on planned results? 

Reporting requirements on results for NCCs are identified

Evidence NCCs are reporting on planned results

Document Review - NCCPH and NCC documents
To what extent are NCCs meeting the requirements of the Contribution Agreements? Evidence the NCCs are meeting the requirements of the Contribution Agreements Document review - NCCPH and NCC documents
Are the NCCs collecting data on immediate outcomes? 

Evaluation

Evidence the NCCs are collecting data on immediate outcomes

Document Review - NCCPH and NCC documents
Are options/alternatives to improve program design/delivery being considered or recommended?

Continuous Improvement

Evidence of continuous improvement mechanisms at the program level

Recommendations to improve program design/delivery

Document review

Interviews and/or focus groups

- NCCPH and NCC documents

- NCCPH Program Secretariat/PHAC staff
- NCC leads and NCC staff
- Advisory Council members
- Advisory Board members

Issue #2 – Achievements / Successes:  What is the status of the NCCPH Program’s progress toward achieving its immediate goals and objectives?
In what manner and to what extent have the NCCPH Program’s planned activities been implemented?

Activities

Evidence planned activities have been implemented, e.g., Terms of Reference for NCCPH/PHAC; establish logic model; establish Advisory Council; provide secretarial support for Advisory Council; establish NCCs; conduct risk assessment of NCCs; completed funding approval forms; provide plans & priority-setting documents; # and type of meetings/year

Document review

Interviews and/or focus groups

- NCCPH documents

- NCCPH Program Secretariat/PHAC staff
- Advisory Council members

Are the planned activities producing the expected (a) management level outputs; and (b) program (NCC) level outputs?

Outputs

# and type of management level outputs, e.g.,

  • Signed Contribution Agreement
  • Advisory Council TOR/protocol
  • Support materials
  • Meetings with NCCs
  • Priority-setting documents
  • Promotional tools
  • Feedback reports
  • Financial review

# and type of NCC level outputs, e.g.,

  • Progress reports
  • Needs assessment
  • Strategic plans
  • Evaluation plans

Document review

Interviews and/or focus groups

- NCCPH documents


- NCCPH Program Secretariat/PHAC staff
- NCC leads and NCC staff
- Advisory Council members
- Advisory Board members

To what extent is the NCCPH Program on target toward achieving its immediate outcomes related to increased opportunity for collaboration?

Immediate outcome: Collaboration

Evidence of increased collaboration among NCCs, e.g., #, type, and variety of connections among NCCs; NCC Collaborative is established; # of NCC Collaborative meetings/year; Summer Institute Retreat is established

Evidence of increased collaboration with health portfolio, e.g., #, type, and variety of connections among health portfolio

NCCs have developed a plan/strategy for increasing collaboration with identified interested and involved parties. Specific activities are identified

NCCs have identified key interested and involved parties for collaboration

Document review


Interviews and/or focus groups

- NCCPH and NCC documents
-  NCC websites


- NCCPH leads and NCC staff
- Advisory Council members
- Advisory Board members
- Interested/involved parties

To what extent is the NCCPH Program on target toward achieving its immediate outcome related to knowledge translation?

Immediate outcome: Knowledge translation

# and types of knowledge products/resources (a) under development; (b) developed; and (c) disseminated/delivered

# and types of activities/mechanisms/pathways being used for knowledge translation

# and types of knowledge exchange activities

# of organizations participating in collaborative product development

Document review


Interviews and/or focus groups

- NCCPH and NCC documents
-  NCC websites


- NCC leads and NCC staff
- Advisory Board members
- Interested/involved parties

To what extent is the NCCPH Program on target toward achieving its immediate outcome related to identifying knowledge gaps?

Immediate outcome: Knowledge gap identification

# of scans used to identify knowledge gaps

Unexpected knowledge gaps are described

Document review


Interviews and/or focus groups

- NCCPH and NCC documents
- NCC websites


- NCC leads and NCC staff
- Advisory Board members
- Interested/involved parties

How are those knowledge gaps being addressed?

Are lessons learned being identified and applied to ongoing activities?

# and types of initiatives/plans to address knowledge gaps

Evidence lessons learned are applied to ongoing activities

Document review


Interviews and/or focus groups

- NCCPH and NCC documents
- NCC websites


- NCC leads and NCC staff
- Advisory Board members
- Interested/involved parties

To what extent have important partnerships and linkages been established?

Immediate outcome: Networking

# and type of partnerships of NCCs with external organizations

NCCs have developed a communication plan to promote awareness of the Centre among key partners, interested parties and potential beneficiaries. 

Key partners, interested parties and potential beneficiaries are aware of relevant NCC

Key partners, interested parties and potential beneficiaries indicate acceptability and credibility of NCC

Document review


Interviews and/or focus groups

- NCCPH and NCC documents
- NCC websites


- NCC leads and NCC staff
- Advisory Council members
- Advisory Board members
- Interested/involved parties

Are best practices being identified and shared among the NCCs?

# and types of mechanisms developed for sharing information among NCCs

# and types of collaboration/partnerships/ coordination efforts among NCCs

# and types of communications/networking activities, e.g., Summer Institute

Document review


Interviews and/or focus groups

- NCCPH and NCC documents
- NCC websites


- NCC leads and NCC staff
- Advisory Board members

Is the program on target toward achieving its (a) intermediate outcomes; and (b) longer-term outcomes?

Intermediate & long term outcomes

Evidence of increased availability of knowledge for evidence-based decision making

Examples of evidence-based decisions informing public health programs, policies, and practices

Evidence of mechanisms and processes to access knowledge

# and types of requests for information
# and duration of website hits

Document review

Interviews and/or focus groups

- NCCPH and NCC documents

- NCCPH Program Secretariat/PHAC staff
- NCC leads and NCC staff
- Advisory Council members
- Advisory Board members
- Interested/Involved parties
- Potential beneficiaries

Issue #3 – Perceived Cost-Effectiveness:  To what extent is the right level of resources available to effectively deliver the NCCPH Program?
Are the allocated resources (i.e., human, financial, etc.) sufficient and/or available to achieve the NCCPH Program’s intended immediate outcomes given the planned reach?

Cost-effectiveness: Adequacy of resources

Type and level of resources contributed

Description and perceived adequacy of type and level of infrastructure created (e.g., facilities, systems, etc.)

Perception of adequacy of resources

Document review

Interviews and/or focus groups

- NCCPH and NCC documents

- NCCPH Program Secretariat/PHAC staff
- NCC leads and NCC staff
- Advisory Council members
- Advisory Board members
- Sponsor site representatives

How could the efficiency and effectiveness of the NCCPH Program be improved?

Cost-effectiveness: Efficiency & effectiveness

Examples of successes / best practices and lessons learned regarding cost-effectiveness

Perception of possible efficiency/ effectiveness enhancements

Examples of cash and in-kind contributions leveraged by NCCs

Document review


Interviews and/or focus groups



Literature search

- NCCPH and NCC documents

- NCCPH Program Secretariat/PHAC staff
- NCC leads and NCC staff
- Advisory Council members
- Advisory Board members

- Comparison of program model with other organizations with similar structure (geographically dispersed centres of excellence), mandate (KSTE), and content (public health)

Issue #4 – Relevance:  To what extent does the NCCPH Program continue to be consistent with federal government and PHAC priorities?
To what extent does the NCCPH Program continue to be consistent with (a) Government of Canada; (b) PHAC priorities; and (c) the needs of Canadians?

Relevance

Evidence and/or perception of alignment of Program with GoC current priorities and mandate

Evidence and/or perception of alignment of program with PHAC ‘s strategic outcomes and goals

Evidence and/or perception of alignment of Program with PHAC’s Program activity architecture

Evidence and/or perception of alignment of Program with population and public health indicators

Document review


Interviews and/or focus groups

- NCCPH and NCC documents
- Government documents

- NCCPH Program Secretariat/PHAC staff
- NCC leads and NCC staff
- Advisory Council members
- PHAC representatives

Is the NCCPH Program still consistent with the program’s original logic model? Evidence of alignment of the current program with the original rationale for the creation of the NCCPH Program as provided through the formative evaluation

Document review

Interviews and/or focus groups

- Logic model

- NCCPH Program Secretariat/PHAC staff
- NCC leads and NCC staff
- Advisory Council members

Is the NCCPH Program meeting the broader public health goals? Examples and perception of alignment of Program with broader public health goals Interviews and/or focus groups - PHAC representatives
- NCCPH Program Secretariat/PHAC staff
- NCC leads and NCC staff
- Advisory Council members
Is this program model still relevant? Perception of relevance

Interviews and/or focus groups



Literature search

- PHAC representatives
- NCCPH Program Secretariat/PHAC staff
- NCC leads and NCC staff
- Advisory Council members
- Advisory Board members
- Interested/involved parties

- Comparison of program model with other organizations with similar structure (geographically dispersed centres of excellence), mandate (KSTE), and content (public health)

Are the public health priorities still the same?

Should there be other National Collaborating Centres (NCCs)? 

Perception that NCCPH Program is addressing current public health priorities

Perception regarding need for other NCCs

Document review

Interviews and/or focus groups

- NCCPH and NCC documents

- PHAC representatives
- NCCPH Program Secretariat/PHAC staff
- NCC leads and NCC staff
- Advisory Council members
- Advisory Board members

What changes, if any, would make the NCCPH Program more relevant? Evidence suggesting enhancements based on best practice

Interviews and/or focus groups



Literature search

- PHAC representatives
- NCCPH Program Secretariat/PHAC staff
- NCC leads and NCC staff
- Advisory Council members
- Informants for other jurisdictions

- Comparison of program model with other organizations with similar structure (geographically dispersed centres of excellence), mandate (KSTE), and content (public health)


Report a problem or mistake on this page
Please select all that apply:

Thank you for your help!

You will not receive a reply. For enquiries, contact us.

Date modified: