ARCHIVED - Formative Evaluation of the National Collaborating Centres for Public Health Program (NCCPH)

Chapter 2: Evaluation Purpose and Methods

Evaluation Purpose and Scope

The stated purpose of this formative evaluation was “to assess achievement of results against stated objectives, goals and immediate outcomes, with a primary focus on the activities and outputs since the inception of the NCCPH Program”.^{Footnote 14} The evaluation was expected to identify gaps and challenges and ensure that NCCPH achievements and successes were well documented, substantiated and shared.  The findings were expected to guide recommendations for further development and implementation of future NCCPH activities, inform renewal of the program’s terms and conditions, and provide baseline data for a summative evaluation.

A Progress Evaluation Framework was developed in February 2008 by PHAC in consultation with the six NCCs.^{Footnote 15} Overarching evaluation questions were posed under four areas of inquiry:

1. Program design and delivery:  To what extent is the design and delivery of the NCCPH Program appropriate?
2. Achievements / successes:  What is the status of the NCCPH Program’s progress toward achieving its immediate goals and objectives?
3. Resource adequacy:  To what extent is the right level of resources available to effectively deliver the NCCPH Program?
4. Relevance:  To what extent does the NCCPH Program continue to be consistent with federal government and PHAC priorities?

Evaluation Design and Methods

A logic model for the NCCPH was established as part of the 2005 RMAF (Figure 1).^{Footnote 16}Detailed questions and associated indicators were initially specified in the RMAF, and further refined in the February 2008 Progress Evaluation Framework.^{Footnote 17}

The evaluation team conducting the formative evaluation adopted the February 2008 Evaluation Framework, with minor refinements to the data matrix (Appendix A).

Data collection methods and sources included:

• Document review;
• Interviews and/or focus groups, conducted on site, with NCCPH Program Secretariat, NCC Leads and staff, and host organization representatives;
• Telephone interviews with key informants, including NCC staff not available during on site visits; and
• Observation and validation of findings at a NCCPH strategic planning retreat.

Figure 1. NCCPH Logic Model^{Footnote 18}

Approximately 600 documents were reviewed for this evaluation, including documents accessed through the NCCPH Program Secretariat, NCCs, NCCPH websites and the literature.  They included:

• NCCPH Program documentation (approx. 85) including but not limited to: early background documents on the establishment of the NCCPH Program, Results-Based Management and Accountability Framework (RMAF) documents outlining accountability expectations, Annual Reports, and minutes of key meetings such as those of the Advisory Council;
• Documents specific to each NCC (approx. 425), including but not limited to: their Contribution Agreement, key documents related to NCCPH expectations (e.g., work plans, progress reports, needs assessment documents, strategic and evaluation plans), and others deemed important to addressing the evaluation questions;
• Documents related to the collaborative efforts of the NCCs (approx. 85), such as minutes of  NCC Leads meetings, communication plans and Summer Institute background materials; and
• Review of materials produced by NCCPH and/or NCCs that provide useful information to address the evaluation questions (e.g., Medical Officers of Health survey regarding awareness of NCCs); and relevant literature (approx. 20).

A document review template was developed for the NCCPH Program Secretariat and each of the NCCs as a mechanism to consistently compile key information needed for the evaluation (Appendix B).  Evaluators then populated the templates based on material compiled and forwarded by the NCCPH Program Secretariat.  The completed templates were presented to the Managing Director or NCC staff for validation during NCC site visits.  At the same time, any items the evaluators had noted as missing were requested from the NCCs or PHAC.  The validated document review tools were used to report on activities and achievements for each group.

A total of 63 key informant interviews and on site focus groups were conducted, representing 87 individuals (Appendix C).  Evaluators compiled a list of 200 potential interviewees based on available contact lists and individuals recommended by the NCCPH Program Secretariat and NCC staff.  Final selection was made by the evaluation team, with consideration to include:

• Those in key positions, including PHAC representatives, NCC senior managers and staff, representatives from host organizations, past and present Advisory Council members, Advisory Board Chairs and other members, and individuals involved in the early stages of NCCPH development;
• Those who had the greatest involvement with the NCCPH, such as those who have played multiple roles or have been exposed to the activities of more than one NCC;
• Representation across the six topic areas;
• Representation across the country;
• Representation across a variety of public health backgrounds – PHAC, provincial government, Medical Officers of Health, public health practitioners, and researchers; and
• Inclusion of interested parties and anticipated beneficiaries who represented potential users of available NCC products.

The number of informants interviewed by informant group is presented in Table 1.  Some individuals interviewed represent more than one informant group.  For example, an individual may be both staff of a NCC and the member of another NCC Advisory Board, thus representing two informant groups. 

Interview questions by key informant group were prepared to guide each interview and focus group (Appendix D).  More specific questions and probes were generated based on responses to these general questions.  Interview questions focused on the four evaluation questions: program design and delivery, achievements and successes, resource adequacy, and relevance.

Detailed notes were taken during interviews and focus group sessions.  Whenever possible, the sessions were recorded and the recordings used as back up to the written notes.  Where more than one evaluator was involved in an interview or focus group, the notes used for analysis were validated by each team member. 

The notes were compiled according to question, then coded and analyzed according to:

• 10 informant groups:
  • NCC representatives, including representatives of host organizations, constitute six informant groups,
  • Advisory Council members constitute one informant group,
  • NCC Advisory Board members constitute one informant group,
  • PHAC and other representatives at the national level constitute one informant group, and
  • Interested/involved parties constitute one informant group; and
• Theme and sub-theme.

Two evaluators attended and presented the preliminary findings of the formative evaluation at a NCCPH strategic planning retreat. The timing of this event afforded the evaluators an opportunity to validate the issues and suggestions heard during the informant interviews and focus groups, as well as to observe the interactions and decision making processes among key NCCPH representatives.  Retreat participants included Advisory Council members, PHAC representatives (including the NCCPH Program Secretariat), and NCC leads.

Limitations

The evaluators recognize several limitations to this evaluation, and caution readers to interpret the findings presented in this report accordingly.  Some of the limitations associated with specific methods were mitigated by the use of multiple data sources to validate findings.  Despite the limitations, the evaluators are confident that the report represents a fair picture of the activities of the NCCPH and the perceptions of key parties.

First, the timeframe and financial restraints of the evaluation did not permit in-depth review of the activities and products of each NCC.  It is noted that the evaluation spans the activities of eight entities – those of the NCCPH Program Secretariat, NCC Collaborative and six NCCs.  The amount of activity undertaken over a four year period could only be summarized and reported at a high level.

Second, while an attempt was made to access all relevant documents, it is possible that some were inadvertently omitted.  The evaluators relied on those received as of November 2008.

Third, due to the large number of potential informants that could have been involved, the evaluators were somewhat reliant on the recommendations of the NCCPH Program Secretariat, NCC Scientific Leads, and Managing Directors in identifying those individuals that had the greatest involvement with the NCCPH.  Thus, there is a possible bias towards favourable perceptions of the NCCPH.  The evaluators attempted to mitigate this by including representatives not suggested by the NCCPH Program Secretariat and the NCCs, such as past members of the Advisory Council or NCC Advisory Boards. 

Fourth, despite efforts to reach key informants involved in the NCCPH conceptualization, design and early implementation phases, many had moved to other positions and could not be located or, in some cases, declined to be interviewed.

Finally, while detailed notes were taken during interviews and sessions, time and resources did not permit transcription and validation of written notes by the participants.