ARCHIVED - Population Health Fund Evaluation 2008 Final Report

 

2. EVALUATION METHODOLOGY

2.1 Data Collection

The 2008 evaluation is based on116 projects completed between October 1, 2005 and October 31, 2008. Of these, 95 were projects funded through the regular solicitation/invitation processes at the National and Regional level. The remaining 21 were projects funded at the National level to further PHAC priorities. These 21 will be referred to as Directed Projects. The number of completed projects in each region, and the fiscal year in which they first received funding, are summarized in Table 2.1.

Table 2.1 - Projects by Region and Fiscal Year Started
Region Fiscal Year Started Total
2002/03 or earlier 2003/04 2004/0 5 2005/06 2006/07 2007/08
*For two projects, one in Atlantic and one in Quebec, the start date could not be discovered from the file
Atlantic 0 0 3 16 3 0 23Table 2.1 - Footnote *
Quebec 1 0 8 1 2 0 13Table 2.1 - Footnote *
Ontario 0 0 0 8 0 0 8
Man/Sask 7 3 1 3 0 0 14
Alberta 0 0 0 6 0 0 6
B. C. 0 0 5 0 0 0 5
Northern Secretariat 0 0 1 1 0 0 2
National - Solicited 0 0 0 24 0 0 24
National - Directed 0 0 7 6 6 2 21
Total 8 3 26 68 12 2 116Table 2.1 - Footnote *

Because there were small numbers of projects from some regions, no attempt will be made to do comparisons among the Regions. Comparisons will be done among Regional, National-Solicited, and National-Directed projects.

Each of the 116 files was reviewed using the same template as the 2006 evaluation (See Appendix A). In addition, a random sample of 50% of the solicited projects, stratified by Regions, was chosen for interviews, using the same interview protocol as the previous evaluation (See Appendix B). Key contacts for the project were contacted and asked to participate in the interview. A copy of the interview schedule was e-mailed to the respondents before the interview was conducted. Interviews were conducted in English or French depending on the respondent’s language preference. They were conducted by research associates in the PHAC national office. Of the 50 projects selected for the interview process, 44 were successfully interviewed, a response rate of 88%.

The National-Directed projects were excluded from the interview sample because of the different nature of these projects. However, four interviews were conducted with key respondents representing clusters of the National-Directed projects. They are discussed separately.

Each region, the Northern Secretariat, and the national office were asked to provide one case study of a project that demonstrated the broad and long-term impact projects could have. Three of the regions submitted case studies, with one region submitting two (Appendix D). Case studies were also submitted from the Northern Secretariat and the National office. These case studies were analyzed to illustrate the leveraging of resources, and/or the ongoing impact of projects after they were completed.

Uniform systematic reporting of PHF outputs and outcomes related to Program Leadership consistent with the PHF logic and structure model was not built into the PHF evaluation strategy. In an attempt to gather output and outcome information, staff and managers with responsibility for the PHF in each region (excluding the national office) were asked to complete a regional questionnaire to summarize their activities in Program Leadership in their regions since 2005, and the perceived outcomes of these activities (See Appendix C). Four of the six regions submitted a questionnaire, and one was received from the Northern Secretariat.

2.2 Limitations

This evaluation incorporates multiple lines of evidence and a combination of qualitative and quantitative measures to ensure a balanced analysis of the evaluation questions. However, there are some limitations that should be noted:

Reporting bias: The data analysis was completed by a third party evaluator. The file review and interview data were obtained and provided by internal PHAC staff. Some of these individuals work with the program staff and may have a vested interest in the program. Attempts were made to compensate for potential reporting bias by training staff, utilizing standardized protocols and providing ongoing support throughout data collection.

Data sources: Much of the data was gathered from the project files kept by PHAC. Because the reporting requirements did not specify in detail what information should be recorded, there may be some under reporting of some variables such as volunteer involvement. The regional questionnaires completed by regional staff were asking for retrospective data over the past three years. The validity of this data could be questioned as a result of such things as staff turnover and the vagaries of memory. The interviews too were retrospective and their accuracy may also be questioned.

Effects on determinants of health and health status: These effects were not part of the Logic and Structure model for the PHF. Many of the most important factors that affect population health in Canada are beyond the scope of the PHF. In addition, the size of the program is such that it is unreasonable to expect it to have measurable effects on health status for the Canadian population at large. A narrow focus on selected long-term indicators would have obscured meaningful successes of the PHF and presented a misleading portrayal of accomplishments. The evidence of population health status effects presented by some projects may be attributable in part to factors other than those projects.

Funding uncertainty: The program has experienced challenges in the ongoing development and implementation of program components during the time period under consideration. In particular, funding uncertainties were linked to the move from the Population and Public Health Branch of Health Canada to PHAC.

Page details

Date modified: