ARCHIVED - Population Health Fund Evaluation 2008 Final Report

EXECUTIVE SUMMARY

The Population Health Fund (PHF) is a grants and contributions program that was established in 1997. It funds voluntary not-for-profit organizations and educational institutions to develop knowledge and program models to address the determinants of health.

The Results Management and Accountability Framework (RMAF) for the PHF specified two major streams of consultant activities: Program Leadership to prepare the community, including governmental organizations, to understand and accept the Population Health Approach (PHA), and Delivery Management to identify and support funding recipients to implement proposed projects.

This is the third evaluation, and it focuses on projects completed between October 1, 2005 and October 31, 2008. It was designed to discover the extent to which the program had achieved the objectives specified in the RMAF, and assess the ongoing relevance, implementation, and cost-effectiveness of the program.

METHODOLOGY

During the time period under consideration, 116 projects were completed. Of these, 71 were funded through Regional solicitation/invitation processes, 24 were solicited at the National level, and 21 were National directed projects funded to promote the Public Health Agency of Canada (PHAC) priorities. Another 29 projects received funding during this period but are still ongoing, 20 regional projects, two national-solicited projects and seven national-directed projects.

Files for all of the 116 completed projects were reviewed using a standardized protocol. A random sample of 50 of the solicited projects was selected for interviews. Standardized interviews were conducted with people involved in implementation of 44 of these projects, a response rate of 88%. In addition, regions and the Northern Secretariat were asked to complete a regional questionnaire about the activities of consultants in their Program Leadership role. Regions, the Northern Secretariat, and the National office were also asked to submit case studies of a project which demonstrated a broad and continuing impact after the project was completed. Three regions, the Northern Secretariat and the National office submitted a total of six case studies. Four regions and the Northern Secretariat responded to the regional questionnaire.

EVALUATION FINDINGS

Activities and Outputs:

With respect to Program Leadership, the regions who reported indicated extensive involvement in intersectoral networks, and most had issued a variety of reports related to the PHF. They also reported having a variety of events to offer orientation and training to PHF recipients. All reported making presentations to inform a variety of community and governmental audiences. Their work in Delivery Management was evaluated highly by interview respondents. The vast majority (80% or more) described the application guidelines as clear, and rated consultant support as excellent or good. Comments described consultants as knowledgeable, helpful and available. Difficulties experienced in the delivery of the program centred around reporting requirements, especially financial reporting and time lines.

Of the 116 completed projects, 33 were grants and 83 were contribution agreements. The mean duration of projects was 25.5 months. Contribution agreements had significantly longer durations than grants. The mean allocation was $228,297. National solicited projects had longer durations than either regional or national directed projects. This reflects the fact that all but one of the national directed projects were grants.

Most projects had a program development objective, though policy development and implementing a community-based intervention were also frequent objectives. Most project sponsors were from the health or social service sectors, and were either community not-for-profit organizations or educational institutions. There were some differences between the regional, national-solicited, and national-directed projects in recipient sectors and types of organizations funded.

Projects were about evenly split between those that targeted community members and those that targeted intermediaries, except among the national-directed projects where 95% targeted intermediaries. Ten projects supported official language minority communities. About two-thirds of the projects targeted a vulnerable or marginalized population.

The projects addressed a variety of health determinants, and most targeted multiple determinants. Social support networks, the social environment and personal health practices and coping skills were the most frequently addressed.

Immediate Outcomes

There is little systematic evidence of greater community acceptance and support for the PHA during the time period under study, but it may be that the acceptance had been developed during earlier periods. There is evidence that the PHA is well accepted by project recipients, and that it has affected the way their organization operates.

The file review indicated that all but six of the projects produced resources that could be used by others. These included research reports, literature reviews and policy analyses, but also included program manuals, and public and professional awareness materials. All but two of the interview respondents also said that they had produced materials that could be useful to other organizations.

The principle of partnership was well integrated into project implementation. All but three projects had at least one partner, and a large majority had intersectoral partnerships, with three sectors represented on average. Partners played many roles, including providing knowledge and expertise, contributing in-kind resources, and providing access to or knowledge of the target population. A little over one-quarter provided additional funding for the project. Most interview respondents reported that it was fairly easy to implement the principle of partnership.

A large majority of projects involved the target population in the design and delivery of the project. A little more than a third of the project files indicated volunteer involvement in the project. However, almost all of the interview respondents reported volunteer involvement. It is likely that volunteer involvement is under reported in the files.

Almost all files had some documentation of project outcomes, and three-quarters reported changes in the target population. Most often, there was documentation of changes in knowledge, though behavior change was reported in about a quarter of the files. A similar pattern emerged from the interview responses, though they more often reported changes in the determinants of health in comparison to the file information. Most respondents indicated some attempts to collect systematic data using formal evaluation tools.

Intermediate Outcomes

In examining the extent to which materials were broadly accessible, we asked interview respondents about their dissemination efforts. All but three said they had made efforts to disseminate the materials, and most used multiple channels. Most frequently, presentations at the local, regional or national level, distribution on-site or through partners, or postings on the web were identified as the means of dissemination. About half of the respondents reported that their efforts at dissemination were hampered by limitations of time or money.

Final Outcome

Evidence of ongoing use of project materials or resources, and evidence that these resources were being used by others, was used to evaluate the final outcome. Over three-quarters of the interview respondents said the materials were still being used in their organization, and almost two-thirds said they knew of other organizations that are using the materials. The case studies also provided positive examples of longer term outcomes and the sustainability of projects.

Directed National Projects

These are projects that were funded to further the priorities of PHAC. They include 21 completed and seven ongoing projects. Eight of these were international projects, six provided start-up funds for the National Collaborating Centres, three targeted official language minorities, six targeted specific health issues, and one each addressed communication with physicians and the role of the voluntary sector in health.

Interviews conducted with respondents representing clusters of projects indicated that, in general, the principles guiding the PHF were implemented. All involved partnerships, all used volunteers, and all produced materials of use to others. The case study submitted at the national level indicated how a project can develop a model that can spread to other regions and produce a plan for ongoing development.

CONCLUSIONS

Relevance

The objectives and processes of the PHF are consistent with PHAC’s Strategic Plan. The official language minority projects contribute to meeting PHAC’s responsibility under Section 7 of the Official Languages Act. Community recipients view the PHF and PHA as highly relevant to their work, and would like to see continuing availability of funds to do this kind of work.

Implementation

The implementation of the PHF program seems to be sound from the point of view of the recipients.

Effectiveness

The PHF has achieved the immediate and intermediate outcomes specified in the RMAF. They developed resources in partnership with multiple sectors and disseminated them to others. There is some evidence that the projects produced changes in the community in which they were implemented.

In terms of final outcome, the respondents report ongoing use of the materials, indicating that projects were sustainable. Respondents also say that other organizations have used the materials. We were not able to gather evidence from those other organizations regarding the extent of use, though the case studies demonstrate that projects often continue and spread after PHF funding ends.

Success

Success in building community capacity is not strongly demonstrated in this evaluation. There were no direct measures of community capacity. However, the availability of resources and the partnerships established suggest that community capacity is enhanced. Furthermore, the case studies indicate that recipients have the capacity to sustain action after the funding ends.

Cost Effectiveness

A formal cost effectiveness evaluation was not completed. However, the development of funding partnerships, partners’ contributions of in-kind resources, and the involvement of volunteers suggest that the PHF investment may be effective in attracting other resources. Some projects are picked up and funded by others and some continue long after PHF funding has ceased.

RECOMMENDATIONS

Two recommendations were made.

1. Future programs designed to encourage action on the determinants of health should build on the strengths currently exhibited by the PHF.

The strengths identified include the positive relationships developed by program consultants with organizations in their regions, the rich networks of partnerships that have been formed, the positive outcomes of many of the community-based projects, and the diversity of issues that have been addressed by the PHF.

2. Efforts should be made to improve evaluative data gathering and reporting.

The Logic and Structure Model should be reviewed and revised as necessary. Then reporting formats and systems should be developed so that information about key elements can easily be retrieved. This needs to be done in a way that does not place an undue burden on fund recipients. Regular reports on achievements at the Regional level would be useful to illustrate the strengths of the program, and to identify any issues that may arise.