ARCHIVED - Population Health Fund Evaluation 2008 Final Report

 

3. EVALUATION FINDINGS

3.2. Immediate Outcomes

3.2.1 Program Leadership: Indications of more skill in, information about, and/or support for the population health fund and/or the population health approach and/or greater understanding of program design and outcomes.

Project participants - Alberta, Saskatchewan, and Northern Region did not report any increase in knowledge or skill among project participants. Quebec gave 15 anecdotal examples of practices consistent with the population health approach. These examples were not tied to specified PHF supported training or projects. Broader community - Alberta, Saskatchewan, and Manitoba presented no information on increased "knowledge of and support for the population health approach." Northern Region gave an example of increased community based proposals and an example of increased requests for active living programming following PHF projects. Quebec reported (a) many partners and individuals attending PHF presentations requested more information about sustainable development, (b) a guide on hazardous household products was distributed and this guide and the PHF became known through media reports, and after PHF funding some projects incorporate determinants of health into their mission and intervention priorities. Atlantic region reported PHF was a particularly important funder for marginalized or under-represented populations (people living in poverty, francophones, aboriginal peoples, seniors, rural residents, new Canadians, and African Canadians)." While praising the PHF evaluation framework and logic model, the Atlantic Region reported only anecdotal information to infer greater community knowledge or support for the population health approach (PHA).

There is evidence that the PHA is well accepted by project recipients. All but two of the interview respondents said the PHA fits “very well” within their organization, and the other two said it fit “reasonably well.” All said their organizations were currently using the PHA and intended to use it in future work.

Respondents’ comments elaborated on the fit. The PHA was seen as a central or integral part of everything the organization does by 17 respondents; 21 respondents mentioned the usefulness of the determinants of health; and 19 commented on the importance of the partnerships established. Ten mentioned the applicability of the PHA with other funders; six of these had received funding from other funders to continue their work, three said they used the PHA in all their funding applications, and one mentioned another funder who required the same approach.

3.2.2 Delivery Management: Completed projects that generate community knowledge, build community models, and increase community partnerships.

The generation of community knowledge and community models was measured by examining the file evidence that relevant resources had been produced. All but six of the completed projects produced resources that could be used by others. These included resources that would contribute to community knowledge such as research reports produced by 62% of projects, literature reviews from 42% of projects and policy analysis from 40% of the projects. Program manuals to support community-based interventions were produced by 50% of projects. Projects also produced resources to enhance public awareness (67%) and professional awareness (78%). Over one-third (35%) of the projects produced bilingual materials, and 35% produced web-based resources. Other materials produced included videos, promotional items, a theatre production, a support line, cookbooks, and a database.

Of the 44 interview respondents, 42 (96%) said they had produced materials they believed could be useful to other organizations. The types of materials produced are summarized in Table 3.3.

Table 3.3 - Materials Available (Interview responses)
Type of Material Number of Projects % of Projects
Information to support Policy Development 25 56.8%
Information to support Program Development 25 56.8%
Public Information materials 31 70.5%
Professional Information materials 24 54.5%
Training curriculum/Educational materials 27 61.4%
Program manuals/materials 17 38.6%
Other materials 8 18.1%

The files were also reviewed for evidence of partnerships and intersectoral collaboration. All but three of the projects (97%) had at least one partner. A total of 683 partner organizations were involved in these projects. The average number of partners was 5.9, and the largest number was 21. Almost three-quarters of the projects had at least one partner from the health sector, and over half had partners from the social sector. Close to one-quarter had partners from the postsecondary education, research/policy, and cultural sectors. Looked at another way, one-third of the partners were from the health sector and one-quarter from the social sector, with other sectors also represented. (See Table 3.4).

Intersectoral collaboration is a principle of the PHF. Most projects (86%) had partners, including the sponsoring organization, from at least two sectors. The mean number of sectors represented was 3.

TABLE 3.4 Organizational sectors Involved as Project Partners (Files)
Sector % projects with partners from sector %partners from each sector
No partners 2.6%  
Health 73.3% 33.4%
Social 53.4% 22.8%
Primary/secondary education 14.7% 4.4%
Post-secondary education 25.9% 6.9%
Research/policy 23.3% 6.4%
Justice/law enforcement 3.4% 1.6%
Recreation 12.1% 2.8%
Cultural 23.3% 10.2%
Environmental 6.0% 2.6%
Private 10.3% 2.8%
Other 25.0% 6.0%
Total N 116 683

Over two-thirds of projects had a community not-for-profit organization as a partner. Close to 40% of projects had partners from educational institutions, municipal/local governments, or provincial/territorial governments. Coalitions were partners in 35% of projects, and professional associations in 26%. Fewer (under 20%) had partners who were federal government departments or agencies, or private businesses (see Table 3.5). The 28 “other” types of organizations were primarily international organizations or foreign governments.

TABLE 3.5 Types of Organizations Involved as Partners (Files)
Type of Organization % of projects with partners of type % of partners of type
No partners 2.6%  
Community not-for-profit 69.8% 31.9%
Educational Institution 37.1% 12.9%
Professional Association 25.9% 6.0%
Municipal/Local Government 36.2% 12.9%
Provincial/Territorial Government 39.7% 10.2%
Federal Government 17.2% 4.0%
Private Business 16.4% 4.4%
Coalition 35.3% 13.6%
Other 7.8% 4.1%
Total N 116 683

Partner organizations played several roles in project implementation, and some played more than one role. Over one-quarter of the projects had a funding partner, over half had partners who contributed in-kind resources, three-quarters had partners who contributed knowledge and expertise, and close to 40% had partners who had access to or knowledge of the target population (see Table 3.6).

TABLE 3.6 Roles of Partners Involved in Projects (Files)
Role % of projects with partners in role % of partners with role
No partners 2.6%  
Funding 27.6% 9.5%
In-kind resources 54.3% 35.0%
Knowledge/expertise 73.3% 57.7%
Access to/knowledge of target population 38.8% 21.5%
Other 1.7% 0.4%
Total N 116 683

Most interview respondents described putting the principle of partnership into practice as either very easy (41%) or somewhat easy (43%). Six thought it was somewhat difficult and one found it very difficult.

Several challenges were mentioned. Identifying and connecting with appropriate partner organizations was identified as a challenge by six respondents; working with partners with different mandates and perspectives was mentioned by six respondents; four found the time it took to get partnerships up and running was a challenge in that it took time and resources away from other activities; three mentioned geographic and regional distribution presented logistical problems; and three said that some groups had no history of working in partnerships. One mentioned that turnover of contacts was a problem, and one commented that the larger the number of partners, the more complicated it became.

On the other hand, nine respondents commented that partnership is part of their tradition, is essential to doing this kind of work, and that it is just a part of the way they work. Three noted that pre-existing relationships made it easier to work in partnership, and two said it was easy to work with partners who are similar and have interests in common.

Respondents were asked if there were any new partners involved in this project. A large majority (89%) said there were new partners, and most of those (70%) said they were continuing to work with these partners on new initiatives.

Another principle of the PHF is involvement of the target population in the design and delivery of a project or program. From the file review, it was discovered that a large majority of projects (84%) involved the members of the target population in the decision-making related to the project. Among those files that indicated participation, 55% indicated that target group members participated in focus groups, 32% had target group members on an advisory committee, 29% had them involved as volunteers in the implementation of the project, 25% had them as members of a steering committee, and 6% of projects had target group members as paid staff. Another two indicated involvement in implementation, but it was not specified if this was paid or volunteer work. So, overall 37% of projects had target group members directly involved in project implementation. Other types of involvement reported included participation in forums or consultations (six projects), five projects sought feedback from target group members through interviews or surveys, and two had them involved in meetings or working groups. One project offered minigrants to target group members to come up with ideas, and one reported involving them in dissemination.

Volunteer involvement is perceived by PHF staff to be a good indication that the project is building community capacity. It is also a means of leveraging resources. Over one-third of the files (40%) indicated volunteers were involved in the project. Regional projects were significantly more likely to mention volunteer involvement (49%) than National-Directed projects (10%), with National-Solicited projects in between (38%) (X2=10.77, df=2, p<.01). Twenty-two (22) of the files gave information about the number of volunteers involved. The range was from two to 950 volunteers, the median number of volunteers was 34, and the mean number 99. Thirty-five files contained information on the volunteers’ roles. Most (23) were involved in some aspect of project implementation, six in committees, including steering and advisory committees, and six distributed materials.

Volunteer involvement was significantly more likely to be identified by interview respondents than in the files. Among the interview respondents, 40 of 44 (91%) reported volunteer involvement in their project. For those same projects, only 48% of the files indicated volunteer involvement(t=-5.25, df=43, p<.001).

Thirty-nine respondents gave an estimate of the number of volunteers, though only 25 said they formally tracked the number of volunteers involved. Most (67%) estimated that 50 or fewer volunteers were involved, 23% said from 51 to 100 volunteers were involved, and 10% said there were over 100. Twenty-four respondents gave an estimate of the total volunteer hours contributed but only nine said they formally tracked volunteer hours. Estimates ranged from 15 to 25,000 hours, with a mean of over 1500, and a median of about 300 hours. According to respondents, volunteers served on advisory committees (85% of projects), served on steering committees (58%), delivered services/conducted events (90%), and distributed materials (80%).

The PHF has an expectation that projects will evaluate their work. Project outcomes, though not identified as an outcome in the logic model, do have an effect on the communities within which the projects operate. Through the file review and interview process, we attempted to discover the type of outcomes that projects might be producing at the community level. Almost all the files (93%) had some documentation of project outcomes. The number of people reached was noted in 53% of the files, and satisfaction of the target population in 50%. About three-quarters of the files (76%) had documentation of changes in the target population/community. These included changes in knowledge (71%), changes in attitude (39%), changes in behaviour (28%), changes in the health status of the population (7%), or changes in indicators regarding the determinants of health (3%). There were 18 projects (17%) that mentioned other outcomes. These included outcomes related to partnerships (7 projects), capacity building (3 projects), and increased awareness of various groups (3 projects).

Interview respondents were also asked about outcomes. The patterns were similar. However, interview respondents were significantly more likely to say they had evidence of target population satisfaction than was documented in the files for those same projects (91% vs 61%, t =-3.1, df=43, p<.01). They were also more likely to say they had evidence of change in the determinants of health (30% vs. 5%, t = -3.4, df=43, p<.001). The file content and interview responses are summarized in Table 3.7.

Table 3.7 Types of Project Outcomes
Type of Outcome Recorded in File Interview responses
*Significant difference between interviews and files for the same projects.
Number of people reached 52.6% 90.9%
Satisfaction of the target audienceTable 3.7 - Footnote 1* 50.0% 70.5%
Changes in knowledge 70.7% 54.5%
Changes in attitude 38.8% 65.9%
Changes in behavior 28.4% 43.2%
Changes in determinantsTable 3.7 - Footnote 1* 3.4% 29.5%
Changes in health status 6.9% 20.5%
TOTAL N 116 44

Interview respondents were also asked about other outcomes. They reported that 57% of their projects provided increased access to formal/informal networks of support for the target population; that policy changes resulted from 45% of the projects; and 73% led to changes in relationships among community stakeholders. Six respondents described changes to school policies, especially with respect to nutrition and three mentioned inclusion on policy committees at the federal or provincial level. Other policy changes were described in the areas of housing, addiction, child and family, learning disabilities, transportation, and occupational health. In terms of community stakeholder relations, 13 respondents indicated that partnerships had changed things in the community, and 12 described new linkages that had been established between disparate groups including health professionals and employers, seniors and eye care professionals, nurses and dieticians, immigrants and health care services, youth and adults, and service providers and advocacy groups.

Comments about the evaluation process indicated that 37 of the 44 projects had used some systematic data collection tools, including surveys, questionnaires, evaluation forms, interviews and/or focus groups. Three mentioned direct observation. External evaluators had been hired by 11 projects.

3.3 Intermediate outcomes: Community-generated knowledge for program and policy development on the determinants of health and community-based models to act on the determinants of health are more broadly accessible.

All but three of the respondents (93%) said they had made efforts to disseminate the materials they had produced. Among these 41 respondents, 83% reported using the web, 59% used the mail system, 61% used media, 78% made local presentations, 85% made regional or national presentations, 73% held workshops, and 81% distributed on-site or through partners. Other dissemination channels were reported by seven (17%) respondents. The number of different channels used ranged from two to eight, with a mean of 5.4.

When asked if anything had limited their attempts at dissemination, 22 respondents (54%) reported limitations due to time or money, though some mentioned that the web and e-mail helped with affordability. Four reported difficulty accessing those who might use the materials. Three reported that there was a delay in finalizing the materials either as a result of the summer hiatus or because they were waiting for feedback and review.

3.4 Final outcome: Increased use of community-generated knowledge for program and policy development and increased application of community-based models to act on the determinants of health.

Over three-quarters of the respondents (77%) reported that they were still using the materials in their organization. Almost two-thirds(64%) say they know of other organizations that are using the materials. Of these 28, 27 gave an estimate of how many other organizations were using the product. This ranged from 1 to 14 organizations, with a mean of 3.7.

Of the 28 respondents who said materials were being used by other organizations, 24 named one or more specific organizations that were using them. The other four named categories of organizations such as schools, unions, public health units, pharmacies, and community centres, but did not give specific organizational names.

The case studies provide some examples of the longer term outcomes and sustainability of projects (See Appendix D).

“Garderie Bio: Manger et Grandir” in Quebec, established a network linking organic farms, child care centres, and citizens. It was funded for 26 months between 2001 and 2004, and was still active in 2008. By fall of 2007, 55 child care centres serving over 4000 children were linked with 33 organic farms. Diets for the children improved, and the farms’ production, incomes, employment and hectares devoted to organic food increased.

A Portrait of the Food Industry Relating to Residents of the Lower North Shore, also a Quebec project, resulted in the establishment of five community kitchens and five community gardens, provided nutrition and cooking classes at three high schools and the community kitchens, and developed business plans for local food production. Ongoing partnerships and activities have been established, and they have continued after the end of funding in 2008.

Project L.I.N.K. for Health in Alberta implemented a Healthy Messages campaign targeted at students and community members through all eight schools in the Holy Family Catholic Regional School Division in Northern Alberta. It emphasized cultural appropriateness for aboriginal students. The outcomes of the project are that all soft drinks and sweetened beverages in school vending machines have been replaced with water, milk, and 100% fruit juices; school cafeteria and canteens now serve food that follows the Canada Food Guide; and fruit, rather than coffee and donuts, is served at staff meetings. PHF funding ended in 2004, but the project continues with a new dietitian starting in October 2008.

Two ongoing projects illustrate the planning for sustainability done by projects. Healthy Housing, Healthy Community Project in Nova Scotia developed tools to assist in designing and developing healthy communities and healthy housing projects. The tools are generic and can easily be used by any size community or neighbourhood to assess and identify specific action areas for improving the health of the community. The process and tools have been adopted by the Halifax Regional Municipality to assist in setting a development framework for the next 20 years.

Atatittiniq-Making the Links between Inuktitut Language Acquisition and Literacy in Nunavut was funded by the PHF 2005-2009. The expected results of the project are increased capacity of communities to promote increased literacy levels and thereby positively affect other social and health issues. The territorial government through the Department of Human Resources and Skills Development has expressed an interest in working with the project’s sponsor, the Nunavut Literacy Council, after PHF funding ends March 31, 2009.

3.5 Directed National Projects

There are 28 national projects funded outside of the normal solicitation process to further priorities of PHAC, 21 completed and seven ongoing projects. Included were eight international projects: three international conferences (two of which were held in Canada), one summit of heads of governments, and four projects to support World Health Organization (WHO) projects on chronic disease and seniors issues. Six projects provided start-up funds for the National Collaborating Centres (NCC). Three ongoing projects were targeted to official language minorities. Another four completed and four ongoing projects were Canadian in focus with six focusing on specific health issues, one on communication with physicians, and one on the role of the voluntary sector in health.

Interviews were conducted with individuals representing four clusters of projects: the WHO Chronic disease projects, the WHO-Seniors projects, the NCC projects and the pandemic preparedness projects. The same interview protocol was used as for the other projects.

All of the respondents saw a good fit with the PHA, though the NCC respondent indicated that not all of the NCCs do population health work. Partnerships were involved in all the projects and seen as easy to implement. The NCC respondent acknowledged trust issues at the beginning of the process. However, their partners continue to collaborate. On the other hand, the respondent for the pandemic preparedness projects indicated that the ability to maintain the networks was lost with the end of funding. All used volunteers. The NCC projects had about 100 volunteers involved contributing about 200 hours of work. The WHO-Chronic disease projects had experts and NGOs as volunteers and the Seniors projects had volunteers from over 20 countries. The pandemic preparedness projects has 11-20 volunteers, and estimated 500 hours of volunteer work had been contributed.

All of these respondents reported that materials had been developed contributing to knowledge and/ or public and professional awareness. The NCC respondent indicated that some program materials have also been developed. All said the materials were being disseminated through multiple channels, and all were using or distributing the materials on an ongoing basis. The WHO projects and the pandemic preparedness projects identified that others were also using the materials.

In terms of outcomes, only the pandemic preparedness projects reported a formal evaluation. It indicated that the participants were satisfied with their involvement in the process, and saw the materials provided as useful and credible. The NCC respondent indicated that a formal evaluation was underway. The WHO-Seniors respondent saw the increasing demand for information as an indicator of success and noted that eight Canadian provinces had adopted the framework they developed. The WHO-Chronic disease respondent indicated that there was policy development occurring in the Americas through two organizations.

When asked for additional comments, the NCC respondent indicated that the PHF provided support to the process at a time when it was really needed. She also commented that staff were very knowledgeable and available when needed. The WHO respondents indicated that they would like to see this type of funding available on an ongoing basis, that criteria for funding international work needed to be developed, and that reporting requirements needed to be minimal. The pandemic preparedness respondent indicated that this was a terrific experience.

The case study submitted by the National office is a National-Directed ongoing project: We Can Act: A Health Promotion Strategy for Quebec’s English Speaking Communities. It has been supported by the PHF since 2004. The project’s sponsor, the Community Health and Social Services Network (CHSSN) adopted a three-point strategy: Mobilizing multi-level community-public partnerships; building community capacity to understand and communicate health promotion approaches; and developing data-based community profiles related to the determinants of health (“vitality profiles”) using census and other relevant data. Accomplishments which will probably sustain this project in the long term include: The formation of a formal strategic partnership between CHSSN, the Ministère de la santé et des services sociaux and the Insititut national de la santé publique; development of a community action model to identify a common and comprehensive approach for all involved; and a five year action plan from 2009-2013 which has resulted in the commitment of new federal investments. Pilot projects have been completed in eleven different regions in Quebec, and regional networks and public health authorities have assumed ownership and sustainability of ongoing initiatives. In addition, CHSSN and the regional networks have begun sharing the tools and information with other official language minority communities across Canada, raising the possibility that this work will have effects beyond the boundaries of Quebec.

Page details

Date modified: