ARCHIVED - Promotion of Population Health Grant and Contribution Programs: Summary of Program Evaluations, 2004-2009
10. Design and Delivery
The evaluation question on “design and delivery” in the PPHGC evaluation strategy is:
How have programs approached the design and delivery of programs?
(What has worked well, what has not?)
The associated indicators are:
- number and types of activities; associated outputs related to research/knowledge development, communication and dissemination strategies, policy development, intersectoral collaboration and capacity building; and
- best practices and recommendations for program change and improvement.
All evaluations, with the possible exception of the ISHLCD implementation review and part of the CDS evaluation, rated program design and delivery positively. Program design and delivery was seen as sound, reflecting well the implementation of a promotion of population approach, and contributing to the realization of expected outcomes as identified in the PPHGC logic model. The same evaluations also reported that the programs are well-managed and costeffective or providing “value for money”, supported by evidence of their activities and output in: knowledge development and dissemination; informing the development of policies, programs and services; collaboration; capacity building; the involvement of target populations in project/program governance and delivery; and the resources contributed by partners and collaborators and, in particular, in-kind contributions and the efforts of volunteers.
The specifics of the key activities and associated outputs are discussed in the assessments of impact and effect, above. Most evaluations identified the types of activities and outputs in general terms and some (such as CAPC) provided quantitative information.
The evaluation question of “what worked well, what has not” was generally not addressed directly or, if addressed, resulted in long lists of activities positive and negative.
The most frequently identified opportunities for program change and improvement in the evaluations were:Footnote 10
- improved coordination and better defined relationships among regions and between regions and the national office and systematic coordinated strategic planning and priority setting;
- funding:
- recognition that administration (including for coordination and evaluation) needs to be resourced as well;
- better match between desired outcomes and funding: some projects, such as those targeting prevention, need to be long-term but funding tends to be short-term and, often, at a too-small scale;
- knowledge management and tool development: consolidating learning or evidence on different approaches to program design and delivery in one organization, which would also be responsible for synthesizing that information, and then developing and disseminating tools and guides that are better tailored to the maturity or knowledge level of potential users (including fund recipients) so that their learning curve could be reduced;Footnote 11
- administrative process: reduce administrative burden, particularly reporting to PHAC, and more timely decisions, especially on funding; and
- synergy: promote greater collaboration, coordination and integration between programs dealing with related issues, risk factors or strategies; using similar approaches; or dealing with the same stakeholder communities.
The Centres of Excellence for Children’s Well-being further noted that despite the generally positive progress in the implementation of the Program, “… the Centres have certainly not achieved their full potential: the progress has not been consistent across the Centres; they have not yet established themselves as a national Program; and most have only begun to engage decision-makers and influence policy in the last year.” (p. 47)
The CDS evaluation reported that key informants perceived the implementation of the Strategy’s National Coordination Component as “… poorly conceived, poorly planned and, until recently, poorly managed” (p. 19) – but considered the development of the National Diabetes Surveillance System a resounding success. The main causes of the negative assessment of the National Coordination Component were uncertainly and turbulence arising from the restructuring of Health Canada at the time of the evaluation, program staff turnover, and differences between the two main (then) Health Canada branches involved in the Strategy.
11. Cost-Effectiveness
Cost-effectiveness is the comparison of the relative cost of alternative ways to achieving a given outcome. Cost-effectiveness was not identified in the umbrella PPHGC evaluation strategy.
Nine of the 14 evaluations addressed cost-effectiveness, even if it is briefly. The exceptions were the four formative/implementation evaluations and the AHSUNC evaluation. However, addressing the question of cost-effectiveness has proven to be a challenge for most of the evaluations. All but one of the nine evaluations either noted that the information was not available or fell back on quoting stakeholder opinions that the program is cost-effective or provides “value for money”. One of the better examples of the latter is the following from the Falls Prevention Initiative evaluation:
“Available evidence from interviews with key informants and project coordinators, as well as from the literature, suggests that this type of health promotion program is a cost-effective approach because of its reliance on existing community resources and volunteers, leveraging of in-kind and some financial resources, likelihood of some enduring impacts (e.g., continued use of tools developed, enduring partnerships) and the ultimate expected savings in health care costs due to a reduction in fall-related injuries.” (p. 59)
The CHN evaluation also acknowledged that assessing cost-effectiveness was challenging because of the lack of benchmarking data and of cost and usership information for similar websites. Nevertheless, it was able to draw on cost per user data from Industry Canada’s Strategies and user-pay studies in New Zealand and the United Kingdom to provide some objective, quantitative insight into CHN’s cost-effectiveness.
12. Conclusions
This review of the PPHGC component-program evaluations conducted in the 2004 to 2009 period, to respond to the umbrella PPHGC RMAF evaluation questions finds that, in general, the PPHGC component programs have contributed substantively to supporting PHAC’s achieving its strategic outcome of “healthier Canadians, reduced health disparities, and a stronger public health capacity”.
12.1. Conclusion on Relevance
Through its population health commitments and actions, including the recent creation of the Public Health Agency of Canada, the Government of Canada has acknowledged the continued relevance of federal involvement in the promotion of population health.
12.2. Conclusion on Relevance
The evaluations reviewed for this report, in general, provided significant evidence on the contribution of the PPHGC programs toward achieving the outcomes articulated in the PPHGC logic model and performance measurement strategy through the implementation of its key activities in:
- research and knowledge development and integrated communication and dissemination strategies;
- leadership and policy development;
- intersectoral collaboration; and
- capacity building.
The programs appear to consistently apply the principles that underlie a population health approach: evidence-based; focused on empowerment, participation, and capacity building; emphasized intersectoral involvement, collaboration and partnerships; and used multiple strategies that are culturally and gender appropriate. They are also health-determinants focused, with programs emphasizing different aspects, depending on their objectives – ranging from disease-specific programs that tend to focus on prevention (including increased public awareness using culturally appropriate approaches; targeting high-risk populations) and mitigation (e.g., personal health practices and coping skills of those with the disease and of those in their support networks, improved access to health services, improving social acceptance), to comprehensive service or research programs targeting children, women or other at-risk populations, that cover a broad set of determinants.
Evidence of achievement of outcomes articulated in the PPHGC logic model and performance measurement strategy and reported in the program evaluations reviewed is presented in Table 2 below.
| Desired outcome | Achievement |
|---|---|
| Improved capacities for influencing actions on the determinants of health (immediate outcome) | Capacity for influencing actions on the determinants of health has been strengthened through PHAC and partners and collaborators contributing both monetary and in-kind resources. Intellectual capability and skills have also been strengthened through the sharing of knowledge, training and networks among program participants and the large number of volunteers (including members of the target population) involved in community organization governance and program delivery. |
| Improved coordination for intersectoral cooperation among governments, sectors and stakeholders (immediate outcome) | All programs (with the possible exception of the Integrated Strategy on Healthy Living and Chronic Disease) have developed well-functioning collaborative efforts with parties ranging from other government departments, other orders of government, businesses, non-profit and professional organizations, and schools and other institutions to individuals and, for some programs, internationally. This outcome area is often cited as a particular strength in many programs. |
| Implementation of strategies and policies to support health promotion (immediate outcome) | All PPHGC programs embed the characteristics of a population health approach in their implementation. The existence of a large number of guides and toolkits on the implementation of a population health approach at the provincial and local levels indicates not only broad support for health promotion but also for the population health approach. PHAC’s recent launch of the initial elements of the Best Practices System will improve not only health promotion practice and policies but also community capacity (by increasing potentially both ffectiveness and efficiency). |
| Increased awareness and use of reliable health related evidence (immediate outcome) | PPHGC programs reported considerable output in research and knowledge and its dissemination. Knowledge development products include project and program evaluations, directed and solicited research, and shared learning from hands-on program delivery and surveillance activities. Dissemination is often through a mix of approaches, such as web-sites; the regular mail system; print, broadcast and electronic (audio, video, CD) media; local, regional, national, or international presentations at conferences; workshops; and face-to- face dealings with potential users. |
| Evidence base to shape promotion of population health policy and practice (intermediate outcome) | Through learning/lessons/best practices from ongoing program operations, collaborative efforts with research organizations such as the Canadian Institutes of Health Research, and the direct funding of research organizations and networks such as the Centres of Excellence for Children’s Well-being and the National Collaborating Centres for Public Health, an increasingly more comprehensive evidence base to support the development and implementation of promotion of population health policy and practice is being developed. An evidence-based approach to decision-making is a key promotion of population health principle. Policy and practice changes reported in the evaluations indicate that this evidence is being used. |
| Strengthened health promotion policies and actions within thehealth system (intermediate outcome) | While none of the evaluations systematically assessed performance in this outcome area, the close working relationships, collaboration and networking among the different orders of government and between government and non-government organizations and groups at all levels – from community project working groups, and privatevoluntary- pubic sector consultations, and federalprovincial- territorial committees, to ministerial and international conferences and meetings – can be expected to contribute to strengthening and better aligning health promotion policies and actions within and outside the health system. |
| Improved access to health and social services for target populations (intermediate outcome) | The evaluations reviewed reported that PPHGC programs have provided information and tools to help members of targeted populations access services they need. Some programs, such as CPNP, AHSUNC and CAPC, provide services directly. Others work with policy or delivery organizations to address identified gaps. An often noted strength of PPHGC programs is their flexibility and readiness to adapt to local conditions and needs, including the strategic location service centres and using approaches that are culturally appropriate. Training and other support services provided through PPHGC programs also contribute to improving the health literacy of their target population so that they can make better decisions on the services they want and need. |
| Long-term and ultimate outcomes:
|
None of the evaluations addressed these longer term outcomes because they were considered too remote. Nevertheless, if the PPHGC programs are achieving their immediate and intermediate outcomes – and the evidence appears to indicate that they generally are – then the longer term outcomes should be achieved as well. If the “improved personal health practices and skills” outcome were interpreted at the program level, however, some programs such as the Falls Prevention Initiative and CPNP, reported contributions. |
12.3. Conclusion on Design and Delivery
Most evaluations rated program design and delivery as sound, reflecting well the implementation of a promotion of population approach and contributing to the realization of expected outcomes as identified in the PPHGC logic model and reported on above. The same evaluations also reported that, generally, the programs are well-managed and cost-effective or providing value for money.
12.4. Conclusion on Cost-Effectiveness
Determining cost-effectiveness has proven to be a challenge for most of the evaluations. The evaluations either noted that the information was not available or fell back on quoting stakeholder opinions that the program is cost-effectiveness or provides “value for money”.
13. Recommendations
All the evaluations provided examples of challenges, lessons learned and recommendations arising from them. The observations noted here are at a high level and touch on those areas that are noted in at least two evaluations. The recommendations are based upon the findings and lessons learned.
RECOMMENDATION 1: PHAC should develop a framework and strategy for its evaluation activities that would standardize: terminology; key issues; evaluation design; and the evaluation report content and format.
RECOMMENDATION 2: PHAC should continue its requirement for a management action plan in response to each evaluation and its implementation should be monitored.
RECOMMENDATION 3: PHAC should complement program evaluations with more strategic evaluations.
Bibliography
- Canada. Health Canada. Evaluation Synthesis: “Looking Back, Looking Forward” -- A Summary of Six Population Health Contribution Program Evaluations of the Population and Public Health Branch, Health Canada. Ottawa: Health Canada, 2003.
- Canada. Health Canada. Strategies for Population Health: Investing in the Health of Canadians. A discussion paper prepared by the Federal, Provincial, Territorial Advisory Committee on Population Health. Ottawa: Minister of Supply and Services Canada, 1994.
- Canada. Public Health Agency of Canada. Population Health Approach: the Organizing Framework. Ottawa, Public Health Agency of Canada. <http://cbpppcpe. phac-aspc.gc.ca/population_health/index-eng.html>
- Canada. Public Health Agency of Canada. Public Health Agency of Canada 2008- 2009 Report on Plans and Priorities. Ottawa: Public Health Agency of Canada, 2008.
- Canada. Public Health Agency of Canada. The Public Health Agency of Canada Strategic Plan: 2007 – 2012, Information, Knowledge, Action. Ottawa: Public Health Agency of Canada, 2007.
- United States General Accounting Office. Program Evaluation and Methodology Division. The Evaluation Synthesis. Washington: Government Printing Office, Revised March 1992.
Page details
- Date modified: