Health Canada
2002
ISSN 1496-466 X
ISSN 1499-3503 (Online)

Table of Contents

• Health Promotion: Does It Work?
• Why? What? How?
• Case Studies
• Moving Forward
• Who's Doing What?
• Regular Columns
• Mark Your Calendar

Health Promotion - Does it Work?

Like most areas of the public sector, the health sector has come under increasing pressure to demonstrate the effectiveness of its policies and programs. As a significant component of most national health strategies, health promotion is also under considerable "pressure to perform."

Since the release of A New Perspective on the Health of Canadians in 1974, Canada has played an important role in shaping health promotion. From an initial focus on behavioural change, health promotion has evolved to include a comprehensive array of activities designed to influence not only health behaviours, but the underlying determinants of health as well. As a result, present day health promotion is a key strategy in a population health approach, one that involves multiple interventions, often for prolonged periods of time, and relies on action at several levels and across a number of sectors. This complexity raises a number of important questions - for example:

• While improved health status is the ultimate aim of health promotion interventions, what immediate outcomes can be expected?
• What types of indicators and methods can be used to assess whether these outcomes have been achieved?
• What is the most effective "mix" of interventions?

This issue of the Health Policy Research Bulletin explores these and other challenges and presents some "real life" examples of how they are being addressed. It also highlights a major international evaluation initiative, as well as other important initiatives under way across the country.

About Health Canada

Health Canada's Health Policy Research Bulletin is published three times a year. The Bulletin is part of a larger policy research dissemination program designed to enhance Health Canada's policy-relevant evidence base.

A departmental steering committee guides the development of the Bulletin. The committee is chaired by Cliff Halliwell, Director General of the Applied Research and Analysis Directorate (ARAD) of the Information, Analysis and Connectivity Branch. The Research Management and Dissemination Division (RMDD) within ARAD coordinates the Bulletin's development and production. RMDD would like to thank the steering committee members for their contributions, as well as Nancy Hamilton, Managing Editor, Marilyn Ryan, Production and Distribution, and Jaylyn Wong, Assistant Editor. Special thanks go to the Guest Editor of this issue, Jim Ball, Director of the Population Health and Health Promotion Development Division, Population and Public Health Branch.

We welcome your feedback and suggestions. Please forward your comments and any address changes to bulletininfo@hc-sc.gc.ca or phone (613) 954-8549 or fax (613) 954-0813.

Health Policy Research Bulletin

The opinions expressed in these articles, including interpretation of the data, are those of the authors and are not to be taken as official statements of Health Canada.

This publication can be made available in alternative formats upon request.

Permission is granted for non-commercial reproduction provided there is a clear acknowledgment of the source.

Published under the authority of the Minister of Health.

© Her Majesty the Queen in Right of Canada, represented by the Minister of Public Works and Government Services Canada, 2002

ISSN 1496-466 X
ISSN 1499-3503 (Online)

Editing, design and layout by Allium Consulting Group Inc.

Publications Mail Agreement Number 4006 9608
Return if undeliverable to:
Health Canada
2750 Sheffield Road, Bay #a1
Ottawa, Ontario
K1B 3V9

Why Assess the Effectiveness of Health Promotion

The following article is based on an interview conducted by Nancy Hamilton, Managing Editor of the Health Policy Research Bulletin, with Dr. Robert McMurtry, former Assistant Deputy Minister of the Population and Public Health Branch, Health Canada. A former Dean at the University of Western Ontario, Dr. McMurtry joined the Department in 1999 as the first G.D.W. Cameron Visiting Chair. As Assistant Deputy Minister, Dr. McMurtry was responsible for a broad spectrum of health promotion and disease prevention and control policies and programs. Dr. McMurtry has recently left the Department to act as a consultant to the Romanow Commission on the Future of Health Care in Canada.

This issue of the Health Policy Research Bulletin addresses the theme: "The Effectiveness of Health Promotion Interventions." How important is this theme for the health sector?

Historically, public sector involvement in health has involved both "one-to-one" patient-provider care through the health care system, and public health measures such as providing safe water and containing contagious diseases. I think it's safe to say that our health care system was initially intended to serve as a "safety net" for dealing with catastrophic illness. Over the years, however, considerable demands have been placed on the system by illnesses and injuries, initially infectious diseases and then chronic conditions such as heart disease and cancer, that to a considerable degree are preventable. While it has often been argued that today's health care system is threatened by the aging of the population, I believe it is threatened more by a failure to move "upstream" to address the factors and conditions that can prevent disease and improve health. Health promotion activities have a significant potential to impact positively on the well-being and quality of life of Canadians, and that is why we need to ensure they are as effective as possible.

When you talk about "moving upstream," what types of interventions are you referring to?

By "upstream" interventions, I mean those interventions that are intended to help people maintain or improve their health before it is compromised. To understand how health promotion fits into this "mix" of interventions, it's important to clarify some terms that are often used interchangeably. To do so, it helps to consider these interventions along a continuum, from the "downstream" interventions provided within the context of the health care system to "upstream" interventions such as health protection, disease prevention and health promotion.

Most people would agree that the major role of the health care system is to restore health once it's been threatened. Health protection and disease prevention, on the other hand, are concerned with maintaining health status by addressing immediate health threats (health protection) or anticipating and avoiding imminent health threats (disease prevention). Health promotion goes even further. It moves beyond maintaining health to improving health status and, consequently, is concerned with health gains.

Over the past couple of decades, health promotion has generated a great deal of interest and support. Yet, you say there's been a failure to move upstream. Could you clarify what you mean by this?

While we've made considerable progress in implementing health promotion policies and programs, we still need to achieve a better balance in our investments as we move along the continuum from health care to health promotion and addressing the broader determinants of health. In fact, it's unimaginable to me that we will be able to sustain our health care system if we don't focus more of our attention on upstream factors - in particular, those affecting the health of expectant mothers and our children. We also need to make the public more aware of the importance of these upstream interventions.

How does health promotion fit with the growing emphasis on population health?

I believe population health and health promotion should be synergistic. A population health approach aims to reduce health inequalities in the population by embracing the full range of protection, prevention and promotion strategies. Building on the Ottawa Charter for Health Promotion, population health focuses our attention on the underlying factors affecting the health status of the entire population. It also reinforces our concern for the health and well-being of particular population groups - for example, children, seniors or those who, because of their living or working conditions, are the most vulnerable within society.

In your view, what are some key considerations in promoting and improving health?

For one thing, we need to focus more on the community, rather than the individual, as the functional unit for our health promotion interventions. By community, I mean geographic neighbourhoods, as well as communities of people who share interests or problems. Communities are an immensely important source of support and contribute to the health and well-being of their members by providing a sense of social inclusion and belonging, promoting self-esteem, encouraging information sharing and facilitating action.

As well, we need to move beyond isolated interventions focusing on separate issues or diseases and move to integrated efforts to address the full range of factors and conditions that affect people's health.

We must also be aware of the important role of spirituality, language, culture and identity - and the respect for same - in improving health status. This is in keeping with the idea that we promote others' health where we create the possibilities for that to occur. Or, to quote Ibn Arabi, "to create for all, the conditions of their fulfillment."

What evidence is needed to achieve a more appropriate balance of population health investments along the continuum of health care to health promotion?

It's extremely important to produce evidence of the utility of health promotion, particularly in the current climate that demands public accountability and quantifiable results. In order to generate a solid evidence base, we need to ask the right questions and answer them well. We must also make wise decisions about what we're going to track and measure. For example, for health conditions that are preventable, we need to ask: What should we have done that would have made a difference? Measurements such as "quality-adjusted life years" and "health-adjusted life expectancy" are good examples of the way to go. They allow us to look at quality and duration of life and, where there is a deviation from the ideal, to ask why that is occurring. These measurements are also quantifiable over the short and long term. This is crucial for generating evidence and should be required for all protection, prevention, promotion and health care interventions.

As well, there are two components of the evidence base that are critical for health promotion efforts. The first is evidence that characterizes the linkages between the determinants of health and health status. This type of evidence provides us with information about where we should focus our interventions. But it says little about the effectiveness of interventions - the second crucial component of the evidence base. While we've been reasonably successful at generating evidence for the first component, we've had more difficulty with the evidence of effectiveness. That's one reason I'm pleased that the Health Policy Research Bulletin is focusing on this theme and exploring the challenges of developing a sound evidence base on the effectiveness of health promotion interventions.

Health Promotion What Is It?

This article traces the evolution of health promotion in Canada and suggests that it has been shaped, in part, by emerging "evidence of opportunity" for health promotion benefits. The authors acknowledge that the article is written from a Health Canada perspective and regret that, due to space constraints, parallel activities occurring at the provincial and municipal levels could not be included.

In the Early Years: A Lifestyle Focus

Canada's leadership in health promotion began in 1974 with the publication of A New Perspective on the Health of Canadians. Also known as the Lalonde Report, it marked the first time a major national government had acknowledged that health is primarily determined by factors outside the health care system. The report identified four fields of influence on health - human biology, lifestyle, the environment and health services. Of these, lifestyle became the initial focus as research revealed links between health status and personal risk behaviours. As a result, early interventions included health education programs and public awareness campaigns designed to influence individual health knowledge, attitudes and behaviours (e.g., to stop smoking, exercise more, make healthy food choices). In some cases, legislative action was taken to reinforce the desired behaviour change (e.g., tobacco legislation, "drinking while driving" laws).

Broadening the Focus: Socio-Ecological Models

By the mid-1980s, there was growing concern about the limitations of an approach focusing primarily on lifestyle. Increasingly, people were recognizing the constraints that structural conditions such as poverty and discrimination placed on individual efforts to achieve health. This, combined with mounting evidence about the health impacts of environmental factors, shifted attention to the influence of the environment, including the physical, social, cultural and economic aspects of one's surroundings. This new evidence of opportunity for health promotion interventions led to innovative initiatives such as the World Health Organization's (WHO) Healthy Cities and Healthy Communities, which originated in Canada ( Healthy Cities and urban governance). Starting with Toronto, municipalities across Canada developed broad-based projects aimed at improving residents' health and the environment.

Health promotion went global with the launch of WHO's Health for All Strategy. Another important milestone was the 1986 Canadian-hosted First International Conference on Health Promotion, which marked the release of two key documents - the Ottawa Charter for Health Promotion and Achieving Health for All: A Framework for Health Promotion. Both documents were pivotal in shifting the focus of health promotion to the broader determinants of health. The Ottawa Charter defined the prerequisites for health (i.e., peace, shelter, education, food, income, a stable ecosystem, sustainable resources, social justice and equity) and recognized the need for coordinated action across many sectors. It also identified five strategies for building a "new" health promotion practice: building healthy public policy; creating supportive environments; strengthening community action for health; developing personal health skills; and reorienting health services.

Strategic Investments, Comprehensive Programs

The five years following the release of the Ottawa Charter marked a period of significant activity in health promotion. A number of large-scale federal strategies were established, including Canada's Drug Strategy, the Tobacco Demand Reduction Strategy, the National AIDS Strategy and the Brighter Futures Initiative for Children. Collaborative programs, such as the Heart Health Initiative, fostered partnerships with provincial governments and the voluntary sector. Settings-based approaches - for example, Healthy Schools and Workplace Health - were also promoted. With the first national Health Promotion Survey and the creation of university-based Health Promotion Research Centres, research activities flourished.

Although most strategies targeted a particular health issue or population group, many included a comprehensive mix of interventions designed to promote change at the individual, community and policy level (see Table 1). Of particular note were community-level interventions such as the Community Action Program for Children (CAPC) and Canada's Prenatal Nutrition Program (CPNP). As illustrated on page 15, these programs pose some interesting evaluation challenges.

Some Key Interventions

Health communication informs the public about health concerns and keeps health issues on the public agenda. It also reinforces health messages and stimulates people to seek more information.

Social marketing campaigns use a variety of media to create a social climate conducive to health (e.g., see "Back to Sleep" on p.19).

Health education involves learning opportunities designed to influence health knowledge, attitudes and behaviours. In recent years, the focus has been on helping people improve decision-making and other life skills.

Social support recognizes the effects of social interaction on health. Activities often take place within communities and are undertaken by voluntary agencies (e.g., parenting support for young mothers).

Community action for health refers to efforts by communities to address local health priorities and increase control over the determinants of health (e.g., see CAPC and CPNP on page 14).

Creating supportive environments refers to activities aimed at establishing policies that support healthy physical, social and economic environments (e.g., WHO's Healthy Cities project).

Developing healthy public policies is concerned with establishing health-enhancing policies in sectors whose actions have health impacts. Such policies support healthy choices and promote the creation of healthy living and working conditions (e.g., the Child Tax Benefit).

Challenges and Setbacks

Health promotion faced a number of challenges during the 1990s, including setbacks in the campaign against tobacco as a result of tax reductions and the temporary repeal of legislation banning tobacco advertising. This was also a period of greater fiscal restraint, which constrained efforts to renew strategies and placed increased demands on all programs to justify their activities. Health promotion came under particular pressure to justify initiatives directed at the underlying prerequisites of health (for example, affordable housing and adequate income), as the policy instruments to influence these were largely controlled outside the health sector.

Expanding the Evidence Base

Just as health promotion was being increasingly challenged as to the appropriateness of its intersectoral policy-level interventions, population health research helped to counter these challenges by providing a compelling synthesis on the socioeconomic determinants of health. In 1994, the Federal/Provincial/ Territorial Advisory Committee on Population Health released Strategies for Population Health: Investing in the Health of Canadians, identifying nine major determinants of health: income and social status; social support; education; employment and working conditions; physical environment; biology and genetics; personal health practices; healthy child development; and health services. Health Canada added two more determinants - culture and gender.¹

A Time to Take Stock

Over the years, health promotion has evolved from an approach focusing on behavioural factors to one that addresses the underlying determinants of health - and their interaction - in a comprehensive manner. Standing the test of time, the strategies for action outlined in the Ottawa Charter are evident in most present-day health promotion programs. These include a broad mix of interventions aimed at empowering people, as individuals and groups, to make healthy choices and at creating environments that provide equitable access to the underlying determinants of health.As this article has suggested, the evolution of health promotion has been influenced by the evidence of opportunity for interventions, including research linking lifestyle, environment and the broader determinants of health to health status. While such evidence is necessary, it is not sufficient to justify future investments. Now is the time to take stock and assemble the evidence on the effectiveness of interventions. As the following articles demonstrate, much has been achieved, yet more remains to be done.

Note: Please see the Health Canada website at Health Canada for the complete text of all Health Canada documents referred to in this article.

Click here for references.

The Who European Working Group on Health Promotion Evaluation

Dr. Irving Rootman, Professor, Department of Public Health Sciences, University of Toronto Dr. Rootman was the Director of the University of Toronto's Centre for Health Promotion from 1990-2001 and is currently the Chair of the Canadian Consortium for Health Promotion Research. He was served as a Senior Scientist to the World Health Organization and was the 2001 recipient of the R.D. Defries Award - the highest award of the Canadian Public Health Association.

Brenda Steinmetz, Strategic Policy Directorate, Population and Public Health Branch, Health Canada

Evaluating health promotion initiatives presents some unique issues and challenges, ranging from debates about appropriate methodologies to concerns about whether evaluation approaches adhere to health promotion principles such as empowerment and equity. International efforts such as the World Health Organization (WHO) European Regional Office's Working Group on Health Promotion Evaluation were established to respond to increasing pressures to address these issues and, at the same time, demonstrate the effectiveness of health promotion initiatives (see also "Who's Doing What," p.28).

A Unique Approach

In 1995, Health Canada joined the newly-created 18-member Working Group on Health Promotion Evaluation, along with the U.S. Centers for Disease Control and Prevention and the United Kingdom's former Health Education Authority. Dr. Irving Rootman, one of the authors of this article, served as chairperson of the Working Group. Dr. Glenn Irwin of Health Canada and three other Canadians participated as members of the diverse team, which included representatives of governments, the WHO and the academic community.

The primary aim of the Working Group was to provide guidance to policy makers and practitioners on appropriate methods for health promotion evaluation. In doing so, members discussed and debated important methodological and philosophical issues in support of a critical approach to improving the evidence base for health promotion. Inevitably, the team's rich diversity of perspectives resulted in some differences in opinion about issues such as the value of using the randomized control trial (RCT) as the "gold standard" for evaluation in health promotion initiatives. Despite the range of opinions, however, members were able to reach a consensus on this and other questions, including: whether evidence is a relevant concept for health promotion; what constitutes sound evidence; and appropriate methodologies for assessing effectiveness in health promotion. The resources and tools developed by the Working Group reflect this process of debate, analysis and ultimate consensus.

Resources and Tools

In 1998, the Working Group summarized its work to date with the publication of Health Promotion Evaluation: Recommendations to Policy-Makers ( Health Promotion Evaluation: Recommendations To Policy-makers). The resource highlighted key actions that policy makers can take to support the appropriate evaluation of health promotion initiatives. These include:

• encouraging the adoption of participatory evaluation approaches
• using multiple methods that provide information on both process and outcomes
• developing infrastructure for funding, training, organizational development and networking
• allocating sufficient resources for evaluation

A more comprehensive document, Evaluation in Health Promotion: Principles and Perspectives, was released in 2001 ( Evaluation in health promotion - principles and perspectives). This collection of more than 20 papers examines the theory, methodologies and practice of evaluating health promotion initiatives, with a focus on the following areas: evaluation work in Western, industrialized societies; issues pertinent to evaluating settings, policies and systems, as well as individual behaviour change; and evaluation of health promotion efforts consistent with the Ottawa Charter for Health Promotion. It covers a wide range of both general and specific topics, highlights efforts to build the knowledge base through improving methods for evaluating community, school, workplace, mass media and policy interventions, and identifies some general lessons about the evaluation of health promotion (as shown below).

Evaluation in Health Promotion

• is an evolving field
• can make a major contribution to practice
• suffers from a shortage of evidence on the effectiveness of initiatives
• involves a wide range of approaches and models
• offers legitimate roles for both quantitative and qualitative methodologies
• employs a wide range of social science disciplines and approaches
• builds on a range of planning models
• requires theory and other conceptualizations to be effective
• offers many potential roles for evaluators/researchers

The document also outlines specific actions that need to be undertaken to implement the Recommendations to Policy-Makers and profiles several tools to aid in doing so, including:

• a generic logic model for planning and evaluating health promotion interventions
• a set of guidelines for conducting participatory evaluations

Next Steps

Although the Working Group did not set out to conduct a systematic review of evidence on the effectiveness of health promotion, it identified a number of areas requiring additional work. In particular, the Group concluded that considerable improvements must be made to the quality of the evidence base on policy interventions for promoting health, a process that should involve political scientists and policy analysts. Professionals in management sciences can also make important contributions about the effect and operation of systems, an area where knowledge is particularly lacking. Related discussions at the Second International Symposium on the Effectiveness of Health Promotion (Toronto, 2001; proceedings available at Centre For Health Promotion) also highlighted the need for:

• stronger links between researchers and practitioners
• greater investment in developing the capacities of communities to conduct research and evaluation and to use results
• more evidence on the effectiveness of organizational, community and policy development and the combined use of these strategies

As the WHO Working Group demonstrated, it is possible for national governments, international organizations, academics, policy makers and practitioners to collaborate in a constructive, cost-effective and meaningful way to develop the evidence base for health promotion. Another excellent example of this type of collaboration is the work of the International Union for Health Promotion and Education (IUHPE) on evidence of effectiveness. The WHO and IUHPE have now jointly initiated a global project on the effectiveness of health promotion, while a complementary project on the effectiveness of healthy municipality efforts in Latin America is being undertaken by the Pan American Health Organization (PAHO). In addition, the Canadian Consortium for Health Promotion Research is reviewing frameworks for the consolidation of evidence on the effectiveness of health promotion (see page 30 ). Building on the momentum, experience and contributions of the Working Group, these and other initiatives are concrete next steps for enhancing the scientific foundation for health promotion to better meet the needs of policy makers and practitioners in population and public health.

Issues and Challenges in Assessing the Effectiveness of Health Promotion

Ron Wall, Applied Research and Analysis Directorate of the Information, Analysis and Connectivity Branch, Health Canada. Special thanks to Tracey Spack, Strategic Policy Division, First Nations and Inuit Health Branch, for contributing the section on qualitative methods.

There are a number of important issues and challenges facing both those who use health promotion evidence in the development of policy and those involved in advancing the evidence base. This article addresses some of these challenges, including:

• While improved health is the ultimate aim of health promotion, what intermediate outcomes can be expected?
• What constitutes "evidence of effectiveness" in health promotion?
• What methods can be used to assess the effectiveness and cost-effectiveness of health promotion interventions?
• How can the evidence base be strengthened to make better investment decisions in the future?

In the Meantime: Intermediate Outcomes

Health promotion is coming of age as a theory-informed, evidence-based and broadly accountable practice.¹ At the same time, however, it presents a number of unique challenges for researchers and policy makers alike. Health promotion is extremely complex, encompassing a wide range of multiple social and behavioural interventions targeting populations not just as aggregates of individuals but also as families and entire communities. Moreover, the outcomes of these interventions - the importance of which will vary depending on the stakeholder's perspective - may take years or even decades to become evident.

Table 1 presents a possible hierarchy of health promotion outcomes. There is general agreement that positive health - as defined by the World Health Organization in terms of physical, mental, and social functioning as a resource for everyday living, rather than just the absence of disease - should be the ultimate goal of health promotion efforts. However, intermediate outcomes can demonstrate more quickly, albeit often with less authority, whether any changes are taking place or not.

It should be noted that, while analytical models integrating available evidence and expert opinion can provide important insights until more rigorous information becomes available, the strength of the link or logic between the intermediate indicators and the ultimate health and social outcomes is crucial. If these links are not well established, the analytical challenge lies in filling in the knowledge gaps. Users should appreciate, however, that even apparently rigorous information may be of limited value, since models are only as good as their input data and underlying assumptions.

Adapted from: Nutbeam D. Health Promotion Effectiveness - The Questions to be Answered. In: The Evidence of Health Promotion Effectiveness - Shaping Public Health in a New Europe. International Union for Health Promotion and Education, 1999, p. 6.

Evidence of Effectiveness

While opinions differ about the nature of evidence and the methods used to generate it, there is growing recognition that both process and outcome evaluations are necessary in demonstrating the effectiveness of health promotion interventions.

Process evaluations are concerned with how interventions are organized, delivered and used - that is, evidence of "best practice." Users of the evidence base need to understand the nature of the intervention, the characteristics of the client population and the ambient social/health/political environment. They also need to be convinced that the intervention did what it was expected to do. For example, did it reach the target population? Did activities occur as planned? Were the theory and assumptions underlying the intervention design verified?

Outcome evaluations are concerned with the results of the intervention. They can encompass various levels of complexity, from assessing how participants in the intervention are faring, to whether they are doing better than a "control" group, or if the intervention actually caused the outcome.² A range of outcomes and evaluation designs can be used to answer these questions.

Assessing Effectiveness and Cost-Effectiveness

Interventions can be assessed both in terms of their effectiveness and cost-effectiveness. Effectiveness is concerned with the outcomes achieved by interventions, while cost-effectiveness compares the outcomes achieved to the cost of the inputs used. The evidence of effectiveness is the first consideration. However, when assembling a portfolio of interventions, policy makers should select those that are also cost-effective - that is, those with the potential to improve the health of the target population and also yield "value for money." The criteria for assessing the quality of the evidence base of health promotion effectiveness is the strength of the attribution of outcomes to intervention (what is called "internal validity"), and the extent that these findings can be generalized beyond the setting of the specific evaluation (what is called "external validity"). Some of the issues associated with assessing effectiveness and cost-effectiveness are addressed briefly below.

Were the outcomes the result of the policy or program intervention?

The attribution of observed outcomes to a policy or program intervention is key for demonstrating effectiveness. While a number of evaluation strategies exist, the simplest strategy is to identify a target population and compare measurements made before the intervention to those made afterwards. The absence of a reference population is problematic, however, if other trends explain at least some of the pre- and post-differences in outcome. For this reason, designs comparing intervention and "control" groups have been widely used to evaluate both individual- and population-based health promotion. When exposure to the intervention is not random, however, it is possible that control and intervention groups are not equally affected by other determinants of outcome.

The "gold standard" design for attributing observed outcomes to planned interventions is the randomized control trial (RCT). Given a sufficiently large sample size and efforts to ensure a comparable distribution of key subgroups (e.g., stratification by age, gender), randomization ensures that the intervention and control groups will be comparable. However, as RCTs are expensive, they should only be used after sufficient (but less valid) evidence of effectiveness has been accumulated and there is an understanding of what constitutes "best practice."

Can the findings be generalized?

Users of the health promotion evidence base should also consider the extent to which sample findings can be generalized to a broader population. For example, outcomes based on samples of higher-risk patients or highly motivated volunteers may not generalize to a population that includes lower-risk patients and less motivated volunteers. This is particularly problematic in obtaining evidence for socially disadvantaged populations. Randomization based on well-defined communities rather than individuals can, to some extent, overcome this problem.³

Types of Economic Evaluations

The way in which non-monetary health outcomes are measured will determine the type of economic evaluation performed. For example:

1. Cost-effectiveness analysis compares interventions affecting the same determinant of health or health outcome using a ratio of the difference in cost to the incremental health effect.
2. Cost-utility analysis compares interventions generating multi-dimensional health effects by establishing the ration of incremental cost to incremental Quality Adjusted Life Years gained.
3. Cost-benefit analysis accounts for multiple health effects by assigning monetary units to loss of life and quality of life and determining whether the health and resource consequences of an intervention are worth its costs.

Does the intervention offer "value for money"?

Economic evaluation extends the evidence base by assessing the cost of the inputs used to achieve the outcome - that is, "value for money." Such information informs decision-making choices among competing uses of scarce resources. Estimating the cost of health promotion involves identifying the resources affected by the intervention, measuring the quantities used and assigning unit costs. Although conceptually straightforward, the practice of economic evaluation is complicated by several factors including methodological issues, the limited availability of financial and statistical information, and problems in attributing costs to specific interventions by organizations engaged in multiple activities.

The other side of the cost-effectiveness equation is outcomes. Health promotion interventions potentially affect future health outcomes and the use of health care and other resources. As above, estimating future costs avoided involves identifying the resources affected and the physical resources used, and assigning unit costs. The value placed on reduced morbidity or mortality must be appraised as well - a complex endeavour. Moreover, as costs and consequences typically occur in different time periods, future dollars and health outcomes must be adjusted to the start of analysis (i.e., "discounted to their present value") using an appropriate interest rate (i.e., "discount rate"). (See sidebar for an overview of economic evaluations.)

Strengthening the Evidence Base

A number of strategies are available to help strengthen the evidence base of health promotion. These include: using complementary approaches to research; combining the results from individual studies; and focusing on the social context of behaviours.

Mutually supportive: qualitative and quantitative methods

While qualitative and quantitative approaches are frequently presented as opposite ends of the methodological spectrum, there is a growing recognition that these methods are, in fact, complementary.

In conducting evaluations of health promotion interventions, qualitative and quantitative research can be combined in at least three ways:⁴,⁵

• Qualitative techniques (e.g., observation, in-depth interviews, focus groups) can be used as a preliminary phase to help specify the intervention and channels of causation for quantitative assessment.
• Qualitative methods can supplement quantitative work as part of the validation process, as in triangulation, in which the results obtained from three or more methods (e.g., a large-scale survey, focus groups and a period of observation) are compared for convergence.
• Qualitative methods can explore complex phenomena or areas not yet amenable to quantitative research, such as aspects of complex behaviours, attitudes and interactions. Information such as this can be used to generate hypotheses and develop indicators for quantitative evaluation.

Combining evidence from individual studies

Policies aimed at improving population health should be based on the best available evidence obtained from a systematic review of individual studies. These reviews should: focus on a well-defined objective, intervention and comparison group; be based on an exhaustive search to locate all published/non-published studies; define how studies are entered into the evidence base; use several reviewers and consistent criteria to assess the quality of each study; and use appropriate methods to synthesize the study findings. Continuing efforts on a number of fronts are advancing our understanding of the effectiveness and the cost-effectiveness of health promotion, but much work remains to be done (see "Who's Doing What," page 28).

Evaluating the social context

While a considerable evidence base exists for health promotion interventions targeting specific health behaviours of high-risk (or highly motivated) individuals and communities, less is known about the role interventions play in addressing the social context of such behaviours. An increasing body of evidence on health determinants argues that social conditions and structures affect the development of health behaviours as well as the ability to intervene. Recent developments in theory, analytical frameworks and statistical methods are advancing our knowledge of how to intervene to improve population health.³,⁶ Evaluations (including RCTs) are also providing evidence of the effectiveness of specific interventions on the social context of health behaviours.³

In Conclusion

Issues such as the nature of evidence, the role of various evaluation methods, the set of outcome indicators and the language of evaluation will continue to challenge both those who use the health promotion evidence base and those who contribute to it. As the health policy agenda shifts to evidence-based practice, however, the future of health promotion depends on whether it can deliver concrete evidence of its effectiveness, particularly cost-effectiveness. Clearly, there is work to be done if Canadians are to realize the "upstream" results that health promotion promises.

Click here for references

A Tale of Two Evaluations: CAPC and CPNP

Evaluation is an integral component of federally-funded community-based health promotion interventions. Evaluation serves an important role as a tool for continuous program improvement, in assessing program impact and in fulfilling Treasury Board mandates. Evidence generated from evaluations serves to answer questions on the outcome side of public policy decisions aimed at improving health and other determinants of health through a variety of interventions. The following case study examines the pioneering national evaluation experiences of two Health Canada programs - the Community Action Program for Children (CAPC) and the Canada Prenatal Nutrition Program (CPNP).

CAPC and CPNP: Similar, But Different

As national community-based health promotion interventions, both CAPC and CPNP support community groups in establishing and delivering services that address the needs of particular at-risk groups. CAPC is designed to improve the health and development of at-risk children and their families, including children living in low-income families and children experiencing developmental delays or social, emotional or behavioural problems. CPNP, on the other hand, addresses the needs of pregnant women most at risk for poor birth outcomes, including women living in poverty, teens and women who use alcohol, tobacco or harmful substances.

Although the two programs perform similar functions, their differences reveal the evolution of Health Canada's approach to delivering and evaluating community-based programs. Created in 1993, CAPC's projects include a wide range of services and activities, reflecting the diverse needs identified in communities across Canada. The projects are united by a common set of Guiding Principles and a national evaluation strategy. The strategy was based on current literature on child development and included the National Longitudinal Survey of Children and Youth (NLSCY) as a comparison group.

Announced in 1994, CPNP reflects lessons learned from the CAPC experiences, as well as existing prenatal programs, such as the Montréal Diet Dispensary. In addition to the existing CAPC Guiding Principles, Health Canada added a set of clearly defined "elements of a comprehensive program," common evaluation objectives and indicators. As a result, CPNP projects share more similarities than those funded under the CAPC program. While CPNP's commitment to maintaining flexibility encourages subtle variation in individual delivery models, each project can be evaluated according to a common set of objectives and outcome indicators.

Evaluation Methodologies: Lessons Learned

The first step in conducting a program evaluation is to develop an evaluation framework. It provides a systematic method for describing all the projects in the program as a whole.

CAPC

The CAPC National Evaluation Framework was designed to reflect the program's guiding principles and respond to questions about program development and implementation (process evaluation), and the benefits of participating in CAPC projects (impact evaluation).¹ As set out in the Framework, process data are collected from project managers, while impact data are collected from project managers and through interviews with a sample of project participants at three different times following enrollment: at baseline (Cycle 1); after nine months (Cycle 2); and after 24 months (Cycle 3). Families and children living in similar economic circumstances from the 1994 and 1996 cycles of the NLSCY were identified as a comparison group for the evaluation.

CPNP

The CPNP national evaluation framework ² was designed in accordance with the program's guiding principles and benefited from the experience of the CAPC national evaluation. Lessons learned included the restrictive nature of NLSCY indicators (which were limited to individual health outcome indicators) and the importance of a participatory approach to evaluation. As a result, CPNP designed a "baseline" study and adopted a participatory, iterative approach that included all stakeholders at the local, regional and national levels in developing evaluation indicators and survey instruments.

CPNP's framework consists of two evaluation tools: the Individual Project Questionnaire (IPQ), which consists of 28 questions on administrative topics and broad program outcomes and is administered annually; and the Individual Client Questionnaire (ICQ), which includes a menu of program indicators (106). ICQ indicators are participant-specific and address issue areas such as risk profile, use of services and pregnancy outcomes - at project entry, pre- and post-delivery, and project exit. All projects collect core indicators (38 items identified by Health Canada as relating to federal objectives) for the national level of the evaluation. Local projects and Joint Management Committees (JMCs) are free to choose other indicators of particular relevance to their specific project objectives from the menu.³

What the Evaluation Showed

The CAPC and CPNP evaluation instruments collect two types of information: process information about program development and implementation, and broad program outcomes; and impact information related to both the characteristics of individual participants (target population) and individual outcome indicators (program impact).

Process: capacity building

In addition to engaging more than 100,000 children and parents per month in more than 464 projects across the country in 2000-2001, CAPC projects have been successful in developing and maintaining partnerships, leveraging support - including financial (3-1 ratio), in-kind ($2.7 million) and volunteer (61,762 hours donated by more than 8,000 volunteers) - and as a national platform for developing and delivering comprehensive community-based health promotion projects. Similarly, CPNP evaluation results indicate that more than 28,000 women participated in CPNP projects in 1999-2000. Moreover, 628 new program activities or services were created as a direct result of CPNP projects, including parent support groups, clothing/baby equipment banks, community kitchens and breastfeeding support groups.

Program reach

One of the most important questions related to program impact is whether the program is reaching its target population(s). Evaluation results showed that CAPC-funded projects were serving families experiencing much higher levels of socioeconomic disadvantage than the nationally representative families from the NLSCY (see Figure 1).⁴ For comparable levels of socioeconomic disadvantage, CAPC families also had higher levels of psychosocial risk (as measured by family dysfunction, maternal depression and negative caregiving) than those in the NLSCY.⁴ Moreover, CAPC-funded programs retained a majority of these at-risk families: "66 percent either completed the CAPC program or were still attending programs; about 15 percent were not attending (and did not complete) CAPC programs; and about 19 percent were lost to follow-up."⁴

Figure 1: CAPC and NLSCY Families: Differences in Levels of Socioeconomic Disadvantage

CPNP-funded projects were also successful in reaching pregnant women living in disadvantaged conditions, who were least likely to participate in, or have access to, prenatal support.⁵ More specifically:

• 58 percent of CPNP participants lived on household incomes of less than $1,000 per month
• 35 percent were teenagers, including 9 percent who were 16 years old or younger
• 56 percent had less than 12 years of education (22 percent had not completed grade 10)
• 47 percent were single, divorced, separated or widowed
• 22 percent were Aboriginal
• 46 percent smoked during their pregnancy
• 14 percent reported experiencing abuse during their current pregnancy

Program outcomes

Related to program targeting is whether there is evidence of beneficial program participation. Evidence from the CAPC national evaluation was inconclusive when comparisons were restricted to the individual health outcome indicators available from the NLSCY. CAPC-funded programs were found to engage at-risk groups, and a majority of respondents indicated that CAPC programs were either "very helpful" (52.4 percent) or "somewhat helpful" (39.9 percent) at 24 months. Preliminary findings at nine months on individual health outcome indicators found downward trends in the levels of maternal depression, negative caregiving, and child emotional and behavioural problems among CAPC participants, which were greater than those among the NLSCY comparison group.² However, these preliminary findings of positive program benefits could not be confirmed at the 24-month follow-up. At that point, "the before-after differences were extremely small; in some instances, they favoured CAPC participants and in other cases they favoured comparison families in the NLSCY."⁶

Results from the CPNP national impact evaluation indicate that CPNP-funded projects appear to be having a positive impact on the target population based on two key indicators: breastfeeding initiation rate and low birthweight rate (LBW).

The overall breastfeeding initiation rate for women in CPNP projects was 78 percent, virtually the same as the national rate of 79 percent.⁷ It is worth noting, however, that although not directly comparable to national surveys, the CPNP at-risk participants appear to initiate breastfeeding at rates higher than those usually reported for similar at-risk subgroups in the National Population Health Survey (NPHS) (see Tables 1 and 2). While it is not possible to definitively describe breastfeeding duration rates as CPNP participants left the program at varying times during the postnatal period, more than one third (35.5 percent) breastfed their baby for at least one month and almost as many (31.4 percent) continued to breastfeed for at least 3.5 months.⁷

Figure 2: Level of CPNP Service and Birthweight

The CPNP low birthweight rate for singleton births was 6.1 percent (7.5 percent when multiple births were included).⁷ This finding is promising for women living in the conditions of risk targeted by CPNP when compared to the low birthweight rate of 5.8 percent for the general Canadian population.⁷ A definitive determination of CPNP's impact on birthweight awaits completion of a baseline comparison study, application and validation of a CPNP service index and the successful completion of other components of the national program evaluation. In the meantime, however, participants who entered late in their pregnancy or received a lower level of CPNP services have been used as a comparison group to assess program impact.

Figure 2 ⁸ summarizes observations on LBW rates according to a composite index that combines service mix and duration of program participation. Drawn from cumulative data on individual project completers between 1995 and 2001, the analysis reveals:

• Overall, LBW rates appear to improve as participants are exposed to higher levels of CPNP services.
• LBW rates among CPNP participants grouped by risk categories for LBW pregnancies appear consistent with the overall findings, with the exception of the high-risk category which requires further study.

Looking Ahead

While evidence from the pioneering national evaluations of CAPC and CPNP is insufficient to conclusively determine program effectiveness, process and impact data point to some promising outcomes. Efforts to further refine evaluation methodologies and instruments, including cost-effectiveness analysis, are being pursued in order to provide decision makers and stakeholders with reliable information about all aspects of program effectiveness (see sidebar). Evaluating national community-based health promotion interventions pose many methodological challenges as such interventions operate in complex social environments where many interrelated factors affect the links between intervention and measurable outcomes. Nonetheless, the evidence generated from such comprehensive evaluations can provide a useful picture of the overall program and its component projects which can guide continuous program improvement, as well as provide a basis for assessing program impact.

CPNP: A Collaborative Evaluation

Health Canada's Departmental Program Evaluation Division is currently collaborating with various branches to conduct a Department-wide evaluation of the CPNP. This evaluation is consistent with the 1994 Treasury Board Submission and the 1996 CPNP evaluation framework and will build on the work presented in this article. The report is expected to be completed by March 31, 2003.

Click here for references

Partnering for Social Change: The "Back to Sleep" Campaign

Rosemary Sloan, Population and Public Health Branch, Health Canada, and Shelley Cotroneo, Health Policy and Communications Branch, Health Canada

Social marketing has been used in conjunction with other interventions to change health behaviours and, ultimately, to improve health status. This article describes a social marketing campaign designed to reduce the incidence of Sudden Infant Death Syndrome (SIDS) in Canada. In addition to presenting the study's key findings, it highlights some "real world" evaluation challenges and lessons learned.

Background

Each week in Canada, an average of three babies die of SIDS, the unexpected and unexplained death of an apparently healthy infant. Once commonly known as "crib death," SIDS is the leading cause of post-neonatal death in the country. While its specific causes are not known, research has identified a number of risk factors, including sex (higher among males) and age (highest among two- to four-month-old babies). Modifiable risk factors include infant overheating, maternal smoking during the pre- and postnatal periods and, most importantly, the infant's sleeping position (infants who sleep on their back are at the lowest risk of SIDS). Research also supports the protective influence of breastfeeding.

Getting Started

Responding to SIDS as a significant public health concern, Health Canada, the Canadian Foundation for the Study of Infant Deaths, the Canadian Institute of Child Health and the Canadian Paediatric Society co-sponsored a national workshop on reducing the risk of SIDS in 1993. One of the outcomes of the workshop was a national social marketing campaign. Based on the scientific evidence available at the time - which included identification of the back and side as safe positions for sleeping - the campaign targeted health professionals and parents of newborn infants with a series of promotional pieces designed to raise awareness and change behaviour related to infant care.

The "Back to Sleep" Campaign

Within a few years, further research on SIDS demonstrated that the safest sleeping position for infants was on their back, not on their "back or side" as previously recommended. Based on this new information, in 1999 Health Canada and its partners launched a re-tooled version of their earlier campaign. The primary objective of the new campaign was to increase awareness about SIDS and to provide the "gatekeepers" to infant health and well-being - including parents, caregivers and health professionals - with information about the risks associated with SIDS. The goal was a 10 percent reduction in the incidence of SIDS over the next five years.

Building on the "Back to Sleep" slogan developed in the United States, the new campaign relied extensively on multiple media to raise awareness about SIDS in Canada. The campaign was based on a thorough analysis of the situation and a careful development and testing of the marketing tools. Focus groups were conducted with members of the target audience, including parents of children under one year of age, those planning to become parents in the next year and professional caregivers of infants.

Results showed that the target audience wanted persuasive and credible information (e.g., statistics that underscored the prevalence of SIDS) and acknowledgment that the campaign's key messages were evidence-based. Parents also wanted a hard-hitting message that captured their attention. These findings were incorporated into the final products, which included an information brochure and poster, a promotional ad for new parents' magazines and a 30-second public service announcement.

Health Canada and its partners also developed a joint statement providing health professionals with clear, consistent messages about how to reduce the risk of SIDS. To extend its reach, the campaign targeted its partners' networks and enlisted the help of Procter & Gamble, whose "Pampers" division markets over 300 products to more than five billion consumers in 140 countries. The company incorporated the "Back to Sleep" message onto its diaper waistbands, created a door hanger promoting the "Back to Sleep" message, distributed the "Back to Sleep" pamphlet to mothers of newborns through hospitals across Canada and promoted SIDS awareness through its advertising campaigns.

The Evaluation Process

Although the initial 1993 campaign did not include an evaluation component, data showed that the rate of infant deaths attributed to SIDS declined during the period of the campaign. For this reason, Health Canada and its partners were optimistic about the revised "Back to Sleep" campaign in 1999. In order to assess the impact of the new campaign, a market research component was developed. As a starting point, a baseline survey was conducted of new and expectant parents and caregivers to establish levels of awareness and knowledge, and behaviours relating to SIDS prior to the campaign launch.¹ While this survey established that awareness of SIDS was generally high, a significant proportion of the target group was not aware of the new findings about infant sleeping position.

In the spring of 2001, Health Canada conducted a post-campaign tracking survey, replicating the 1999 benchmark survey to test awareness, attitude and behaviour shifts since the earlier survey and to suggest any necessary modifications to the campaign.² A total of 605 respondents were surveyed, including representatives of the three target groups - parents of children under one year of age, those planning to become parents in the next year and professional caregivers of infants. Sampling was conducted among the general population as a whole, as well as among households with respondents between 18 and 40 years of age. Within households, respondents were screened according to the criteria established for the target group. The sample was composed of 87 percent women and included respondents from all regions of the country. The findings from the post-campaign tracking survey were then compared to the pre-campaign benchmark survey.

The Results

The results showed a substantial increase in knowledge and awareness levels about the importance of placing babies to sleep on their back from the first to the second survey. Moreover, there was a significant increase in parents' behaviours related to infant sleeping position. Among the study's key findings were:

• awareness of SIDS as an important cause of infant death had increased since 1999 (from 94 percent to 97 percent)
• o awareness that the back sleeping position reduces the risk of SIDS had increased substantially since 1999 (from 44 percent to 66 percent)
• behaviours had changed considerably over the two-year period; about 70 percent of parents and caregivers - an increase of about 30 points - said they placed infants on their back to sleep
• advice from health professionals had changed substantially - 67 percent advised a back sleeping position in 2001, up from only 21 percent in 1999

Issues and Challenges

The "Back to Sleep" campaign highlights a number of important issues relevant to the evaluation of health promotion interventions. These include:

Length of time

A key issue in assessing the effectiveness of health promotion interventions is the length of time required to determine whether changes have occurred in the intended health outcomes. This was less of a problem in the "Back to Sleep" campaign as SIDS is an absolute event that usually occurs within the first year of life. As the tracking survey showed a substantial increase in the number of parents placing their infants on their back to sleep, one would expect a corresponding decline in the number of SIDS deaths during this period. However, as a two-year wait is generally required for mortality data to become available and be analyzed, it is too early to determine whether the campaign had an impact on the rate of infant deaths due to SIDS.

Attribution

While the results showed a sizable increase in the number of parents placing their infants on their back to sleep, one cannot say, for sure, that these positive program outcomes were caused by the campaign. As pointed out in the article entitled "Correlation and Causation Demystified" (see page 31), correlation is not evidence of causation. Thus, without controlling for other, potentially influential factors during the 1999-2001 period, these changes in knowledge, awareness and behaviour cannot be definitively attributed to the "Back to Sleep" campaign. However, questions included in the post-campaign survey regarding changes in respondents' awareness and behaviours over the period of the campaign help to strengthen the evidence base.

When is there enough evidence?

This is a frequently asked question in assessing health promotion effectiveness. In an ideal situation, one might prefer to reserve the label "effective" for interventions in which "cause and effect" can be demonstrated. However, since randomized controlled trials are often neither feasible nor appropriate in assessing health promotion effectiveness, assigning attribution can be problematic. Therefore, in most real-life situations, judgments of effectiveness must be made - not on evidence of causation - but on demonstration of a correlation between the intervention and the outcomes, and on the strength of that correlation. Such is the case with the "Back to Sleep" campaign, at least until the SIDS mortality data become available.

Figure 1: Awareness of Ways to Reduce Risk of SIDS

Figure 2: Personal Actions to Reduce Risk of SIDS

Click here for references

Gender Matters: Evaluating the Effectiveness of Health Promotion

Sari Tudiver, Women's Health Bureau, Health Policy and Communications Branch, Health Canada

Women and men differ in health status and patterns of disease and in their use of the health care system. Understanding why these differences occur (i.e., how gender is a determinant of health) and how to improve health outcomes requires a range of research and evaluation. This may include evidence on sex (biological) differences which offer insights into possible chemical, genetic and metabolic differences in disease susceptibility and treatment approaches. Understanding why health promotion interventions are or are not effective also requires consideration of gender differences - the socially constructed roles and relationships, attitudes, meanings and relative power ascribed to men and women in society (Women's Health Bureau website at Bureau of Women's Health and Gender Analysis).¹

Integrating Gender into Evaluation

Gender-based analysis (GBA) is an analytic tool applied to research, policies and programs to ensure that appropriate questions are asked about both men and women so that the analysis yields sensitive and accurate results. It is not sufficient to add sex and/or gender as independent variables to the analysis, since exploring these differences often challenges the assumptions underlying health promotion programs and the interpretations of behaviour. Applying a gender lens can identify other variables appropriate for analysis and the paths by which they operate. Evaluation outcomes may also be different for women and men.

GBA should be an iterative, creative and systematic process, applied at all stages of an evaluation. A gender-sensitive evaluation asks:

Are the differing contexts of women's and men's lives addressed in this intervention?

Gender differences help to explain why women and men do or do not engage in health-promoting activities. For example, many women cannot participate in recreational exercise due to social barriers such as lack of child care or transportation. Men also encounter barriers to program participation.²Thus, evaluations of the effectiveness of such programs must assess whether gender-specific obstacles were considered and addressed.

Does the intervention identify and analyze the diversity within sub-groups?

When appropriate, a gender-sensitive evaluation analyzes the diversity within and between the populations of women and men, in terms of age, culture and ethnicity, abilities/disabilities, sexual orientation and socioeconomic status. This allows for a better assessment of how health promotion interventions might have different relevance to women and men from varied circumstances.³ Analyzing diversity also contributes to greater clarity about the generalizability of findings.

Does the intervention engage women and men in meaningful ways?

Positive health promotion outcomes are most likely when people are actively engaged in participatory research and as partners in designing interventions appropriate to their needs.⁴ A gender-sensitive evaluation assesses how women and men are involved in program design and follow-up, and tries to identify those whom the program failed to reach. If evaluations are to be developed through the involvement of "hard to reach" groups, it is important that the necessary resources are provided. Participation can place particular burdens on women and their community organizations, many of which have limited resources.⁵

Does the intervention reveal unintended outcomes?

Evaluation methods must be flexible enough to distinguish outcomes not previously anticipated. For example, quit rates for a women-specific smoking cessation program did not differ significantly from those in other types of programs, but participants identified other benefits such as increased self-care, a sense of empowerment, self-awareness and new social support networks.⁶

GBA: A Mental Health Example

Research has shown that the diagnoses of borderline personality disorder (BPD) and dissassociative identity disorder (DID) are more often given to women, and that these are associated with childhood sexual abuse and trauma. Research suggests that this population has difficulty accessing services and, as a result, repeatedly use emergency services. One Canadian study found that women with DID often go undiagnosed for approximately eight and a quarter years. An estimated savings of $84,899.44 per person could be achieved if earlier and accurate diagnoses were to occur.

A gender-based analysis corrects for this problem by: including sex-disaggregated data; integrating an understanding of gender in diagnoses; and identifying service provision sites more often used by women and vulnerable groups. Such analysis reveals more clearly how the system functions differently (or the same) for men and women, leading to better service delivery.

Adapted from: Pederson A, Hankivsky O, Morrow M, Greaves L. Gender-based Analysis: Using a Better Evidence Base for Research, Policy Making and Program Development in Health, Vol. 1. Women's Health Bureau, 2002 (forthcoming).

Where does the intervention fit in the context of other social, political and economic realities?

Gender-sensitive evaluation offers a method to analyze the interplay among interventions and social forces to assess whether different approaches may reinforce or undermine what are thought to be positive outcomes. For example, since women traditionally assume family caregiving roles, the need for policy change is often ignored.⁷ Evaluation of a skills training program for women providing home care for relatives may show short-term benefits to the caregivers in alleviating personal stress, but fail to address the underlying need for institutional and policy change (e.g., respite care). The evaluation process should be able to identify how multiple interventions, or the lack of intervention, affect outcomes differently for men and women.

Gender Does Matter

Gender-sensitive evaluation contributes to a more rigorous body of evidence about the outcomes, effectiveness and relevance of health promotion policies and programs. There is a substantial body of conceptual tools for gender analysis, as witnessed by major initiatives in "gender mainstreaming" in Europe and North America.⁸ As the Ottawa Charter envisioned, gender does matter in achieving equity and enabling people "to take control over those things which determine their health."

Thanks for their input to Women's Health Bureau colleagues: Lynne Dee Sproule, Virginia Adamson, Cathy Mattern and Susannah Bush; and to Dr. Wilfreda Thurston, University of Calgary.

Creating the Right Mix

Phyllis Colvin, Policy Division, Health Policy and Communications Branch, Health Canada

Health promotion strategies are comprehensive "baskets" or "portfolios" of different types of interventions. As a result, it is not just the effectiveness of individual interventions that is important, but knowing how they can be combined to create the most effective "mix". The health literature provides some fine analyses of the effectiveness of individual health promotion interventions, and of the comparative effectiveness of interventions based on analytical measures, such as cost per year of life saved.¹ While this type of analysis is very valuable, it is not sufficient to underpin complex national health promotion strategies.

There is increasing recognition that, to be effective, policy models must employ complex, non-linear and dynamic approaches to encompass the full range of interventions available, as well as the potential interactions among these interventions. The automobile provides a useful analogy. In order for a car to function, its various parts (e.g., fuel source, chassis, engine, battery, carburetor) must all be present and work in synergy. Extrapolating to health promotion, what is needed is the right mix of parts - or interventions - to produce the best functioning, most cost-effective "health promotion car" possible.²

Integration in Policy Development

The challenge of taking an integrated approach to policy development is faced by Health Canada as a whole. Like health promotion, the Department encompasses a variety of activities and programs, and it is developing an integrated approach to policy development and priority setting so that it can determine the most effective mix of interventions across the board. This challenge is being addressed using some fairly straight-forward management tools described in business literature on conglomerates.³ These tools include: continuous corporate-level discussion about policy and priorities; ongoing maintenance of a "portfolio" of proposals from across the institution; proposal integration and reinforcement of integration through the policy/planning/budgetary process. A key to this type of exercise is management agreement on policy criteria, such as those now in use in the Health Canada policy community. (see sidebar).

Health Canada's Policy Evaluation Criteria

• Potential for improved health outcome.
• Potential for reduction in health inequalities.
• Established government or ministerial priorities.
• Established federal-provincial priority.
• Optimal choice of instruments: potential for improved health outcome relative to other interventions.
• Appropriate exercise of federal/Health Canada role.
• Appropriate involvement of partners, including prior consultation.
• Adequate human and other resources in place to ensure relevant health outcomes.
• Is there an evidence base that supports the specific approach/measures being proposed?
• How does the proposal compare with other jurisdictions (e.g., WHO, OECD)?
• Potential for improved health outcome for the health issue being addressed relative to other health issues and activities.
• Degree to which an essential element in fulfilling elements of other priorities.
• Value in retaining capacity to participate constructively in related opportunities or necessary collaborations.
• Potential for "bridging" to future opportunities.

The main justification for this type of process investment is that organizations with "baskets of activities" are capable of absorbing risk more effectively than those without them.⁴

Templates as Policy Development Tools

Health Canada is currently developing new mechanisms to help make complex policy decisions involving multiple components. This promises to provide useful insights for health promotion as it also takes on the challenge of assembling effective intervention mixes. A useful starting point is the aggregation into a "template" of interventions with clear and well-understood linkages. This is demonstrated in a recent policy exercise relevant to early childhood development, which was undertaken to develop a comprehensive "basket" of policy interventions in response to the increase in mothers' age at first birth - a trend that has been observed in Canada since 1976.⁵ An important first step involved the identification of all the possible policy areas that could be affected by this trend - abortion, assisted human reproduction, prenatal care, parenting, family income and debt and career plans, to name just a few. From this list, a comprehensive template was constructed that not only accounted for all policy areas but also made explicit the complex web of interrelationships, linkages and synergies among them. This template was then used to develop a comprehensive policy "mix" by selecting options from each of the policy areas that could be assembled with the best possible "fit".

In Conclusion

Multiple interventions are often employed in health promotion strategies as well as other complex policy initiatives. As each intervention has the potential to affect the outcomes of other interventions, decisions makers need to take such linkages and synergies into account when selecting the most effective "basket" of interventions.

Click here for references

Evidence of Effectiveness: Assessing the Gaps and Moving Forward

Jim Ball, Director, Population Health and Health Promotion Development Division, Population and Public Health Branch, Health Canada, and Guest Editor for this issue of the Health Policy Research Bulletin

This issue of the Health Policy Research Bulletin has addressed the theme of effectiveness of health promotion - an "upstream" approach to improving population health and addressing inequalities in health status. The authors have provided a solid overview of information applicable to policy and decision making, program design and implementation, and evaluation and applied research. The articles also discuss national and international efforts to develop and apply quality evidence on the effectiveness of health promotion interventions.

Evidence for Planning and Priority Setting

Evidence plays a fundamental role in the processes of health planning, priority settings and policy development. Funding and accountability requirements have also resulted in an increasing demand for proof that policies and programs are based on sound evidence.

Recent investments in health information and research (e.g., Canadian Community Health Survey, Canadian Population Health Initiative, Global Burden of Disease) are significantly advancing our understanding of health status, as well as the determinants of health and their relationship to health issues. This evidence of "opportunity" is beginning to give us clear information about what health problems and factors, if acted on effectively, would yield the best health gains. However, we still have much to learn about what these "effective actions" are. This imbalance in the evidence base is perpetuated by not giving sufficient attention to applied or implementation research, or to comprehensive policy and program evaluation processes. As a result, we often have an imcomplete picture when undertaking population health planning and attempting to establish policy priorities and program initiatives.

In recognition of this problem, work is currently under way at the international level through initiatives such as those sponsored by the International Union for Health Promotion and Education (see "Who's Doing What," page 28). Health Canada and/or Canadian experts have been key participants in these initiatives and it is clear that progress is being made. Evidence that health promotion can achieve outcomes leading to improvements in health status has been documented and disseminated. At the same time, however, there remain a number of gaps and challenges related to the development and application of evidence of effectiveness in our planning and policy work.

Assessing the Gaps

Developing a clear understanding of the gaps and challenges in generating and applying evidence of effectiveness is a critical first step in moving forward to address them. Some of the most significant gaps are:

Lack of an adequate base of applied knowledge in key and emerging health issue areas

A number of key health issue areas (e.g., mental health, community capacity development, at-risk population groups, etc.) are recognized as being sufficiently important to warrant attention as policy priorities.

However, many of these issues and their underlying determinants lack an organized, credible information base on how they can be effectively addressed through policy and program initiatives.

In the short term, some of these knowledge gaps could be addressed by implementing specific synthesis projects and related activities such as consensus conferences. Canada's current participation in global initiatives represents a strategic opportunity to align international evidence gathering with Canadian needs.

Limited systematic evaluation of the effectiveness of existing health promotion strategies

One of the main reasons for the limited evidence base relevant to the Canadian context is the lack of long-term, comprehensive evaluation studies of current policy and program initiatives. While information about the experiences of comparable nations is useful, future decisions regarding Canadian policy and program development should be based on core evidence about the processes, outputs and impacts of Canadian . While it is recognized that there is an increasing level of evaluative activity in Canada, these efforts are often not sustained long enough to determine the full extent of the benefits, or lack tereof, of the intervention(s), particularly in terms of affecting health outcomes. There is also a need to aggregate the results of program evaluation studies and processes, and share information across various issue and program areas.

Need for improved accessibility to, and transfer of, knowledge on effectiveness

Knowledge about the effectiveness of interventions is only useful if it is readily available and applied appropriately in analysis and planning processes. Currently, however, accessing relevant, user-friendly evidence to support these processes is a challenge. Not only is the evidence base limited, but what does exist is housed in numerous sources around the globe.

The development of an integrated system with a common approach for the ongoing identification, storage and retrieval of information on effectiveness relevant to Canadians needs could form the basis for responding to these issues. This system could involve a virtual repository by linking existing sources of credible information and providing users with a portal and "road map" to facilitate access. The experiences of other countries, such as the "Gateway" initiative of the Health Development Agency in Britain, can provide useful insights to inform the potential design of such a system in Canada.

Knowledge transfer strategies are also needed to provide an analysis of evidence in formats that can be readily used in the current Canadian policy context. Training initiatives would also be valuable in supporting the appropriate interpretation and application of evidence in policy and planning processes.

Need to address methodological issues and develop standards for assessing and applying evidence

As noted elsewhere in the Bulletin, numerous issues related to methodology and terminology need to be addressed as part of the process of developing a sound evidence base and applying it appropriately in policy and planning work. The most significant of these issues should be identified and work initiated to build consensus on a consistent set of standards and guidelines. As a starting point, efforts should be directed to such questions as:

• What constitutes sound evaluation methodologies?
• As what point can the findings of evaluation studies be generalized?
• What constitutes sufficient evidence for decision making and planning?
• What do we mean by best practices?
• Under what circumstances is it acceptable to use logic or outcome models to fill knowledge gaps in policy analysis and development work?

Moving Forward

Given the fundamental role of evidence of effectiveness in our efforts to improve health outcomes, these gaps and questions, as well as others than can be identified, require concerted national attention. Such an effort is an essential first step to enhancing the knowledge base needed to guide priorities for investment and to support the design of sound policy and program initiatives that will contribute to the health and well-being of Canadians.

Who's Doing What?

Halina Cyr and Brenda Steinmetz, Strategic Policy Directorate, Population and Public Health Branch, Health Canada

A number of major initiatives are under way to synthesize and disseminate evidence of the effectiveness of health promotion interventions. This article profiles key international initiatives and Canadian policy research networks contributing to the development of the evidence base for health promotion.

The Canadian Corner

The Canadian Consortium for Health Promotion Research

The Consortium, a collaboration of 14 university-based centres, aims to enhance health promotion research, policy and practice in Canada ( Canadian Consortium for Health Promotion Research).

Member centres conduct important work related to effectiveness, including:

• developing community health indicators and the creation of evaluation tools
• preparing working papers such as "Healthy Lifestyle: Strengthening the Effectiveness of Lifestyle Approaches to Improve Health" and "An Assessment of the Methods and Concepts Used to Synthesize the Evidence of Effectiveness in Health Promotion: A Review of 17 Initiatives"
• sponsoring international symposia on effectiveness and collaborating on international work on health promotion evaluation (see page 8)

Research Funding Organizations

The Canadian Institutes of Health Research fund innovative research on health promotion and knowledge transfer strategies. For example, McMaster University is being supported for a project on the Development and Evaluation of Strategies to Summarize and Disseminate the Findings of Systematic Reviews in Public Health and Health Promotion to Practitioners, Health Policy Makers and Consumers, to be completed in late 2002 ( Canadian Institutes of Canadian Research).

The Canadian Population Health Initiative, part of the Canadian Institute for Health Information, provides research funding to address questions such as: To what extent do Canada's major policies and programs improve population health? What are the health status benefits of population-based interventions? What key strategies could be integrated into a comprehensive approach addressing the determinants of health? ( The Canadian Institute for Health Information (CIHI)).

Systematic Review Initiatives

The Effective Public Health Practice Project, an initiative of the Ontario government's Public Health Research, Education and Development (PHRED) Program, is linked to the Cochrane Collaboration. Approximately 20 systematic reviews of effectiveness are available on their website, on topics such as tobacco prevention, adolescent health and nutrition ( City of Hamilton).

The Canadian Task Force on Preventive Health Care conducts systematic reviews and generates recommendations for clinical preventive actions, including primary prevention ( Evidence-Based Clinical Prevention).

Policy Research Demonstration Projects

Ontario's community-based Better Beginnings, Better Futures initiative is a longitudinal policy research demonstration project to provide information on the effectiveness of prevention and promotion as a policy for children. The "Short Term Findings Report" from the demonstration phase (1993-98) is posted on the website ( Better Beginnings).

Established in 1991, the Social Research and Demonstration Corporation is a non-profit organization that evaluates existing social programs and tests new program ideas to provide policy makers and practitioners with reliable evidence about what works in social policy and what does not ( Social Research and Demonstration Corporation).

Sharing Tools and Learning

The Canadian Heart Health Initiative's dissemination research and health promotion capacity building findings are profiled in a recent supplement of Promotion & Education (No. 1, 2001), published by the International Union for Health Promotion and Education.

A national initiative launched in 1999, the Prevention Dividend Project aimed to demonstrate where good investments can be made in prevention and early intervention and to showcase the tools and methods for calculating their economic impact. Funding comes from various sectors, including the federal government ( Prevention Dividend Project).

For up-to-date information on health promotion and population health, visit Health Canada' Population Health website ( http://www.hc-sc.gc.ca/hppb/phdd/).

Did You Know?

Did You Know is a regular column of the Health Policy Research Bulletin examining aspects of health research and data that may be subject to misconceptions. In this issue, we examine some statistical myths related to correlation and causation.

Correlation and Causation Demystified

Allan Pollock, Applied Research and Analysis Directorate, Information, Analysis and Connectivity Branch, Health Canada.

Nearly everyone involved with statistical research has likely heard the long-standing mantra: "Correlation is not causation."

When trying to assess the effectiveness of health internventions, it is often easy to find changes in health indicators that correlate with the implementation of any given health intervention. However, effectiveness can only be inferred from a causal relationship between the intervention and the health outcome. Unfortunately, the true effects of health promotion interventions on health promotion interventions on health outcomes often take considerable time to become evident, meaning that one must consider the delay between a health promotion intervention and health outcomes in order to infer a causal relationship between them.

The purpose of this article is to explain the differences between correlation and causation, and to point out that recent statistical developments are helping resarchers make inferences about causal relationships from their statistical correlations.

Hospitals and mortality: a misinterpreted correlation

In reviewing hospital utilization data, it is apparent that mortality rates are lower for people who have not been admitted to a hospital within a given time period than for those who have. In statistical terms, there is a correlation between the probability of death and whether an individual has been hospitalized. Should we then infer a causal relationship from this statistical correlation - namely, that going to the hospital can increase the chances of dying? In considering this question, we need to be aware that there are three reasons why mortality and hospital utilization may be correlated:

• one may be causing the other
• both variables may be causally related to a third, possibly hidden, variable
• the correlation may be purely the result of chance

While there may be cases of death due to medical malpractice or accident, it is highly unlikely that these constitute a significant proportion of hospital deaths. A closer analysis, however, would reveal that severe illness often leads to death and that individuals in hospitals are much more likely to be seriously ill than those who are not in hospital. After properly considering the relevant relationships between illness, hospital utilization and mortality, it is likely that the opposite relationship will be indicated - namely, that hospitals save lives.

Correlations are rarely simple

The only time that one can infer a causal relationship with confidence is when the correlation is generated by a random experiment. In a random experiment, the only possible explanation for differences between the control and experimental groups is the treatment itself. Since random experiments are costly and sometimes unethical, most relationships must be estimated using data from other sources. Thus, making causal inferences can be difficult, as most correlations are the combined results of a direct causal relationship, an indirect relationship involving a third variable and coincidence.

How researchers deal with the problem of a third variable (such as the severity of illness) depends on whether the variable can be fully observed. In the case of in-hospital mortality, access to data on the severity of the illness leading to death for individuals both in and out of hospitals would allow researchers to model the relationship as described above. If, however, detailed individual-level data do not exist, or if the cause of death cannot be determined for each individual, it is more problematic.

New techniques are available

Recent developments in statistical methods, such as instrumental variables, difference in differences and other methods of exploiting "natural experiments," have resulted in the introduction of a number of techniques for addressing the empirical problems associated with hidden variables. All of these techniques were developed in hopes of simulating a random experiment, which would then allow researchers to make inferences about causal relationships without the need to conduct a truly random experiment.

While the example of in-hospital mortality may appear both extreme and obvious, it highlights a critical aspect of the application of any statistical method - namely, that no statistical technique (regardless of how recent or complex) can yield reliable estimates of a causal relationship unless it is applied within an appropriate theoretical framework.

Using Canada's Health Data

Using Canada's Health Data is a regular column of the Health Policy Research Bulletin highlighting some of the methodologies commonly used in analyzing health data. In this issue, we present an overview of summary health measures of population health.

Beyond Life Expectancy ...

Ron Wall and Robyn Foster, Applied Research and Analysis Directorate of the Information, Analysis and Connectivity Branch, Health Canada

Composite summary measures are widely used to monitor changes in the health of populations and can aid policy makers in comparing the health impacts of diverse health promotion and other interventions. This article describes some composite indicators used in measuring length and quality of life. Beginning with life expectancy, it provides an overview of key measures and describes how they are calculated.

Level and Causes of Mortality: The Impact of Fatal Disease and Injury

Life expectancy (LE) is the average length of life than an individual is expected to live, starting from a given age, if prevailing mortality rates were to continue. Although it can be calculated at any age, LE at birth is the most useful summary measure of the level of mortality for comparing populations as it is sensitive to differences in mortality among infants and the very young.

Potential years of life lost (PYLL) by cause of death is an important measure of premature mortality that weights age of death relative to an arbitrary age (e.g., 75 years) and provides information on the specific causes of mortality. PYLL gives injuries, which result in the deaths of many younger people, greater weight than the circulatory diseases and cancers that are primarily associated with deaths among older people. In identifying deaths that could potentially be avoided, PYLL is a useful tool in informing priorities for prevention and health promotion programs. One of the drawbacks to this measure is that it does not account for years of life lived beyond the arbitrary age.

Disability and Quality of Life: The Impact of Nonfatal Diseases

Disability-free life expectancy (DFLE) reduces LE for the period of time a person lives with a disability. In Canada, self-reported activity limitation has been used as a proxy for disability.

Quality-adjusted life expectancy (QALE) is a broader measure of health expectancy that advances measurement by adjusting LE for the quality of life lived with impaired physical, mental and/or social functioning. Using this approach, years spent in health states that are less than optimal can be aggregated with years spent in good health.

Disability-adjusted life years (DALY) calculates years of healthy life lost to disease and injury by combining years of life lost to premature mortality prior to an arbitrary age with years lived with disability. The weights used to aggregate years lived with disability reflect preferences for different states of disability. Unlike rankings based on premature mortality, the DALY recognizes the health effects of non-fatal health conditions. Disability-adjusted life expectancy (DALE) has also been estimated by adjusting LE for the prevalence of disability from health conditions by age group and the weight estimated for each type of disabilit. The DALY and DALE are limited by the availability/quality of epidemiological data.

New and Noteworthy

New and Noteworthy is a regular column of the Health Policy Research Bulletin highlighting "up and coming" policy research in the health field.

Deputy Ministers Health Services Research Priorities

In the spring of 2001, the provincial, territorial and federal Deputy Ministers of Health identified their joint priorities for health services and related policy research. These priorities are being shared with Canadian stakeholders in the consolidated report Listening for Direction, which is available from the following websites:

• Canadian Health Services Research Foundation - The Canadian Health Services Research Foundation
• Canadian Institutes of Health Research - Candian Institutes of Health Research
• Canadian Coordinating Office for Health Technology Assessment - The Canadian Agency for Drugs and Technologies
• Canadian Institute for Health Information - The Canadian Institute for Health Information (CIHI)

New and Powerful Policy Tool

A new policy tool - the Health Tax Microsimulation Model (HTSIM) - enables health policy makers to stimulate changes in the tax treatment of health-related expenses and assess the distributional impacts and costs of these changes. HTSIM shows the revenue changes and distributional impacts of: modifying the tax parameters of existing measures such as the medical expenses tax credit; assigning (new) taxable benefits; assessing new ways to help with out-of-pocket health expenses; and even modifying existing or introducing new income support programs. The Microsimulation Modelling and Data Analysis Division (MSDAD) and Health Canada's health policy community are working together to formulate viable tax-delivered health options. For more information, contact Anil Gupta at anil_gupta@hc-sc.gc.ca.

OECD 2001 Health Data CD-Rom

The Organisation for Economic Co-operation and Development (OECD) has released its 2001 Health Data CD-ROM containing health data on OECD member countries for multiple years. The CD-ROM allows users to develop multi-country comparisons on issues such as mortality, morbidity, health care resources and utilization, and financing and remuneration. The results are available in a variety of formats and can be exported for use in Excel® and other documents. For more information, visit the OECD website at Organisation for Economic Co-operation and Development

Funding for Health Policy Research

A strategic, targeted contribution program of Health Canada's Applied Research and Analysis Directorate (ARAD), the Health Policy Research Program (HPRP) generates extramural policy-relevant research designed to meet the needs of the Department. The HPRP supports research and developmental projects; policy-relevant projects, workshops, seminars and conferences; and federal/provincial/territorial health research partnerships.

Requests for proposals in Health Canada's priority areas will be posted on the website on the following dates:

• primary research and/or synthesis - annually on January 15, September 15 and November 15
• workshop and/or seminar - annually on March 1, October 1 and December 1

Best Practices for Comprehensive Tobacco Control Programs

A new evidence-based guide from the U.S. Centers for Disease Control and Prevention (CDC) describes key elements of effective state-level tobacco control programs, including programs for communities, schools and the entire state. Best Practices for Comprehensive Tobacco Control Programs (August 1999) drasws on lessons learned to address the impact of various programs in reducing the burden of tobacco-related diseases. The guide is available on the CDC website at Tobacco Information and Prevention Source

A Tool for Population Health Action

Population Health Template: Key Elements and Actions That Define a Population Health Approach is a resource for advancing understanding of the population health approach. A work in progress by Health Canada, it organizes and consolidates current knowledge about population health under eight "key elements." The template has a variety of uses, including: policy and program development and implementation; training and education; and evaluation. The template was used to develop the framework for analysis for the Case Studies of the Regional Mobiliation of Population Health. The documents are available on the Population Health website at http://www.hc sc.gc.ca/hppb/phdd/resources/index.html

Special Issues of the American Journal of Health Promotion

"The Financial Impact of Health Promotion" is the subject of a special issue of the American Journal of Health Promotion (May/June 2001, Vol. 15, No. 5). The publication presents corporate and federal goverment policy perspectives and addresses methodological challenges. It also includes literature reviews of the financial impact of health promotion programs and smoking on health-related costs. For a detailed table of contents, visit the Journal website ( American Journal of Health Promotion ). Also, watch for upcoming special issues on "The Most Effective Health Promotion Strategies" and "The Health Impact of Health Promotion."

Indicators that Count! - Measuring Population Health at the Community Level

A resource from the University of Toronto's Centre for Health Promotion sets out a framework for assessing health at the community level. The framework encompasses a broad range of indicators. The authors conclude that, for indicators to have meaning and benefit at the local level, they must be relevant to community values, goals and issues. Moreover, they must be developed in a process that engages both policy makers and the public. For additional information, visit http://www.utoronto.ca/chp/p-titles.htm

An Ounce of Prevention ... What Are the Returns?

The U.S. Centers for Disease Control and Prevention have produced An Ounce of Prevention ... What Are the Returns? (2nd Edition, 1999), which reports on 19 disease prevention and healthy lifestyle strategies that promote health and make good economic sense.

A compilation of one-page summaries, the report addresses strategies such as those to prevent smoking, low birthweight and various cancers. It is available at Centers for Disease Control and Prevention

Canada's Health Care Providers

The Canadian Institute for Health Information (CIHI) has just released a report entitled Canada's Health Care Providers. This special report is the first of its kind and presents a fact-based compilation of current research, historical trends and new data, and findings and analysis on Canada's health care providers. While the scope includes regulated, unregulated and informal care providers, the focus is on the former because of the richer data available and the growing public and policy interest. This report is available at The Canadian Institute for Health Information (CIHI)

Socieconomic Inequality and Heart Disease

Inequality Is Bad for Our Hearts: Why Low Income and Social Exclusion Are Major Causes of Heart Disease brings together the latest evidence of the role that low income plays in the development of cardiovascular disease. Income effets manifest themselves through processes of material deprivation, excessive phychosocial stress and adoption of unhealthy behaviours. Reasons for the increasing incidence of low income among Canadians are outlined and policy solutions are offered. Commissioned by Dennis Raphael of York University, the report is available at http://depts.washington.edu/eqhlth/paperA15.html

Mark Your Calendar

References

References for "Health Promotion: What is it?" (p. 5)

1. Health Canada. Towards a Common Understanding: Clarifying Core Concepts of Population Health. A Discussion Paper Developed by the Conceptual Framework Subgroup of the Working Group on Population Health Strategy ( http://www.hc-sc.gc.ca/hppb/phdd/docs/common/index.html). Ottawa: Health Canada, 1996.

Return

References for "Issues and Challenges in Assessing the Effectiveness of Health Promotion" (p. 10)

1. Rootman I et al. (Eds.). Evaluation in health promotion: Principles and perspectives (European Series, No. 92). Denmark: World Health Organization Regional Publications, 2001.

2. Posavac EJ, Carey RG. Program evaluaton methods and case studies, 5th Edition. Upper Saddle River: Prentice Hall, 1997.

3. Emmons KM. Health behaviour in a social context. In: Berkman LF, Kawachi I (Eds.), Social epidemiology. New York: Oxford University Press, 2000: 242-66.

4. Mays N, Pope C. Qualitative Research: Rigour and Qualitative Research. British Medical Journal 1995; 311: 109-12.

5. Murphy E, Dingwall R. Qualitative methods in health services research. In: Black N, Brazier J, Fitzpatrick R, Reeves B (Eds.), Health services research methods: A guide to best practice. London: BMJ Books, 1998.

6. MacIntyre S, Ellaway A. Ecological approaches: Rediscovering the role of the physical and social environments. In: Berkman LF, Kawachi I (Eds.), Social epidemiology. New York: Oxford University Press, 2000: 332-48.

Return

Additional References:

Drummond MF, O'Brien B, Stoddart GL, Torrance GW. Methods for the economic evaluation of health care programmes, 2nd Edition. New York: Oxford University Press, 1997.

Nutbeam D. Health Promotion Effectiveness - The Questions to be Answered. In: The Evidence of Health Promotion Effectiveness - Shaping Public Health in a New Europe. International Union for Health Promotion and Education, 1999.

Pope C, Mays N. Qualitative Research: Reaching the Parts Other Methods Cannot Reach: An Introduction to Qualitative Methods in Health and Health Services Research. British Medical Journal 1995; 311: 42-5.

References for "A Tale of Two Evaluations: CAPC and CPNP" (p. 14)

1. Boyle MH, Offord DR. Community Action Program for Children (CAPC) Proposal for a National Evaluation Framework. Centre for Studies of Children at Risk, McMaster University Faculty of Health Sciences and Chedoke-McMaster Hospitals, 1995.

2. Health Canada, Evaluation Group, Strategic Planning and Review Directorate. Canada Prenatal Nutrition Program Evaluation Framework. Prepared by Gail V. Barrington and Associates, Calgary, Alberta, 1996.

3. Health Canada, Departmental Program Evaluation Division. Module 3: Performance Measurement Training Series. Case Study on the CPNP Data Collection System. Prepared by Barrington Research Group, Inc., Calgary, Alberta, 2001.

4. Boyle MH. Community Action for Children (CAPC) National Impact Evaluation: Preliminary Findings Interim Report. 1999.

5. Health Canada. CPNP PPHB Participants: National Frequency Report, January 1995-May 2001. Prepared by Barrington Research Group, Inc., Calgary, Alberta, 2001.

6. Boyle MH. Community Action Program for Children (CAPC) National Evaluation: Impact Analysis. 2000. For a complementary national analysis see: Palacio-Quintin E, Community Action Program for Children (CAPC) National Impact Evaluation: New Analysis, 2001 at p. i.

7. Health Canada. Canada Prenatal Nutrition Program.ICQ report 2000. Prepared for Health Canada by Barrington Research Group, Inc., Calgary, Alberta, 2000.

8. -Guyn L. Canada Prenatal Nutrition Program: Measuring the Level of Implementation. Unpublished paper prepared by Barrington Research Group, Inc., Calgary, Alberta, 2001.

Return

References for "Partnering for Social Change: The 'Back to Sleep' Campaign" (p. 19)

1. Health Canada. Benchmark Survey on Awareness, Knowledge and Behaviours Relating to Sudden Infant Death Syndrome (SIDS). Prepared for Health Canada by Environics Research Group, March 1999.

2. Health Canada. Sudden Infant Death Syndrome: A Tracking Survey on Awareness, Knowledge and Behaviours. Prepared for Health Canada by Environics Research Group, June 2001.

Return

References for "Gender Matters: Evaluating the Effectiveness of Health Promotion" (p. 22)

1. Health Canada's Women's Health Strategy (Minister of Public Works and Government Services Canada, 1999) identifies 64 commitments pertaining to women's health and gender-based analysis. For working definitions of key terms, see also: Health Canada's Gender-based Analysis Policy (Minister of Public Works and Government Services Canada, 2000); and Wizemann TM, Pardue M (Eds.), Exploring the Biological Contributions to Human Health: Does Sex Matter. Institute of Medicine: National Academy of Sciences, 2001. For a list of resources pertinent to this article see the Women's Bureau website: http://www.hc-sc.gc.ca/english/women/

2. Colman R. Women's Health in Atlantic Canada: A Statistical Report. Centres of Excellence for Women's Health Research Bulletin Fall 2000; Vol. 1, No. 1: 5.

3. Health Canada's Gender-based Analysis Policy, op. cit., 1.

4. For examples of participatory research in health promotion, see: Denton M, Hakjukowski- Ahmed M et al. Women's Voices in Health Promotion. Toronto: Canadian Scholars Press, 1999; and resources produced until 1999 through the McMaster Research Centre for Promotion of Women's Health ( http://www.mcmaster.ca). See also research projects funded through Health Canada's Centres of Excellence for Women's Health ( http://www.cewh-cesf.ca).

5. Thurston Dr. Wilfreda. Personal communication. November 2001. Thurston WE, Crow BA, Scott CM. The role of women's organizations in health policy development, implementation and dissemination. Report to Health Canada, Health Promotion and Programs Branch, 1998.

6. Horne T, Kirby S, Trudeau J. Final Evaluation Report of Catching Our Breath Too. Winnipeg: Women's Health Clinic, 1999.

7. Grant K. 2002. GBA: Beyond the Red Queen Syndrome. Presentation at the Gender-based Analysis Fair. Ottawa, January 31, 2002. See also: Armstrong P, Amaratunga C, Bernier J et al. Exposing Privatization: Women and Health Care Reform in Canada. Toronto: Garamond Press, 2001.

8. See Baume E, Juarez M, Standing H. Gender and Health Equity Network: Gender and Health Equity Resource Guide. Sussex: Institute of Development Studies, 2001; Status of Women Canada. Gender and Performance Measurement: A Summary Review and Report. 2001. Includes a useful bibliography on gender-sensitive evaluation guides.

Return

References for "Creating the Right Mix" (p. 24)

1. Tengs TO et al. Five Hundred Life-Saving Interventions and Their Cost Effectiveness. Harvard Centre for Risk Analysis, 1994. Tengs et al. stress the effectiveness of "medical" interventions by comparison with interventions from other sectors, and the relatively high cost of primary as compared with secondary and tertiary prevention.

2. This concept is sometimes summarized by use of the term "comprehensiveness" in health promotion; the challenge is to deliver on the concept.

3. Allen MS, Navigant Consulting. Business Portfolio Management. New York: John Wiley and Sons, 2000. If Health Canada was a private sector organization, it would be described as a conglomerate or group because it is the sum of several smaller organizations with discrete mandates and cultures.

4. Ibid., 13

5. Lochhead C. The Trend Toward Delayed First Childbirth: Health and Social Implications. Isuma: Canadian Journal of Policy Research, Vol. 1, No. 2 (Autumn 2000): 41-4.

References for "Using Canada's Health Data" (p. 32)

Additional References:

Manuel DG, Goel V, Williams JI, Corey P. Health-adjusted life expectancy at the local level in Ontario. Chronic Diseases in Canada 2000; 21(2): 73-80.

Statistics Canada and Canadian Institute for Health Information (2001a). Highlights of selected indicators. Health Indicators; 2001(3): December issue.

Statistics Canada and Canadian Institute for Health Information (2001b). How healthy are Canadians? The health divide - How the sexes differ. Health Reports, 2001; 12(3): special issue.

Wolfson MC. Health-adjusted life expectancy. Health Reports 1996; 8(1): 41-6.

World Bank. World development report 1993: investing in health. New York: Oxford University Press for the World Bank, 1993.

World Health Organization. The world health report 2000. Health systems: Improving performance. Geneva: World Health Organization, 2000.

World Health Organization. The world health report 2001. Mental health: new understanding, new hope. Geneva: World Health Organization, 2001.

Further reading: for applications of LE, PYLL, and DFLE indicators, see: Statistics Canada and Canadian Institute for Health Information (2001a, 2001b); for QALE see Manuel et al. (2000), Wolfson (1996); for DALY see World Bank (1993), World Health Organization (2001).