ARCHIVED - Summative Evaluation of the Canada Prenatal Nutrition Program 2004-2009

Overview

1.0 Introduction and Context

1.1 CPNP Overview

Background and Description

At the 1990 United Nations World Summit for Children Canada made a commitment to invest in the well-being of vulnerable children. In 1992, the Government of Canada launched Brighter Futures: Canada’s Action Plan for Children, which put in place a number of initiatives aimed at promoting the health and well-being of children. One of these initiatives was the launch of the Community Action Program for Children (CAPC) which focused on providing comprehensive, culturally appropriate prevention and early intervention initiatives to promote the health and social development of young children zero to six years of age and their families facing conditions of risk. Two years later, in 1994, the federal government announced the Canada Prenatal Nutrition Program (CPNP), which extended the support offered by CAPC into the prenatal period.

CPNP provides long-term funding to community groups and coalitions to develop or enhance services that address the needs of at-risk pregnant women and their babies. Operated by the Public Health Agency of Canada (PHAC)^{Footnote 1}, the program supports comprehensive, community-based services and is designed to build upon existing prenatal health programs or to establish them where they do not exist. CPNP operates within PHAC authority. A separate CPNP operates through Health Canada to deliver programming specific to the northern and on-reserve Aboriginal population. Only CPNP operating through PHAC is evaluated in this report.

CPNP Objectives and Framework

The goal of CPNP is to reach pregnant women living in conditions of risk that are known to increase the likelihood of unfavourable outcomes for themselves and their infants. These conditions of risk include: poverty, teenage pregnancy, social or geographic isolation with poor access to services, recent arrival to Canada, alcohol or substance abuse and family violence. CPNP also increases the availability of culturally sensitive prenatal support for Aboriginal women who are not living on reserves.

CPNP is a community-based initiative founded on the principle that communities are well positioned to recognize the needs in maternal and infant health and have the capacity to draw together the resources to address those needs. The following six Guiding Principles inform program design, implementation, operations, governance and evaluation and are rooted in health promotion principles:

• Mothers and Babies First
• Strengthening and Supporting Families
• Equity and Accessibility
• Partnerships
• Community Based
• Flexibility

As a comprehensive program, the services provided by CPNP include food and vitamin supplementations, nutritional counselling, food preparation training, breastfeeding education and support, education and support on infant care and child development and referrals or counselling regarding health and lifestyle issues. CPNP is designed to build upon existing prenatal health programs or to establish them where they do not currently exist using a population health approach. The cornerstone of the program theory is that the health and development of even the most vulnerable children can be protected from conditions of risk by providing funding for early intervention. Experts point to community-based models as being the most effective in delivering services, in part because of the community’s understanding of its own culture and specific needs (Human Resources and Skills Development Canada (HRSDC), 2004).

Employing a population health promotion approach, the objectives of CPNP are to:

• Reduce the incidence of babies born with birth weights that are too high or too low
• Improve the health of pregnant women and their infants
• Promote and support the initiation and duration of breastfeeding
• Increase the accessibility of services and community supports for pregnant women
• Build partnerships, linkages and collaboration within communities

The relationship between the activities implemented by CPNP and the short and long-term outcomes of the program is presented in the CPNP Logic Model (Figure 1), which is part of the Results-based Management Accountability Framework (RMAF) that guides the national evaluation of CPNP.

CPNP Resources

The annual budget of CPNP is $30.8 million. Of this, nearly $27.2 million goes directly to communities to fund local projects. The balance of funds is allocated for operations and management, salaries and a national strategic fund. CPNP funding levels are determined on a provincial/territorial basis that considers the number of children zero to six years of age in each province or territory. A breakdown of funding and number of projects by province is provided in Table 1.

(PHAC, DCA, 2008-2009)

CPNP Governance

CPNP is jointly managed by the federal government, provincial and territorial governments, the community, and other stakeholders through the collaborative work of Joint Management Committees (JMC). A JMC exists in each of the seven CPNP regions: Northern Region, British Columbia, Alberta Region, Saskatchewan and Manitoba Region, Ontario Region, Quebec Region, and Atlantic Region. Coalitions comprised of Project Coordinators and PHAC representatives also operate in some CPNP regions. Administrative protocols established for CPNP outline the terms and conditions regarding how the program will be managed in each province and territory.

Within PHAC, a National Evaluation Team for Children (NETC) comprised of both national and regional evaluators works collaboratively to support CPNP evaluation. NETC is structured to include the perspectives and experience of the national and regional PHAC offices and is mandated to advise on evaluation considerations across the continuum of PHAC investments in children’s health. Members of NETC have been involved in the design, integration and implementation of performance measurement and evaluation processes at the national, regional and local levels thereby supporting policy, evaluation and research decisions related to CPNP. An effective communication system links the national office to CPNP staff in regional offices, who in turn link directly to communities. This system enables national and regional staff to respond to the diverse needs of community groups.

1.2 CPNP Evaluation Context

Evaluation Purpose

The completion of this summative evaluation will satisfy the final reporting commitments outlined in CPNP’s RMAF. The RMAF is a requirement for the accountability of federal contribution funding. Reporting commitments for the program are also captured under the RMAF for the Promotion of Population Health (PPH RMAF) which is PHAC RMAF. Terms and Conditions for health promotion programs within PHAC are scheduled for renewal in 2010; therefore, each program supporting PPH RMAF outcomes must fulfill their reporting commitments as part of the 2010 renewal process. As such, findings from this summative evaluation will also satisfy reporting commitments for the PPH RMAF.

As with all summative evaluations, this report will analyze the performance of the program with respect to achievement of intended outcomes. Given that the program has been successfully delivered for many years, this evaluation will also be able to assess the effectiveness and overall value of the program. This evaluation contains findings and provides lessons that will help decision makers to plan for the future and build on experiences of the past.

The intended audience for this report includes: PHAC Evaluation Committee members, staff from other areas within PHAC, program participants, stakeholders, researchers and domestic and international audiences interested in public health and, in particular, community-based health programs.

Evaluation Scope

The scope of this evaluation is limited to program findings gathered between 2004 and 2009 (five years), given that a comprehensive evaluation was last completed in 2004. The evaluation is intended to address select questions as outlined in CPNP’s RMAF and arrive at a set of conclusions that describe: the continued relevance of the program, its impact and its cost-effectiveness. Addressing these three areas of evaluation is a requirement of the 2001 Treasury Board policy on evaluation. The evaluation questions are:

Relevance

• To what extent do threats to the health of pregnant women, their infants and families persist?
• Is CPNP in line with current federal government domestic and international commitments addressing population health and health disparities affecting pregnant women, their infants and families?
• Is a federal role in addressing health disparities affecting pregnant women, their infants and families still valued by other stakeholders?

Impact

• To what extent are CPNP projects successfully reaching pregnant women facing conditions of risk?
• How has CPNP contributed to improved health and reduced health disparities for pregnant women and their infants facing conditions of risk?
• To what extent do CPNP projects implement a population health approach?^{Footnote 5})

Cost-Effectiveness

• Are Canadians getting value for their tax dollars from this program?

CPNP’s RMAF was created in 2004 to more concretely connect the activities of CPNP to its ultimate objective of contributing to improved health and reduced health disparities for pregnant women and their infants facing conditions of risk. The evaluation questions from the RMAF, listed above, will be assessed across comprehensive indicators that will illustrate CPNP’s impact on the following intended immediate and intermediate outcomes:

• Increased capacity of CPNP to inform policy and programming for populations facing conditions of risk.
• Projects successfully implement a population health approach
• CPNP projects successfully reach and provide access to appropriate services for pregnant women and their infants living in conditions of risk
• Increased support at the community level for pregnant women facing conditions of risk
• Improved personal health practices of CPNP participants

CPNP’s Logic Model as presented in the RMAF is presented in Figure 1.

Figure 1 – CPNP Logic Model

2.0 Methodology

2.1 General Approach and Design

The foundations for this summative evaluation are the national CPNP performance measurement tools that were developed to collect data on an annual basis spanning the period of the program RMAF. The tools were designed to accomplish unique objectives but also to complement each other with the over-arching goal of achieving comprehensive data collection and robust analyses. The strength of these tools and their linkages to one another laid the groundwork for the analyses that informed the findings on program relevance, impact and cost-effectiveness.

National Evaluation Tools

Individual Project Questionnaire (IPQ):

The IPQ was an annual survey designed to collect information on individual project administration, the provision and utilization of services, and participants and their birth outcomes.

The IPQ was used to capture project-level information from CPNP projects starting in 1996-97 until 2005-2006. Information was reported at the national, regional and project level.

Individual Project Questionnaire (IPQ2):

The ICQ2 was a participant level tool, was designed to measure program impacts. It collected information on participants’ health and demographics, use of CPNP services and pregnancy outcomes. This tool was linked through a sampling strategy to a second participant tool called the Welcome Card described below.

In 2006, the ICQ2 tool was suspended to allow for the assessment of the impact of CPNP on health practices and birth outcomes. The impact analysis of ICQ2 data is expanded upon in Section 2.3

Welcome Card (WC):

The Welcome Card gathered aggregate information on 100% of CPNP participants. The tool consisted of a short set of questions administered when participants entered the program as a part of the welcoming process. It was used to generate participant counts, identify a participant sample, and provide an entrant profile.

In 2008-2009 the tool was revised to include additional demographic information in order to provide improved data on program reach and to align it with public health goals.

2.2 CPNP National Evaluation Framework

The IPQ and ICQ2 data collection tools were suspended in 2006 so that analysis could be performed regarding the areas of program relevance, impact and cost-effectiveness. Detailed data collection through this series of linked national evaluation tools created a strong data set of national information that permitted the completion of several detailed analyses addressing specific RMAF evaluation questions. The reports, developed between 2004 and 2009, are described in greater detail below. These evaluative studies were conducted using varying methods and captured both quantitative and qualitative data. The completion of these reports allowed for a meta-analysis of CPNP evaluation findings critical to the development of this summative evaluation.

2.3 Data and Limitations

This summative evaluation used a mixed-methods approach to consolidate the findings of the national evaluation reports outlined below; however, each method relied upon common primary data that was generated by the three national data collection tools described above. The methods and limitations specific to each national evaluation report are outlined below. Note that not all projects participated in the ongoing national evaluation of CPNP. Respecting regional joint management decisions, CPNP projects in Quebec and projects serving off-reserve Aboriginal communities in Ontario participate in alternate data collection systems and are not included in the analysis of findings. Analyses of IPQ, ICQ2 and WC data excludes Quebec and Ontario-Aboriginal projects unless otherwise stated.

National Evaluation Reports

1. CPNP Profile of Participants – (PRA Inc., 2007)

Methodology: Quantitative analyses were conducted using program reach and outcome data that was then compared to a subsection of the general population using Statistics Canada’s Canadian Vital Statistics – CANSIM 2004-05; Birth Database 2004; and Canadian Community Health Survey 2005. The purpose of this report was to provide a description of the CPNP participant population and a comparison of CPNP participants to the general population of expectant and new mothers in Canada.

Limitations: The following two factors may limit the data comparability. First there are differences between CPNP participants and the reference population (e.g., age limitations within the reference population and time limitations within the CPNP population). Second, there are differences in data collection instruments, such as question wording, that create some limitations around the analysis.

2. CPNP National Individual Project Questionnaire (IPQ) Report (2005 to 2006) – (PRA Inc., 2007)

The ICQ2 was a participant level tool, was designed to measure program impacts. It collected information on participants’ health and demographics, use of CPNP services and pregnancy outcomes. This tool was linked through a sampling strategy to a second participant tool called the Welcome Card described below.

In 2006, the ICQ2 tool was suspended to allow for the assessment of the impact of CPNP on health practices and birth outcomes. The impact analysis of ICQ2 data is expanded upon in Section 2.3

3. Investments in Children’s Health: Systematic Review of Evidence – (Shiell, 2007)

Methodology: This meta-analysis assessed the results of previous reviews within the early childhood development field to determine similarities between CPNP and other initiatives that had been assessed in greater detail. The search strategy was refined and applied to the Cochrane, Medline and several other prominent databases. A two-stage screening process followed.

Limitations: Study design and presence of a control group were not used as eligibility criteria in this study. As such, it is difficult to gauge the quality of the reviews that were assessed and determine whether the findings were applicable to CPNP.

4. Assessment of the Economic Impact of CPNP – (McMurchy, 2008)

Methodology: The methodology consisted of a comprehensive literature review; project documentation and data review and analysis plan development (Statistics Canada, Ontario Antenatal Records, Vital Statistics);data collection, preparation and analysis.

Limitations: There were gaps in some information related to program costs and effectiveness. However where there were gaps, results were applied from the published literature and population-based datasets.

5. Qualitative Analysis of CPNP Success Stories – (Estable, 2008)

Methodology: Qualitative methods were primarily applied in order to analyze narrative sections of documents (“stories”) that had been systematically collected within the IPQ over a ten-year period. A multiple stream design was applied to the available data that combined a thematic analysis with a quantified content analysis. Select participant and project quotes are included throughout the impact section of this report in text-box format.

Limitations: The stories provide ‘snapshots’ that focus on project activities and participant responses or observations from project staff. Because these stories were generated in response to a request from the projects’ funder so unsuccessful experiences were not highlighted.

6. CPNP Relevance Literature Review – (McGuire, 2009)

Methodology: Qualitative and quantitative methods were used, including: (1) a comprehensive literature review of both academic and grey publications and (2) analyses of Statistics Canada data, Census data and data from the Welcome Card and IPQ. This report was intentionally designed to support the summative evaluation by addressing questions about the relevance of the program.

Limitations: Literature on health threats associated with the determinants of health, specifically among pregnant women and their infants and families, is very limited. As such, it was necessary to reference articles and data from outside the period of the RMAF (years prior to 2004) in certain circumstances. Certain quantitative data were not accessible due to authorization restrictions and were therefore not included.

7. Application of Best Practices in CPNP – (PHAC, 2009a)

Methodology: Data was collected from key informants (PHAC employees only) from June to August 2008 using four different questionnaire templates, based on the type and governing level of work of the key informant. Qualitative analyses were then conducted to produce program-wide findings.

Limitations: Specific examples from regional Program Consultants and regional Program Evaluators may not apply to other regions and therefore cannot be generalized to the program as a whole. Due to staff turnover in the regions, regional responses may reflect a variety of CPNP sources.

8. Understanding the Impact of the CPNP: A Quantitative Evaluation – (Muhajarine, 2009)

Methodology: Quantitative analyses including statistical modeling and several types of regression analyses were completed on clients’ participation in the program, their health behaviours and birth outcomes. The primary data source was a four-year data set from the ICQ2.

Limitations: While analyses controlled for the socio-demographic risk factors of participants on which data were available, it is difficult to attribute the effects solely to exposure to CPNP. Quebec and off-reserve Ontario Aboriginal projects did not administer the ICQ2 tool, therefore no data are provided from these projects were included in the analyses.

Within each national evaluation report, findings are supported by evidence from more than one source, given that typically, a “multiple lines of evidence” technique is used to confirm or refute the evaluation findings and support balanced public reporting. In accordance with Treasury Board standards, the questions of validity, credibility and reliability of the data were consistently measured to ensure quality.