ARCHIVED - Summative Evaluation of the Canada Prenatal Nutrition Program 2004-2009

Conclusions & Recommendations

6.0 Key Conclusions

6.1 Relevance

A myriad of social, economic, and physical issues threaten the health of pregnant women and their infants. Some of these threats include: poverty, teen pregnancy, poor access to health services and basic amenities, and health behaviours. While these threats are declining for the general population in Canada, there is evidence that they persist among at-risk populations. For example, Aboriginal women continue to experience conditions of risk to a greater extent than the general population of Canadian women. Threats to health must be continually monitored and investigated given the way they disproportionately affect certain populations. Vigilant monitoring will safeguard against the assumption by policy makers and other health professionals that declines in health risks among the general population are representative of the risks affecting vulnerable populations.

As a federal government program, CPNP is accountable to government commitments at both the domestic and international level. Alignment with these commitments ensures the efficacy of the government system and the ability to tackle issues of health on a domestic and international level. The federal government has made several domestic commitments to address health disparities that impact pregnant women, infants, and their families. CPNP is aligned with several of these, including Brighter Futures: Canada’s Action Plan for Children, the National Children’s Agenda, the ECD Agreement (Social Union, 1997; Social Union, 2000), and the current PHAC RPP (PHAC, 2008a). Internationally, CPNP is aligned with federal commitments such as Articles 18 and 24 in the UNCRC, A Canada Fit for Children (Canada’s Second Action Plan), and Canada’s Action Plan for Food Security (United Nations, 1989; Government of Canada, 2004; Government of Canada, 1989). CPNP remains a valuable program to at-risk populations, communities and other stakeholders as it increases access to health services, provides support not found elsewhere in the community, encourages funding initiatives and functions as a venue for skill development.

CPNP is part of Canada’s solution to the global challenge of addressing issues of maternal and infant health. Although CPNP is primarily a community program, it is understood that it is also part of the larger federal role to address health disparities in Canada, especially given CPNP’s capacity to provide leadership and opportunities for collaboration among various levels of government and other stakeholders on issues of maternal and infant health. Furthermore, CPNP collaborations on issues of health inequalities extend into the international community.

6.2 Impact

Although many of the factors threatening the health of pregnant women and their infants are declining in Canada, at-risk populations still experience these threats at a disproportionately higher rate. The proportion of CPNP participants characterized by conditions of risk – such as poverty, teenage pregnancy, lone parenthood, alcohol use or smoking during pregnancy – is far above the rates reported for these characteristics among the general Canadian population, demonstrating that CPNP effectively reaches women and infants in greatest need.

To assess the ability of CPNP to achieve its key outcomes, the effect of exposure to the program was examined. Specifically, the analysis sought to determine whether CPNP has improved the health practices of at-risk pregnant women and the birth outcomes of their infants. Overall, it was found that participants who had more exposure to CPNP were more likely to make positive behaviour changes and that their infants were consistently less likely to experience adverse birth outcomes. Specifically, participants who had high exposure to the program were more likely to increase use of vitamin supplements; decrease the number of cigarettes smoked; quit drinking alcohol; initiate breastfeeding; and breastfeed their infants longer than clients who had lower overall CPNP exposure. The infants of participants who had more exposure to CPNP were also less likely to experience preterm birth, LBW, be SGA or be of poor neonatal health. Some outcomes were unexpected, such as the increased likelihood of having a LGA infant among women with high exposure to CPNP. As well, overall high exposure to CPNP and receipt of several CPNP services were inexplicably associated with increased likelihood of gaining more than the recommended amount of weight during pregnancy. An analysis was conducted to see if a relationship existed between those women most likely to gain more than the recommended amount of weight during pregnancy and those most likely to have LGA infants but none was found.

Both the service-specific analysis of impact of exposure to CPNP and the analysis by socio-demographic group produced mixed results. For example, it was found that women who received one-on-one nutrition and lifestyle counselling were less likely to quit drinking while pregnant; however, this may have been due to the fact that women who received this service were at greater risk than those who were not offered this service. It may also have been that those women were less likely to quit drinking regardless of the services they received. Another unexpected finding was that women with high exposure to CPNP who reported “no income” were more likely to breastfeed their baby for a shorter duration than women from all other income categories. As such, future analyses for the program should be structured to address these counter-intuitive results, and provide a greater understanding of other factors that may influence maternal health behaviours and birth outcomes. It would also be useful to determine the defining characteristics of participants who have benefited the most from specific services and attempt to structure the program such that participants will be matched with the services that have shown to yield the greatest possible impact.

Despite the unexpected results described above, the significant contribution of CPNP to the health and well-being of infants has been demonstrated. For example, the comparative analysis revealed that the rates of breastfeeding and LBW among CPNP participants were generally as good as, if not better than those found among the general population. These are striking results given the additional hardships faced by many CPNP participants. In particular, it is worthy of note that in the case of LBW outcomes, CPNP infants were compared to all Canadian births as birth weight data from a matched population were not available. It is highly likely that a comparison of LBW outcomes between CPNP participants and a matched population of non-participants from the general public would further illustrate the considerable contribution CPNP makes regarding the health and well-being of infants born into challenging life circumstances. The cost savings associated with avoidance of LBW outcomes is described in section 6.3.

Nonetheless, caution must be exercised when interpreting impact results given that attenuation of risk among vulnerable populations can be difficult to quantify. In many cases, participants who are offered specific support services are facing more difficult life circumstances thereby putting them at greater risk of unhealthy behaviours and unfavourable birth outcomes for their infants. Mitigation of that risk may not always be apparent, tangible or measurable. Inferring causality is inadvisable when reviewing the results of the analysis of exposure to CPNP across services and socio-demographic categories given that it is difficult to attribute changes in behaviour to a single intervention; rather, it is more accurate to examine overall program contribution. Thus, it would not be appropriate to conclude that the increased likelihood of negative outcomes associated with particular services as found in this evaluation mean that the support is harmful. The same caution must be exercised with regard to the CPNP support that was associated with increased likelihood of positive outcomes among some participants, given that the individual characteristics of these women (such as level of motivation and family support) may make them more likely to succeed in changing their health and lifestyle behaviours than other women.

Furthermore, given that CPNP projects are unique within their communities, considerable variation exists across CPNP projects with respect to how, by whom and to whom specific services are provided. Variation may also exist in the way receipt of these services is recorded. Finally, participants who initiate contact with CPNP earlier in pregnancy, have more contact with the program and stay in the program longer are likely to differ in significant ways from those who have lower levels of program exposure. For example, a women with higher levels of exposure to CPNP may be at a different stage of readiness for change, have greater support from a family member or friend to attend, or be mandated to participate in CPNP activities by the justice system. As such, a subject for further study may be to establish a common definition of CPNP services that are offered by projects nationally. With more standardized definitions the impact of services would be easier to measure and analyse.

Addressing the health of pregnant women and their infants is complex, and more research is needed to determine the presence of cause and effect relationships between CPNP and positive or negative outcomes among participants. Consequently, more research is also needed to determine whether CPNP has the ability and capacity to reduce health disparities in Canada. Generating longitudinal data regarding CPNP participants, if only from program entry to exit, poses a unique challenge among vulnerable populations given that they are often hard to reach. Considering that the health of a population is dynamic and influenced by fluctuations in social, economic and environmental factors, capturing data and determining the CPNP’s impact on health disparities among pregnant women, their infants and families in Canada will remain an ongoing challenge.

6.3 Cost-Effectiveness

Early child development programs have been shown to be among the most cost-effective of public health interventions leading to a wide range of beneficial social outcomes. However, the potential for application and extrapolation of this evidence to policy and practice in Canada needs to be examined further. The economic evidence that has been reviewed provides a strong case for public investment in early child intervention. The value of the benefits that are generated by ensuring healthy mothers, healthy infants and healthy children and families far exceed the costs incurred, even when considering the time it takes to realize those benefits. The impact of early childhood intervention and prevention measures with respect to health and social development impact not only children, but their parents and society at large. Indeed, taxpayers are by far the biggest beneficiaries of health and social support services for mothers, infants, children and families, which supports the importance of federal investment in such programs.

Cost savings generated by breastfeeding initiation pertain to the rates of illness reported by CPNP participants during the first two weeks of their child’s life and based on rates of infant illness in the first year of life as reported in the literature. The $42 in cost savings estimated for the first year of life of a breastfed baby are extremely conservative and do not consider the cost savings due to illness avoided throughout childhood and adulthood that may also be attributable to breastfeeding during the first year of life. With respect to illnesses avoided, these savings increase to $63 among Aboriginal participants, which significant given that the mandate of CPNP is to increase the availability of culturally sensitive services for Aboriginal women. In fact, when assessed across socio-demographic categories of risk (poverty, teen pregnancy, etc…), each individual at-risk group experiences greater savings than the $42 average savings for all CPNP participants combined. This is notable because many participants experience more than one risk characteristic. In future analyses it may be useful to evaluate how costs savings are affected when considering participants with multiple characteristics of risk.

When considering prevention of LBW outcomes, the estimated cost savings associated with hospital costs avoided at birth were $89 per infant ($233.35 among infants of teen mothers) and $645 per infant with regard to avoidance of illnesses. For the 17,689 CPNP participants included in the costing analysis, the avoidance of LBW outcomes alone represents an average cost savings of $1.6 million. Recall that these comparisons were made between CPNP participants and all Canadian births as birth data were not readily available for a matched cohort of non-CPNP participants from the general public, a comparison which would undoubtedly yield far more dramatic savings. Savings on this scale for a program of this size is strong evidence that CPNP is achieving its intended outcomes for a good value.

Summary

This summative evaluation contains many compelling findings such as improved use of vitamin/mineral supplements, cessation of alcohol consumption, increased initiation and duration of breastfeeding as well as reduced likelihood of preterm birth and LBW outcomes among the infants of CPNP participants. This evaluation has also illustrated the cost-savings associated with good neonatal health achieved through healthy birth weights and breastfeeding. The positive impact of CPNP as described here should be considered while forming recommendations for the future of the program. Additional analyses should revisit positive findings – such as the relationship between the level of program exposure, type of service provided and birth outcomes – with the goal of identifying and expanding upon the most effective services. Certain counter-intuitive findings should also be explored in greater detail given that participant exposure to CPNP is clearly not the only factor influencing these results. These unexpected findings included:

• The association between greater program exposure and increased likelihood of (a) having a LGA baby and (b) gaining more than the recommended amount of weight during pregnancy.
• The absence of a relationship between the LGA babies and maternal weight gain above the recommended range.
• The 19% increased risk of preterm birth among those who received lifestyle counselling
• The decreased likelihood of alcohol cessation among women who received one-on-one nutrition or lifestyle education/counselling
• The increased likelihood to breastfeed for a shorter duration among women with high exposure to the program who reported “no income.”

Limitations identified in the methodology remain and must be weighed against the value and strength of any conclusions on CPNP relevance, impact, and cost-effectiveness. The aim of the recommendations in the following section is to address some of these unexpected results and limitations and provide guidance for the future direction of the program.

7.0 Recommendations

Several recommendations flow from the findings of this report and will help to guide decisions on future programming and evaluation planning.

1. Continued Monitoring of Threats to Maternal, Infant, and Child Health

It is recommended that CPNP continue monitoring threats to maternal and infant health so as to inform implementation and enable the program to adapt and respond to emerging health priorities.

Ongoing review of threats to maternal, infant, and child health will be necessary to ensure children's policies, programs and initiatives, including CPNP, remain aligned with domestic and international commitments that address children's health and with emerging public health priorities.

Current priority areas identified in the Chief Public Health Officer’s 2009 Report on the State of Public Health in Canada include:

• Injury prevention;
• Childhood obesity;
• Abuse and neglect;
• Prenatal health risks;
• Socio-economic status; and
• Mental health.

The delivery model for CPNP or any similar investment should be adaptive enough to strengthen the response to ever evolving priority health issues as they emerge. Of particular interest and relevance to the federal leadership role, will be threats to those Canadians who experience disparities in health and a disproportionate burden of disease relative to the general population.

Tracking and analysing the ability of the delivery model to adapt to and to respond to emerging priorities should be part of an overall strategy to strengthen CPNP evaluation.

For example, existing data on weight gain during pregnancy and CPNP’s focus on healthy eating could be assessed relative to the costs of obesity, while data on maternal depression could be assessed in relation to the impact of social support and mental health promotion provided by CPNP. Costs associated with infant injuries could also be analysed in relation to the pre- and post-natal education provided by the program. It would be necessary to expand data collection in these areas to conduct these kinds of analyses. As well, continued support and enhancement of mechanisms for sharing information and best practices within and among regional infrastructures with responsibility for oversight of the contribution agreements will be essential to ensure the strength of the program nationally.

The information from the newly developed INET and the recently updated Welcome Card will be essential in determining the characteristics of the population reached by CPNP and the threats to health that affect this population.

2. Continue Data Collection to Support Planning, Performance Reporting, and Evaluation

It is recommended that the CPNP continue to collect data to support planning, performance reporting, and evaluation, including:

• Ongoing collection of the new Integrated National Evaluation Tool (INET) and Welcome Card;
• Exploration of other strategies to measure broader or longer-term outcomes; and
• Periodic review of tools to ensure the data they seek respect the program logic and are responsive to the information needs of the program.

One of the greatest strengths of the CPNP evaluation strategy has been a high degree of commitment to on-going performance measurement by funding recipients. The information that will be gathered from the newly developed INET will be essential to the assessment of performance measurement as previously assessed through the IPQ. This data will inform findings on program reach, implementation and design.

New questions in the INET (replacing the IPQ), will further strengthen national level performance measurement data and inform future analyses. Moreover, now that the Ontario Aboriginal projects have adopted the INET, a more comprehensive understanding of the program outside Quebec should emerge. A continuation of collecting program data, including participant outcomes, on a national scale in a rigorous fashion will help to improve the validity of future analyses as the need for proxy data will be reduced.

The inclusion of literature-based evidence into the program logic model would be useful to guide the development of performance measures. As the program moves forward, it will be necessary to conduct periodic reviews of existing data collection tools for clarity, validity and utility – ensuring that questions in the data collection instruments seek the best data that can be collected for the intended purpose.

Better outcome data is also required in order to ensure that the research and analysis suggested in the following section is actionable. Short-term outcomes have been well assessed using the data from the ICQ2 tool. However, with the discontinuation of this tool, it will be important to look to other strategies to measure broader or longer-term outcomes than have been assessed to date. To this end, suggestions for data collection include: case studies on a sample of CPNP projects; a snapshot census to collect outcome data; or systematic sampling. All would be beneficial initiatives to ensure necessary data is available to support further evaluation plans.

Another area for consideration is the future investment in a longitudinal study of CPNP. Each of these suggestions to strengthen data collection would require adequate resourcing and are dependant on decisions related to PHAC decisions on future investments in children’s health.

3. Undertake Further Study/Analysis to Identify Most Effective Delivery Models within the Overall Program Design

It is recommended that further study/analysis be undertaken to:

• Explore quantitative observations through qualitative health impact analysis;
• Identify a typology of promising delivery models within the overall program design;
• Determine relationships between maternal risk, program exposure, and outcomes; and
• Further develop economic assessment of impacts associated with the program.

A number of subjects for further study arose from the quantitative impact analysis conducted by Muhajarine (2009). Inconsistencies in some expected outcomes relative to levels of program exposure and services offered require additional data and further exploration.

Further study could help establish a typology of CPNP models of delivery, facilitate their classification, and determine the potential for causal relationships between CPNP and participant outcomes. Additionally, it would be valuable to determine whether there is a link between (a) the initial level of risk of the mother upon program entry, (b) the amount of contact she has with the program, and (c) the types of services she receives. This research will help inform conclusions that can be drawn around program exposure/services offered, particularly with respect to identifying which results that can be attributed to the program itself compared to the effects of previously existing conditions or personal circumstances.

Such analysis will help focus improvements to the program and, specifically, will identify which outcome areas require the greatest attention at individual project sites.

Costing analyses on various aspects of the program will also be important. For example, an economic assessment of the impact of improvements in maternal health practices experienced by CPNP participants would be highly relevant. Such an assessment could also identify how to best direct program activities by focusing on the positive health practices that have the greatest impact on optimal birth outcomes with the greatest cost efficiency. Analyses of this type, with a cost or outcome focus will help to address some of the discrepancies and unexpected results uncovered in this evaluation.

4. Consider Evaluation Findings in Decisions Regarding Future PHAC Investments in Maternal, Infant, and Child Health

It is recommended that all results of this Summative evaluation, including key findings, conclusions, and limitations of the analysis, be considered in the context of long-term decisions regarding future PHAC investments in child health.

Overall the results from this assessment of national evaluation data and commissioned reports on the areas of relevance, impact and cost-effectiveness of the program, demonstrate that CPNP plays an important role in improving maternal and infant health for vulnerable populations. The trends, key findings and conclusions put forward in this document must be considered when planning the future direction for this program and the strategy surrounding Canada’s role in the field of maternal and infant health.

Findings show that CPNP successfully reaches a large proportion of the at-risk populations within Canada and demonstrates a clear commitment to implementing a population health approach. Results also demonstrate that CPNP is providing enhanced support at the community level for pregnant women and their infants and families living in conditions of risk across a wide variety of outcomes. This summative evaluation highlights many compelling findings regarding improvements in participant health practices and birth outcomes following exposure to CPNP. This evaluation has also illustrated the cost-savings associated with good neonatal health achieved through healthy birth weights and breastfeeding. Collectively this evidence points to the continued relevance, positive impacts, and cost-effectiveness of CPNP and calls for continued federal support of community-based investments in maternal, infant and child health.

5. Consider Findings on the Economic Benefits of Early Intervention and Adequacy of Current CPNP Funding Levels in Decisions Regarding Future PHAC Investments in Maternal, Infant, and Child Health

It is recommended that consideration be given to the economic benefits of early intervention as well as a review the economic realities of current funding levels of CPNP to inform future PHAC investments in maternal, infant and child health.

For 13 years, CPNP has been providing community-based organizations with funding to support the health and well-being of pregnant women, their infants and families living in conditions of risk. The last upward adjustment to funding dates back to 1999.

Overall, there is a strong evidence base for the economic benefits related to investments in maternal and infant health. The costing analysis discussed in this report provided estimates of savings generated by participation in CPNP with respect to initiation of breastfeeding and avoidance of LBW. Cost savings were estimated to exceed $1.6 million for avoidance of LBW alone.

6. Position evaluation to meet the mandate of the 2009 Treasury Board Policy on Evaluation

It is recommended that any future evaluation of CPNP be positioned to meet the mandate of the 2009 Treasury Board Policy on Evaluation.

This Summative Evaluation of CPNP has demonstrated adherence to the 2001 Treasury Board Policy on Evaluation. Going forward, evaluations of CPNP should be rooted in the new 2009 Treasury Board Policy on Evaluation. This new policy will expand evaluation to 100% programmatic coverage and will require collaboration throughout PHAC to plan evaluation over a 5-year cycle. This acknowledges the need for greater synergy in evaluation efforts across maternal and child health programs. A future evaluation strategy should focus on fewer accountability-driven questions and continue to address broader evaluation priorities. This will enhance learning and respond to the comprehensive evaluation requirements of the new policy. In determining the most compelling evaluation issues for CPNP in the future, it is recommended that the following questions be considered:

• How are the evaluations of other PHAC investments in maternal and child health (i.e., CAPC, AHSUNC and FASD) linked with the evaluation of CPNP?
• What are the cross-cutting issues in maternal and child health that CPNP can support through evaluation?
• Can CPNP cluster evaluation strategies with other PHAC investments to illustrate collective impact?
• What are the most effective mechanisms for sharing evaluation findings and lessons learned with key stakeholders across the program, the Agency, and the Government of Canada?