ARCHIVED - Summative Evaluation of the Community Action Program for Children : 2004-2009

CONCLUSIONS & RECOMMENDATIONS

6.0 Conclusions

This Summative Evaluation provides evidence that CAPC continues to be relevant for Canada, that it reaches vulnerable children and their families living in conditions of risk, and contributes to their health and social development. The evidence also supports the cost-effectiveness of CAPC.

Limitations noted in Section 2.3 remain and must be weighed against the value and strength of any conclusions on CAPC relevance, impact, and cost-effectiveness. Recommendations for action to address and mitigate these limitations for the future are addressed in full in Section 7.0.

6.1 Relevance

The evidence demonstrates the on-going relevance of CAPC in two areas:

• Threats to children’s health in Canada; and
• Alignment with federal government objectives and international priorities for children.

CAPC continues to be a relevant program in Canada as many threats to child and family health in Canada persist. Many Canadian families live in conditions of risk that have been shown to have adverse impacts on the health and social development of their children. CAPC provides programs and services that respond to the needs of these families.

CAPC is also relevant in its contribution to the priorities of the Public Health Agency of Canada and the Government of Canada. CAPC is a viable investment in the Public Health Agency’s effort to reduce health disparities and strengthen public health, thereby contributing to the Government of Canada’s outcome of “healthy Canadians”. CAPC contributes to fulfilling the Government of Canada’s important international commitments to the United Nations Convention on the Rights of the Child and the Millennium Development Goals.

6.2 Impact

CAPC is achieving the outcomes set out in the CAPC RMAF. The findings are summarized under three broad areas:

• Reaching children (0-6 years) and their families living in conditions of risk;
• Contributing to healthy child development; and
• Implementing a population health approach.

Findings indicate that CAPC reached a high proportion of families living in conditions of risk. The majority of the parents and caregivers reached by CAPC are affected by multiple risks. For every risk criteria examined, the proportion of families participating in CAPC is higher than the proportion found in the general Canadian population. This early intervention is important as the research has shown that children growing up under these conditions often experience less than optimal development. For example, at-risk children have higher rates of poor health, developmental difficulties and social, cognitive and behavioural problems.

The synthesis of national/regional findings has shown that CAPC has had a positive impact on the health and social development of children in the following areas:

• Improved health and social development for children;
• Parental self-improvement;
• Increased parenting knowledge and skills; and
• Enhanced community capacity.

These findings are consistent with the research literature which shows that early childhood development initiatives like CAPC have been found to contribute to children’s general development, school readiness, and educational performance and employment prospects later in life.

This Summative evaluation has demonstrated that CAPC projects effectively employ a population health approach to mobilize communities in support of at-risk children and their families. CAPC projects established substantial community-level support by developing partnerships and collaboration and fostering opportunities for participant involvement.

6.3 Cost-effectiveness

Research indicates that preventing health problems early in life has one of the greatest potentials to reduce health inequities within a generation (WHO, 2007). The value of early prevention also can exceed the initial costs of programming, and is the most cost-effective way to promote human health and well-being (Bennett 2008; Heckman, 2002). An assessment of the economic impact of CAPC demonstrated that:

• CAPC projects have successfully leveraged significant additional funding and in-kind resources;
• CAPC operational costs per participant are within the mid-range of the World Bank’s assessment of the cost per participant for similar programs; and
• CAPC outcomes demonstrate potential cost savings which are linked to the overall impacts of CAPC on societal, health and social system costs.

7.0 Recommendations

Several recommendations flow from the findings of this report and will help guide decisions on future programming and evaluation planning.

Ongoing review of program priority issues will be necessary to ensure children’s programming, including CAPC, remains in line with domestic and international commitments that address children’s health. Programs also must ensure continued alignment with new and emerging public health priorities. The recommendations below will help to guide future strategies on children’s health within PHAC.

1) Continue monitoring the threats to child health:

It is recommended that CAPC continue monitoring threats to child health so as to inform implementation and enable the program to adapt and respond to emerging health priorities.

Threats to child health continue to exist in Canada. Ongoing review of threats to child health will be necessary to ensure children’s policies, programs and initiatives, including CAPC, remain aligned with domestic and international commitments that address children’s health and with persistent and emerging public health priorities.

Current priority areas identified as persistent or worsening in the Chief Public Health Officer’s 2009 Report on the State of Public Health in Canada include:

• Injury prevention;
• Childhood obesity;
• Abuse and neglect;
• Prenatal health risks;
• Socio-economic status; and
• Mental health.

The delivery model for CAPC or any similar investment should be adaptive enough to strengthen the response to ever evolving priority health issues as they emerge. Of particular interest and relevance to the federal leadership role, will be threats to those Canadians who experience disparities in health and a disproportionate burden of disease relative to the general population.

Tracking and analysing the ability of the delivery model to adapt to and to respond to emerging priorities should be part of an overall strategy to strengthen CAPC evaluation.

2) Continue data collection to support planning, performance reporting and evaluation:

It is recommended that the CAPC continue to collect ongoing performance measurement data through the revised Integrated National Evaluation Tool (INET).

One of the greatest strengths of the CAPC evaluation strategy has been the commitment to on-going performance measurement, which has provided national findings on program implementation, design and reach. The information from the newly developed Integrated National Evaluation Tool (INET) will be essential to continue to assess performance measurement previously assessed in the NPP. This data can provide key findings on program reach, implementation and the applicability of CAPC to respond to priority public health issues for children.

New questions in the INET will further strengthen national level performance measurement data and inform future analyses. Moreover, now that the Ontario Aboriginal projects have adopted the INET, a more comprehensive national understanding of the program can be tracked through on-going performance measurement.

As the program moves forward, it will be necessary to conduct periodic reviews of existing data collection tools for clarity, validity and utility – ensuring that questions in the data collection instruments seek the best data that can be collected for the intended purpose. Certain limitations noted in this report should be considered for the improvement of future data collection through the INET or the CAPC Snapshot Census. These include:

• The assessment of geographical and community reach. To minimize double counting, it is recommended that CAPC consider a comprehensive community definition for future data collection through the INET.
• The comparability of Snapshot participants to the general population could be improved. To ensure accurate comparability in the future, it is recommended that CAPC continue to review data collection on socio-demographic characteristics from Statistics Canada to ensure accurate comparability.
• Priorities such as injury prevention, mental health and childhood obesity should be tracked and monitored through performance measurement. An examination of how CAPC projects develop and deliver services to respond to these issues will provide an evidence base to position community-based programs to be responsive to the priority public health issues within PHAC.

3) Undertake further study to identify the most effective delivery models within the overall program design:

It is recommended that core program objectives are reviewed to inform the identification of a program typology of interventions and services. This would enhance the identification of best practices and the development of a national outcome evaluation strategy.

This Summative Evaluation has provided strong evidence on program implementation, reach and impact. However, limitations identified through this summative evaluation (see Section 2.3) illustrate the need to consider changes to strengthen both program and evaluation design. These recommendations could guide future decisions on CAPC and children’s health priorities for PHAC.

a) Review and Identify Core Objectives:

With consideration to these emerging priorities, it is recommended that a review of program objectives and the establishment of core program elements be completed for CAPC in the context of long-term decisions on future PHAC investments in children’s health.

Introducing some degree of homogeneity will facilitate the tracking and assessment of on-going performance measurement and evaluation of CAPC and strengthen the program’s ability to affect and measure systematic national change on Canada’s priority public health issues.

In consideration of identifying core program elements and reassessing objectives, it is recommended that the following questions be considered:

• Are the CAPC program objectives still relevant?
• What core program elements could be introduced nationally?
• What are the best practices in ECD programming?
• How could core program elements respond to the emerging priority public health issues for children including injury prevention, childhood obesity and mental health?

b) Review and Identify a Typology of Core Interventions and Services

One noted strength in CAPC is the diversity of project activities and supports to respond to community needs and the unique needs of those at-risk. However, this heterogeneous structure remains a challenge for national evaluation. Outside the scope of this Summative Evaluation was an assessment of whether a particular healthy child development outcome was any more or less attributable to a particular intervention or service (i.e., home visiting).

It is recommended that a typology of core activities and services be identified in the context of long-term decisions on future PHAC investments in children’s health. Building on a review of core objectives and elements, the identification of core activities and services would make it more feasible to identify best practices within CAPC nationally. Key questions for any future evaluation strategy could include:

• How do CAPC interventions and services align with best practices in ECD programming?
• What are the most effective interventions? Are specific CAPC interventions or services more attributable to healthy child development outcomes?
• How could core CAPC interventions and services respond to the priority public health issues for children including injury prevention, childhood obesity and mental health?
• Can CAPC cluster interventions or activities by type to support the design of a national outcome evaluation?

c) Develop a National Outcome Evaluation Strategy

The methodology for reviewing and analyzing outcomes in the national and regional evaluations was qualitative in nature and used a blend of principles of meta-evaluation. This is a strength of CAPC, as evaluation findings were extracted from various national data, supplemented by regional evaluation findings, and synthesized to provide an overall picture. However, measurements varied in each of the regional evaluations, making it difficult to confirm healthy child development outcomes, observed regionally, to the national level.

To reduce this limitation in the future, it is recommended that a standardized national level evaluation on the impact of CAPC be considered. Confirmation of program impacts at the national level requires measurement of participant-level outcomes measured systematically. Recommendations for the development of a national outcome evaluation include the following:

National and regional evaluations should be planned and executed under a common national evaluation framework. This will require comprehensive national/regional evaluation priority setting and planning.

• The future evaluation design should acknowledge the heterogeneity of the target population and the heterogeneity of project delivery in diverse regions. The identification of core program elements and a national typology of services will mitigate this consideration.
• It is recommended that the strategy review long-term longitudinal design, comparison group and mixed-method (qualitative and quantitative) approaches for applicability to CAPC. Strong consideration should also be given to reviewing existing, validated child outcome tools for applicability to CAPC. A national outcome evaluation may also include a component that would allow for a systematic cost-benefit analysis.
• The specific methodology for a systematic national impact evaluation should be designed in collaboration with key stakeholders in program accountability and delivery across PHAC.

Such an evaluation strategy would require a commitment of dedicated funds for implementation and be contingent on long-term decisions on program direction within PHAC.

4) Consider evaluation findings in decisions regarding future PHAC investments in children’s health:

It is recommended that all results of this Summative evaluation, including key findings, conclusions, and limitations of the analysis be considered in the context of long-term decisions on future PHAC investments in child health.

Overall, results from this synthesis of CAPC national and regional evaluation reports demonstrate that CAPC plays an important role in improving the health of Canadians. CAPC is an avenue through which Canada works toward its objectives of supporting the health and development of vulnerable children and families. Findings show that the majority of CAPC participants are families living in conditions of risk, and that CAPC demonstrates a clear commitment to implementing a population health approach. Results further demonstrate that CAPC is contributing to healthy child development in a wide variety of outcome areas by building community capacity, encouraging parent self-improvement and through the development of parenting knowledge and skills. Although long-term impact on healthy child development has not yet been systematically examined, results from an economic analysis indicate that CAPC is potentially generating sizeable economic returns for Canada. Collectively this evidence points to the continued relevance, positive impacts, and cost-effectiveness of CAPC and calls for continued federal support of community-based investments in child health.

5) Consider findings on the economic benefits of early intervention and adequacy of current CAPC funding levels in decisions regarding future PHAC investments in child health:

It is recommended that consideration be given to the economic benefits of early intervention as well as a review the economic realities of current CAPC funding levels to inform future PHAC investments in child health.

CAPC has been in place for approximately 16 years, providing community-based organizations with funding to support the health and well-being of children and families living in conditions of risk. The last upward adjustment to funding dates back to 1997.

Overall, there is a strong evidence-base for the economic benefits related to investments in early childhood health. Early interventions for at-risk children promote educational achievement, raise the quality of the workforce, reduce crime and welfare dependency and provide economic benefits evident throughout the lifecycle (Heckman, 2008).

6) Position evaluation to meet the mandate of the 2009 Treasury Board Policy on Evaluation:

It is recommended that any future evaluation of CAPC be positioned to meet the mandate of the 2009 Treasury Board Policy on Evaluation.

This Summative Evaluation of CAPC has demonstrated a commitment to the accountabilities as outlined in CAPC’s RMAF in adherence to the 2001 Treasury Board Policy on Evaluation. Going forward, CAPC evaluation (and PHAC evaluation strategies) should be rooted in the new 2009 Treasury Board Policy on Evaluation. This new policy will expand evaluation to 100% programmatic coverage and will require PHAC corporate evaluation planning over a 5-year cycle, integrating the need for greater synergy among evaluation efforts across maternal and child health programs. A future evaluation strategy should focus on fewer accountability-driven questions and continue to address broader evaluation priorities to enhance learning across PHAC and respond to the broader evaluation requirements of the new policy.

In determining the most compelling evaluation questions for CAPC in the future, it is recommended that the following be considered:

• How are the evaluations of other PHAC investments in maternal and child health (i.e., CPNP, AHSUNC and FASD) linked with the evaluation of CAPC?
• What are the cross-cutting issues in maternal and child health that CAPC can support through evaluation?
• Can CAPC cluster evaluation strategies with other PHAC investments to illustrate collective impact?
• What are the most effective mechanisms for sharing evaluation findings and lessons learned with key stakeholders inside and outside of the program, PHAC, and the Government of Canada?