ARCHIVED - Summative Evaluation of the Community Action Program for Children : 2004-2009
Background and Context
The origins of the Community Action Program for Children (CAPC) can be traced to 1990 when Canada, along with 71 other nations made a commitment in the United Nations World Summit for Children to invest in the well-being of vulnerable children. In response to this commitment, the Government of Canada, in 1992, initiated Brighter Futures: Canada’s Action Plan for Children, which included the five-year, $500 million Child Development Initiative (CDI). The CDI was based on evidence that poverty, low education and unemployment had a negative impact on healthy child development.
The CDI outlined four approaches to addressing the conditions of risk for children, including: prevention, promotion, protection and partnerships. CAPC was identified as a significant part of the partnership approach and is designed based on community development principles. Inherent within the program is a strong emphasis on partnerships, community capacity- building and other health promotion strategies that influence the broader determinants of health. CAPC, the largest of the CDI programs, was announced in 1992, created in 1993 and first implemented in 1994 by Health Canada.
CAPC is a federal initiative of the Public Health Agency of Canada (PHAC)Footnote 1. The program provides funding to community-based groups and coalitions to develop and deliver comprehensive, culturally-appropriate prevention and early intervention initiatives to promote the health and social development of young children (0-6 years) and their families facing conditions of risk.
CAPC is founded on the principle that communities are well positioned to recognize the needs of their children and have the capacity to draw together the resources to address those needs. CAPC projects often act as an entry point where families, that are geographically and socially isolated, are linked to the health system and to additional supports in the broader community.
Because community needs as well as local and regional resources are unique, CAPC projects are as diverse as the regions in which they are locatedFootnote 2. Some activities include: drop-in childcare; school readiness programs for pre-schoolers; outreach and home visiting; nutritional support and collective kitchens; child development activities; cultural programs and celebrations; healthy physical activities; literacy development; and community capacity building. Currently, there are 441 CAPC projects operating in more than 3,000 communities across the country.
CAPC Objectives and Framework
The following six Guiding Principles inform program design, implementation, operations, governance and evaluation and are rooted in health promotion principles:
- Children First;
- Strengthening and Supporting Families;
- Equity and Accessibility;
- Partnerships and Collaboration; and
- Community-based Delivery.
The established objectives of CAPC are to:
- Improve the health and social development of children and their families;
- Increase partnerships and collaboration;
- Increase the number of effective community resources and programs;
- Increase recognition and support for communities at-risk, their needs, interests and rights;
- Increase empowerment and knowledge of families and communities; and
- Increase accessibility to culturally-appropriate and linguistically-sensitive programming.
The relationship between the activities implemented by CAPC and the short- and long-term outcomes are presented in the CAPC logic model, which was part of the Results-based Management Accountability Framework (RMAF). The CAPC logic model is provided in Annex-1. An exploration of CAPC program theory linking the principles of community-based interventions to at-risk populations is expanded upon in Section 2.4.
CAPC has been designed to serve children who are at the greatest risk of not reaching their potential. CAPC targets children living in low-income families; children living with teenage-parents; children experiencing developmental delays, social, emotional or behavioural problems; and abused and neglected children. Special consideration is given to Métis, Inuit and off-reserve First Nations children, children of recent immigrants and refugees, children in lone-parent families, and children who live in remote and isolated communities.
The total annual budget of CAPC is $59.5 million, of which $52.9 million goes directly to CAPC communities to fund local projects (Table 1). The balance of funds is allocated for operations and management, salaries and a national strategic fund that supports short-term projects. CAPC funding levels are determined on a Provincial/Territorial basis, which is partially influenced by the number of children aged 0 to 6 years in each Province/Territory. A breakdown of funding is provided in Table 1.
|Province/ Territory||Funding Allocation
||Number of CAPC Projects|
|Ontario Aboriginal (off reserve)Footnote 4||$5,112,300||44|
|Newfoundland and Labrador||$1,601,900||9|
|Prince Edward Island||$941,600||7|
Source: PHAC Division of Childhood and Adolescence (2008-2009)
CAPC is governed by administrative protocols signed at the Ministerial level and jointly managed by Federal and Provincial/Territorial governments through Joint Management Committees (JMCs) or Program Advisory Committees (PACs) based in each province and territory. The mandate of a JMC/PAC is to allocate resources, respond to jurisdictional priorities and develop strategies to address evolving issues related to program implementation. The protocols identify funding priorities and set out provincial/territorial terms and conditions for managing CAPC projects. The priorities and target groups identified within the protocols reflect the specific needs of each jurisdiction which in turn allows for a greater degree of flexibility in program delivery.
Within PHAC, a National Evaluation Team for Children (NETC) comprised of both national and regional evaluators, works collaboratively to support CAPC evaluation and performance monitoring. NETC is structured to include the perspectives and experience of the national and regional PHAC offices and is mandated to advise on evaluation of PHAC investments in children’s health. Members of NETC have been involved in the design, integration and implementation of performance measurement and evaluation processes at the national, regional and local levels to support policy, evaluation and research decisions related to CAPC. NETC links the national office to CAPC staff in regional offices, which in turn link directly to communities. This enables national and regional staff to respond to the diverse needs of community groups.
Purpose and Scope
Evaluation has been an integral aspect of program improvement and accountability for CAPC since its inception. Projects across Canada have participated in national and regional evaluation processes on an ongoing basis. Comprehensive evaluation and performance measurement commitments are outlined in CAPC’s Results-based Management and Accountability Framework (RMAF). These RMAF commitments are in adherence to the 2001 Treasury Board Policy on Evaluation. The commitments include the following:
- Ongoing Performance Measurement
- Formative Evaluation (completed 2007-2008)
- Summative Evaluation.
An assessment of the program design, delivery and reach was conducted through the Formative Evaluation of CAPC in 2007. The objective of the Formative Evaluation was to examine whether the CAPC program had been implemented as it was originally designed. More specifically, data from three comprehensive evaluation sourcesFootnote 5 were presented to illustrate both the extent to which the program was reaching families living in conditions of risk and the extent to which a population health approach was being applied. These findings helped to support the development of a new, integrated performance measurement system for CAPC and the Canada Prenatal Nutrition Program (CPNP). Updated findings on reach and design are included in this Summative Evaluation Report.
The Summative Evaluation Report provides essential program evaluation data on reach, implementation and impact for use by key stakeholders both inside and outside PHAC. These stakeholders include the PHAC Evaluation Committee, provincial and territorial partners (i.e., JMCs/PACs), community projects and partners, program participants, the health research community and the Canadian population at large.
Conclusions and recommendations address strengths and weaknesses of both the program and its evaluation to improve the continued delivery and on-going evaluation of CAPC: for example, the Summative Evaluation:
- assesses achievement of results against stated objectives of CAPC
- identifies continuing gaps, lessons learned and challenges related to PHAC’s mandate and the future of the program.
This Summative Evaluation also fulfills the accountability requirements of the program’s RMAF. The focus of the CAPC Summative Evaluation is on RMAF questions that address Program Relevance, Impact and Cost-effectiveness as outlined in the 2001 Treasury Board Policy on Evaluation. These questions are:
Q1 – To what extent do threats to children’s health persist?
Q2 – Is CAPC in line with current government objectives on children?
Q3 – To what extent is the program reaching families living in conditions of risk?
Q4 – How has the CAPC program contributed to healthy child development?
Q5 – To what extent is the program implementing a population health approach?
Q6 – To what extent has the program operated in a cost-efficient manner?
Q7 – To what extent has the CAPC program resulted in cost-savings / cost-avoidance to the health, social and justice system within Canada?
The scope of this evaluation has been defined by the three CAPC RMAF components (the evaluation criteria of relevance, impact, cost-effectiveness) and the seven questions pertaining to these components. Figure 1 illustrates the linkages between these questions and the immediate and intermediate outcomes described in the CAPC Logic Model (Annex-1).
The foundation for this Summative Evaluation is the national evaluation tools developed to collect data on an annual basis spanning the period of the program RMAF (2004-2009). The tools were designed to accomplish unique objectives with the over-arching goal of achieving comprehensive data collection and robust analyses. The strength of these tools and their linkages to one another laid the groundwork for the analyses that contributed to the findings on relevance, impact and cost-effectiveness of the program. These national tools are described below.
National Evaluation Tools
National Program Profile (2001-2006)
Tool: The CAPC National Program Profile (NPP) was designed to provide descriptive information on the implementation of CAPC and to assist PHAC in responding to reporting and accountability requirements. The NPP was administered in the form of a questionnaire on a fiscal-year cycle, beginning April 1st. Projects reported on activities retrospectively (for the fiscal year that had just ended). For example, cycle 7 of the NPP, which is a primary source of data referred to in this report, accounts for CAPC project activities during the period of April 1, 2005 to March 31, 2006.
NPP Project Success Story Form (2001-2006)
Tool: Success stories were generated as part of standardized annual reporting requirements for CAPC projects across Canada, which included the completion of a NPP. The NPP included a five-page document entitled "Do you have a story to tell?”. This voluntary form allowed projects to submit success stories directly from CAPC staff, participants, and community partners on the impact of their CAPC project over the previous fiscal cycle.
CAPC Snapshot Census (2006, 2008)
Tool: The CAPC Participant Snapshot Census was a one-month demographic survey of all children, parents and caregivers who participated in CAPC projects across Canada during November 2008 and previously in May 2006. The goal of the Snapshot project was to produce a national picture of CAPC participants and determine the extent to which the program was reaching its priority populations.
Recent perspectives on evaluations of health promotion initiatives emphasize the importance of using multiple methodologies and synthesizing findings from various approaches (McQueen, 2007; WHO, 1998). The overall methodology used for this summative evaluation has been a meta-evaluation which includes a review of multiple national/regional evaluations and survey reports, and the analysis and synthesis of data from these reports (Annex-2) to present findings on the evaluation of CAPC. A review of research literature was also completed to further substantiate the evidence underlying the linkages between CAPC outputs and outcomes, and the potential long-term impacts on children and their families living in conditions of risk. Such a review of other evaluations of this type falls into the field of meta-evaluation (UNDP, 2008; UNAIDS, 2008).
Table 2 outlines the national and regional evaluation reports used as lines of evidence in this Summative Report, including noted limitations for each as applicable. As well, additional research evidence is also outlined. These multiple lines of evidence provide both a strong basis for analyses and offer a diverse range of both qualitative and quantitative evaluation data for synthesis.
The CAPC program design is comprehensive and complex. The program design is based on well-established theory informed by extensive research in the areas of health promotion, child development, early intervention, and public health. This Summative Evaluation provides a review, synthesis, and analysis of data collected within the program from numerous national and regional evaluations and survey reports in order to present observations and evidence of a relationship between CAPC activities, outputs, and outcomes. This evaluation also incorporates the theory underpinning the program to develop a detailed examination of the linkages between the program activities and more distal, downstream health impacts. This theory is explored in greater detail in Section 2.4.
A breakdown of the methodology for each primary line of evidence for reporting on CAPC relevance, impact and cost-effectiveness is outlined below.
Relevance: Data from the 2006 Census were analyzed to describe the demographic profile of Canadians and demographic trends in Canada. Information was used to support the program relevance findings in this report. A comprehensive literature review comparing CAPC objectives and programming with current international, Government of Canada, and PHAC priorities was also conducted to assess program relevance.
Impact: This evaluation includes an analysis of the national data, collected through the CAPC Snapshot Census and the NPP, to synthesize findings for the reach and population health approach components of this report. Part of the analysis was completed during the Formative Evaluation and was updated with current data. A description of the Formative Evaluation is included in Table 2.
National qualitative data analysis, supplemented by regional evaluation findings, were synthesized to provide an overall picture of CAPC impact on healthy child development. Each national/regional CAPC evaluation had been completed using a variety of methods to collect quantitative and/or qualitative data. According to Bridie et al., (2005), an important methodology in the evaluation of health promotion initiatives is the synthesis of findings from a variety of different methodological approaches, including quasi-experimental designs, observational studies, and qualitative interviews and stories. Accordingly, this evaluation includes findings from regional evaluations as secondary lines of evidence within this report.Footnote 6 The outcomes that have been demonstrated nationally are supplemented with regional findings, which include both qualitative and quantitative impact evaluations, and in one case the utilization of a quasi-experimental design with a comparison group. Findings from the literature review were used to support and validate CAPC outcomes.
Cost-effectiveness: For the analysis of cost-effectiveness, cost per participant data were compared to other similar early childhood intervention studies. The data for comparative analysis was obtained from document and literature reviews. The diversity and share of various sources of funding (i.e., CAPC funding and other sources of leveraged funding obtained by projects) were examined to compare the CAPC cost per participant to overall cost per participant in similar programming. The amount of societal cost savings / cost avoidance were estimated by conducting sensitivity analyses using economic modeling established as a methodology through previous analysis. The 2009 Assessment of the Economic Impact of the CAPC reviewed and synthesized cost data to estimate the potential economic impact in terms of cost savings / cost avoidance for CAPC participants and the Canadian health, social and justice system.
In addition to the limitations associated with each line of evidence in Table 2, there are several overarching limitations are listed below along with solutions or considerations to address them. These limitations will be further examined in the Conclusions and Recommendations sections of this report.
|Limitation||Solution / Considerations|
|Lack of standardized national quantitative outcome data in the program datasets||Due to multiple factors outside the scope and funding of the evaluation of CAPC, the collection of national outcome data was not feasible for this Summative Evaluation. Meta-evaluation was employed as a method by which to strengthen and validate the qualitative analysis. This approach, supported by research literature, ensured that any qualitative findings were substantiated with regional findings and linked to the broader research in children’s health.|
|Difficulty in weighing the strengths of the methodologies employed in the regional evaluations||CAPC regional evaluations have consistently proven to be very innovative and allowed for regional distinctiveness in their implementation and design. While use of different methodologies in various regional evaluations is considered a strength for meta-evaluation, the fact that outcomes were measured differently in each of the regional evaluations makes it difficult to determine outcomes at the national level. In accordance with CEEPD guidelines, regional findings are situated as “secondary lines of evidence” introduced in those areas where the indicators of healthy child development were common across national/regional reports.|
|Lack of a comparator group for CAPC outcomes||Literature from international and Canadian sources has been used for comparison and validation, as well as the results from one regional quasi-experimental study that utilized a comparison group, which are highlighted throughout the impact section. A study that includes a comparison group at the national level would have provided further support to the findings and improve the generalisability to the Canadian population.|
|Lack of CAPC assessment of interventions and activities against program outcomes||CAPC is a program rooted in the principles of flexibility and community-based delivery. As a result, the activities and intervention strategies vary considerably across the country. Outside the scope of this Summative Evaluation, it was not feasible to assess program outcomes against type of service on a national scale or to establish a typology of core activities. This limitation is addressed in the recommendations section (7.0) on potential improvements to the national evaluation strategy for CAPC.|
The theoryFootnote 7 for CAPC program design is grounded in the science of child development, early intervention, and public health. The central tenet is that the health and development of the most vulnerable children can be protected from conditions of risk by investing in early intervention that addresses the needs of the whole family, thereby producing greater family stability, and improving the child’s context of development. Furthermore, the underlying philosophy of CAPC is that local agencies are in the best position to identify effective interventions as they are respected, and well-placed within their communities to know the unique conditions of their children and families.
CAPC exists to address the risks that place the development of some groups of children at a disadvantage. CAPC projects address the four central components of this child development theory described below, including the biological, social, family, and community contexts in which children develop.
Child Development Theory
Decades of research have made it clear that both the brain and new skills develop from the bottom up. Early childhood is the time for building of the brain’s foundation, framing and wiring. The first simple circuits that an infant forms are the base for, and contribute to, the formation of more complex circuits (Center on the Developing Child, 2007). If the early circuits governing basic skills are not sound, the child’s brain will be ill-equipped to construct new circuits relating to higher-level skills. Along with being the most critical circuits for later development, these early connections in the brain are also the most vulnerable to toxins in the environment.
Due to the fact that a healthy start is so critical to lifelong brain architecture upon which children’s social, emotional, and cognitive skills are built, supporting children starting in infancy is crucial to their later development. CAPC uses community-based intervention to promote young children’s well-being by addressing factors that place children at risk. This section provides an examination of CAPC in light of the biological forces that combine with a child’s individual experience, interactions with parents, and place within the community context.
a) Biological Forces
A large body of research demonstrates the critical importance of early development – from conception to school age – for brain development (Center on the Developing Child, 2007; Hertzman, 2000). These years are essential for learning reading, arithmetic and concomitantly adopting healthy social skills and cultural values, norms and beliefs. Biology interplays with the environment to guide children’s development in each domain; and some children and families are more at-risk than others. The Government of Canada (GoC) recognized that early investments in health would pay off richly for the futures of both individual children and society. The importance of early childhood experiences was emphasized in the federal, provincial and territorial Early Childhood Development Agreement (ECDA), which was signed in September 2000. The ECDA commits governments to improving children's health and to reporting regularly on child health outcomes. Thus, the main goal of CAPC is to contribute to the health of children 0 - 6 years of age and their families who are living in conditions of risk. CAPC “was conceived as a program to assist community groups in the establishment of services for children with particular needs, and attract partners from across the communities in the elaboration of services” (Synthesis of National and Regional Evaluations, 2009). National concern over the impact of social and biological forces on child development led to the birth of CAPC and continues to be addressed in CAPC programming.
Children need a wide range of stimuli—visual, verbal, social, emotional, and physical—to promote normal progression into adulthood. Research on young children with high-risk backgrounds suggests that such positive factors are associated with fewer behavioural problems later on (Appleyard et al, 2007).
Furthermore, negative experiences and sources of toxic stress such as neglect and abuse in the early years are likely to hinder individual development (McCain and Mustard, 1999). These related findings point to the importance of early intervention as the most effective way to promote continued health and development for children as they grow into adult community members. Child-focused early interventions have been linked to improvements in IQ and school performance and to reductions in school dropout rates, unemployment, delinquency, and criminal activity (National Research Council & Institute of Medicine, 2000; Heckman, 2008).
Accordingly, CAPC incorporates child-centered programming into the community-based intervention model. By providing young children with stimulating activities, interaction with other children, and child-level services, CAPC aims to reduce emotional-behavioural problems and enhance motor and social development of children. Moreover, each of these outcomes is an important component of young children’s readiness to learn once they reach elementary school (Lewit & Baker, 1995; Thompson & Raikes, 2007).
Parents and families are front and centre in the child’s life from day one. Consequently, they play a monumental role in building the foundation for the brain’s construction (Center for the Developing Child, 2007). Healthy and positive interactions with adults are central to children’s ability to grow and mature (Hertzman, 2000; Siegel, 1999). This means that early experiences of social-emotional nurturing and exploration are intricately linked to early brain development (Nelson, 2000; Siegel, 1999) and can actually protect children against the adverse effects of other social and biological toxins in their environments.
CAPC aims to promote child health through activities focusing on quality parenting. Children need responsive, consistent and nurturing parenting to grow physically, cognitively, socially, and emotionally (Hertzman, 2000). Children who experience high quality care from their parents during the early years fare better than those with lower quality home environments (Hertzman, 2000; McCain et al, 2007).
When parents experience personal difficulties, they are often unable to provide optimal care for their children; for example, maternal depression may compromise parenting with negative effects on children’s well-being and is associated with atypical brain development in young children (Center for the Developing Child, 2007; Cummings & Davies, 1994; Dawson et al, 1992). Families who experience poverty, lower levels of education, unemployment or underemployment are also more likely than their peers to have children who experience developmental problems and who are exposed to unsafe social settings (Black & Mittlemark, 1999). Other family characteristics that relate adversely to children’s health include single parenthood, psychological distress, substance abuse, and teen parenting, as well as low levels of social support and parental education. Many of these family characteristics impact parent-child interactions and expose the child to toxic stress which can erode the brain’s foundation (Conger et al, 2002; National Research Council & Institute of Medicine, 2000).
Based on these findings, CAPC works to enhance parenting skills of at-risk families. For example, some CAPC projects provide parents with information about effective parenting and demonstrate positive parenting models. Research shows that providing these types of services to parents of young children at-risk for later difficulties can improve children’s developmental outcomes (Brooks-Gunn, 2000).
Since communities provide a context within which parents raise their children, it is not surprising that characteristics of communities can impact children’s development; for example, children living in dangerous or unhealthy communities have higher rates of disease, injuries, violence, and maltreatment and have fewer opportunities to play outdoors and to engage with their environments (Duncan & Brooks-Gunn, 1997; Earls & Buka, 2000). Community danger and poverty can also lessen the quality of parent-child interactions (Duncan & Brooks-Gunn, 1997; Furstenberg, 1998). All these factors minimize the support that the developing brain needs, while producing toxic stress that hampers its ability to form a solid foundation for further development (Center for the Developing Child, 2007, Hertzman & Irwin, 2007).
Taking into account these factors, the originators of CAPC referred to the program as a community development approach that would improve the health and functioning of children living in conditions of risk, such as poverty, which are related to poor health outcomes in childhood and over the life course (Black & Mittlemark, 1999; Scott, 2002). CAPC uses a targeted rather than a universal approach to remove barriers and enhance opportunities for participation for disadvantaged groups. Thus, targeted programming is a means of achieving equity and reducing disparities that stem from socio-economic forces within the community. This approach is central given that such disparities can have negative health consequences for children.
The CAPC model builds up communities by encouraging collaborative projects that mobilize community resources to improve parenting skills and children’s health. Leading international and national research identify community-based models as being most effective in delivering services to children and their families, in part because of the community’s understanding of its own culture and specific needs (Hertzman, 2000; McCain et al, 2000; Wandermann & Florin, 2003). CAPC capitalizes on the support that communities can offer by making available high-quality projects, opportunities for healthy interaction, stimulation and learning, all of which can facilitate young children's early development. Furthermore, CAPC projects tend to be located in communities where there are concentrated populations of people at risk, such as neighbourhoods with low-income residents, few community resources and programs, and low levels of social capital. Thus, in keeping with the goal of achieving equity, CAPC delivers its programming to communities in greatest need of that aid.
Report a problem or mistake on this page
- Date modified: