Formative Evaluation: Understanding the Early Years Initiative, June 2009

Official Title: Formative Evaluation of the Understanding the Early Years Initiative - June 2009

Executive Summary

This report presents the findings of the formative evaluation of the Understanding the Early Years (UEY) Initiative. The evaluation was conducted to examine issues of implementation and design, early progress in achieving immediate objectives, and issues related to accountability. The evaluation team was also asked to provide preliminary guidance on the design of a summative evaluation of the Initiative.

Program Description

The UEY Initiative is delivered by Human Resources and Skills Development Canada (HRSDC) to help communities generate and use local information about their children to:

  • identify gaps in services and programs for young children and their families, and
  • foster partnerships among community groups and individuals to make informed decisions about the best programs and services for young children to thrive.

The UEY Initiative was announced in 2004 as a seven-year national initiative (2003-04 to 2010-11) to support up to 100 communities across Canada. It provides three years of contribution funding to sponsoring organizations, which act in concert with their communities to use UEY research tools to collect local information on:

  • the development of kindergarten children;
  • family and community factors that influence child development;
  • programs and services for young children and their families; and
  • local socio-economic characteristics.

This information allows members of funded communities to work together to identify gaps in services and programs for young children and their families and to develop a Community Action Plan to guide community decisions on how best to support families with young children.

Evaluation Methodologies and Limitations

The formative evaluation focused on the 21 communities selected for UEY funding through the 2005 Call for Proposals and was conducted approximately half way into the three-year funding agreements with these communities. The approach developed for the evaluation was based on the use of multiple lines of evidence drawn from three main sources: a review of documents and files (including a review of the administrative files for each of the 21 UEY sites); a literature review; and key informant interviews with 44 individuals (program management and staff (n=6), provincial/territorial government representatives and experts in childhood development (n=5), UEY community representatives (n=24), non-funded applicants (n=5), and others (n=4)). Findings from the Early Implementation Review were also used to augment the formative evaluation. Footnote 1

Combined, these sources provide a useful information base from which to draw findings on the overall development of the UEYInitiative. Certain limitations should be noted, however. For example, staff turnover within UEY communities meant that it was sometimes difficult to reach the most suitable key informants. Another potential limitation is that key informant interviews were conducted with representation from only ten of the 21 UEY sites. The effects of this limitation were minimized, however, because the majority of data collection focused on all 21 sites (e.g. the file and document review and the other key informant interviews).

Evaluation Findings on Issues of Design and Implementation

i) UEY Objectives

The document and file review indicated that the mandate, objectives and intended outcomes of the UEY Initiative are consistently communicated . This finding is consistent with the Early Implementation Review, which concluded that the various elements of strategic planning for the Initiative - including mandate, objectives, and intended outcomes - support the vision, goals and priorities of the department.

The focus and measurability of some of the Initiative's intended outcomes could be improved, particularly in the case of the longer-term and ultimate outcomes . Program officials who were key informants had mixed opinions on whether the Initiative's outcomes are well-defined and achievable. For example, a few felt that the ultimate goal (i.e. strengthened Canadian families with children by providing choice and support that recognizes the diversity of their needs) is very broad and should be more focused.

Key informants indicated that participants appear to have a clear understanding of their project's immediate objectives and expected results for the UEY Initiative . The file review and key informant interviews with program officials indicated that all 21 UEY communities have made progress in implementing their project and were making progress towards their immediate objectives and expected results. Although some early difficulties were experienced by most sites, the early difficulties were largely a result of the challenges/characteristics within individual communities (e.g. buy-in challenges and staffing) rather than a result of specific flaws in UEY design .

ii) Flexibility and Project Selection

Evidence from the document and file review and key informant interviews indicated that the UEY model is inclusive of a broad range of communities and is sufficiently flexible to accommodate their participation. For example, the 21 UEY sites selected through the 2005 Call for Proposals are very diverse in terms of population density, socio-economic status, ethno-cultural composition and languages.

Looking at the process used to select projects for UEY funding, the Early Implementation Review concluded that the assessments conducted by provincial/territorial government representatives and early childhood development experts are intended to ensure objectivity in decision-making and minimize the risk of error, abuse or fraud. The formative evaluation found that the UEY proposal review process is generally considered to work well and to adequately include participation of regional representatives and experts. Key informants familiar with the process noted a good balance between UEY staff and provincial/territorial representatives. The experience of non-funded applicants suggested, however, that the mandatory requirements (which are intended to ensure the capacity of the community to be a UEY site and the integrity of the data) were not consistently well understood by all applicants.

iii) Roles and Responsibilities

The evidence from key informants indicated that the roles and responsibilities of key players in the UEY Initiative are well understood and have been implemented as intended , although clarification was provided to respond to some initial challenges.

iv) UEY Design and Key Research Methods

The key strengths of the UEY design are considered to include collaboration, community focus, credibility and communication. The main design concern identified by key informants was a lack of sufficient linkage between two of the main sets of research data. Improvements suggested for UEY design focused on improving or refining processes, rather than on a redesign of the initiative (e.g. improving coordination with provincial/territorial programs and projects). Many of the suggestions have been a focus of the UEY Initiative from the beginning.

The UEY Initiative uses three data collection methods: the Early Development Instrument (EDI), the Parent Interviews and Direct Assessments of Children Survey (PIDACS), and data collection and analysis for the Community Mapping Report.

Key informants indicated that a key strength of EDI is that it is a standardized instrument that is nationally used and recognized. Most of the challenges/limitations identified for the EDI were related to the EDI process (e.g. certain aspects of communication, reporting and data sharing), rather than related to the EDI instrument. The suggested improvements were largely related to these challenges.

Similarly, a key strength of the PIDACS is that it is a validated, standardized instrument based on the National Longitudinal Survey of Children and Youth. Another key strength is that the instrument can be tailored and customized to accommodate the differences across UEY sites. The most frequently cited challenge was the delayed implementation of the PIDACS. A particular concern was that, because of the delay, the cohort for the PIDACS data is different from the cohort for the EDI data. Some of the suggested improvements for the PIDACS tended to be similar to the suggestions made for the EDI. Other suggestions were process changes specifically related to the PIDACS (e.g. adding further flexibility to accommodate differences across communities, and hiring local/known people as PIDACS administrators).

Key strengths of the Community Mapping Report are the provision of useful data and information, and the development of local capacity to undertake research. In this case, challenges/limitations related to capacity issues within the community (i.e. a limited capacity to develop the report) and challenges in obtaining relevant data. A few key informants suggested the community mapping exercise could be improved by encouraging information sharing among UEY communities and by providing more/better guidance and assistance when communities are preparing the Mapping Report.

The document review indicated that the UEY Initiative has been pro-active in responding to challenges/limitations in design and delivery . This included working to resolve process-related issues in the areas of planning/delays, clarity/guidance/tools and communication.

Progress Toward Immediate Objectives

i) Increased Support for Communities

Evidence from key informants and the file review suggests that progress has been made in providing communities with funding support. Although it is too early to assess the ability to attract additional support, the file review indicated that the UEY Initiative has enabled community coalitions to identify priorities for grants, incorporate UEY into funding proposals, and apply for and secure funding. The evidence also indicated that progress has been made in providing communities with technical assistance , with community mapping being cited as a main catalyst.

ii) Increased Availability and Sharing of Information

Both the project file review and key informants revealed that UEY communities have been successful at forming or strengthening partnerships as a result of the Initiative. As well, evidence from the document and file review and key informants suggests that the Initiative has increased the number of opportunities and tools for knowledge sharing and networking within communities on the role of families and communities in child development.

iii) Increased Awareness Regarding Child Development

Most key informants who were program officials felt there has been progress in terms of increasing parental and community awareness and knowledge regarding child development , although it was too early for this progress to be fully evident. Cited examples of progress included the creation of brochures and presentations at workshops.

Issues Related to Accountability

Despite the array of provincial programs aimed at children, the program scan conducted for the formative evaluation found no evidence of overlap or duplication with the UEY Initiative. In addition, a number of safeguards have been put in place to avoid or reduce the potential for overlap and duplication (e.g. UEY officials provide program updates at the semi-annual meetings of the Federal/Provincial/ Territorial Committee on Early Childhood Development Knowledge Information and Effective Practices).

The general view among key informants is that the results-based reporting strategies for the UEY Initiative are appropriate and compatible with federal government accountability requirements. Most key informants who were community representatives expressed confidence in their community's capacity to undertake the necessary reporting, although most also expressed the desire for more clarity and feedback from HRSDC to support reporting on the results of data collection. All key informants who were program officials felt that the performance monitoring reports of UEY sites were being sent in a timely manner. The file review found, however, that two-thirds of project files were missing at least one quarterly report.

Some Preliminary Guidance for the Summative Evaluation

A number of features of the UEY Initiative (e.g. a multi-site funding framework with community-level interventions) were identified as important considerations when determining how best to conduct a summative evaluation. Taking into account these features and the uniqueness of each UEY community, the evaluation team suggested that a theory of change approach Footnote 2 would be the most appropriate for the summative evaluation.

There is evidence that not all of the baseline data necessary for the summative evaluation is being collected consistently. For example, there is a need to define more clearly the information/data being requested from UEY sites (e.g. to clarify what is meant by partnerships). As well, measures identified in the Performance Measurement Framework for the UEY Initiative should ideally be designed to be on-going, rather than providing a single snapshot.

Overall Conclusions

The formative evaluation found the design of the UEY Initiative to be generally sound. For example, the evidence showed that the UEY model is inclusive, as well as sufficiently flexible to allow for the participation of a broad range of communities. Collaboration, community focus, credibility and communication were cited as key strengths of the UEY design.

The evaluation found that the EDI and PIDAC are viewed positively as standardized tools that are recognized nationally. Key strengths of the Community Mapping Report include the ability to provide useful data and to develop local capacity to undertake research.

The evaluation found that participants appear to have a clear understanding of their project's immediate and expected results for the UEY Initiative and have made progress in implementing their project. The early evidence suggests that progress has been made towards providing the community with funding support and technical assistance. As well, there is evidence that the UEY communities are forming and strengthening partnerships. There is also evidence of progress in terms of increasing parental and community awareness and knowledge regarding child development.

The following areas were identified for consideration/improvement:

  • Some of UEY's long-term objectives may be too broad relative to the scope and size of the UEY Initiative. These objectives should be clarified or modified to better link them to the resources and scope of the UEY Initiative.
  • The UEY Initiative is working to resolve process-related issues (e.g. related to planning, delays) and this work should continue.
  • Consider streamlining parent interviews.
  • The linkage of research data from EDI and PIDACS may better enable research on possible relationships between early childhood outcomes and a number of family and community factors. As well, there is a need for program officials to better communicate with community coordinators and others in the community about the cohort used for the PIDACS and the usefulness of these data. The continued concern expressed by community representatives in this area indicates the need to validate the PIDACS results once they are all available.
  • It would be helpful to further assist communities in completing the Community Mapping report. This could be accomplished in different ways (e.g. additional in-person training sessions, an on-line training module or a helpline).
  • To better support the summative evaluation, the Initiative needs to consistently collect data at the community level on how knowledge gained within the UEY communities is being applied regarding child development. Guidance will need to be provided to the communities on how best to collect these data.
  • To better support the summative evaluation, the UEY Initiative and the UEY sites should ensure that outcomes are linked to measurable indicators and that the necessary data can, and are, consistently collected. Consideration should be given to improving the activity report template to better facilitate the quantification and analysis of UEY project progress/performance. As well, there is a need for improved quantitative baseline data for indicators of intermediate and longer-term outcomes.

Management Response

Introduction

The purpose of this management response is to address the areas for program consideration/improvement, identified through the formative evaluation of the Understanding the Early Years (UEY) Initiative. The management response provides HRSDC officials with the opportunity to communicate their response to the key evaluation findings, to indicate where program policies have been modified, and to identify plans for further change.

The formative evaluation was undertaken by HRSDC to fulfill Government of Canada accountability requirements, and to provide HRSDC with considerations for management and program policy design decisions. The evaluation examined issues of implementation and design, early progress in achieving immediate objectives, and issues related to accountability.

The evaluation focused on the 21 UEY community projects selected for funding through the 2005 Call for Proposals, and was conducted approximately half way into the three-year funding agreements which ended in late 2008. The evaluation results are being used to inform the delivery of the final set of 15 UEY projects, which commenced in spring 2007 and are scheduled to end in March 2010. The UEY Initiative sunsets in 2010-2011.

Key Findings

The evaluation findings indicate that the UEY design is sound and that progress towards full implementation in the 2005 projects was being made. For example, the Initiative's mandate, objectives and intended outcomes have been consistently communicated and support the goals and priorities of the Department.

The evidence also shows that the UEY model is inclusive, as well as sufficiently flexible to allow for the participation of a broad range of communities. Collaboration, community focus, credibility and communication are cited as key strengths of the Initiative.

The evaluation identified seven considerations/improvements for the UEY Initiative that required action on the part of the program area, which are outlined below with the corresponding program response.

Areas for Consideration/Improvement and Actions Taken

1. Some of UEY's long-term objectives may be too broad relative to the scope and size of the UEY Initiative. These objectives should be clarified or modified to better link them to the resources and scope of the UEY Initiative.

The program agrees with this recommendation:

  • Departmental officials undertook a redesign of the Initiative's Integrated Accountability, Audit and Risk Framework (RMAF/RBAF), performance measurement framework, evaluation strategy, risk assessment and audit requirements.
  • The Initiative's long-term objectives have been modified to demonstrate a clearer link to resources and scope of the UEY Initiative.
    • The long-term outcomes, "increased accountability of social policies and programs through the incorporation of local knowledge and data" and "enhanced support system within communities based on successful and sustainable community initiatives" have been revised to state, "Inclusive communities that are responsive to the needs of children and families."

2. The UEY Initiative is working to resolve process-related issues (e.g. related to planning and delays) and this work should continue.

The program agrees with this recommendation. HRSDC officials have been pro-active in responding to process-related issues with respect to planning and delays in the two data collections, clarity of some data collection tools and communication between HRSDC and projects and will ensure that this continues:

  • Both UEY data collections, the Parent Interviews and Direct Assessments of Children Survey (PIDACS) and the Early Development Instrument (EDI), have been moved to the second year of the project life cycle to allow recipient organizations' time to develop relationships within the community and to work with their community partners to ensure that the project's objectives and research activities are understood. The inventory of programs and services has been moved to the first year of the project life cycle to enable the project coordinator to develop relationships with key service providers in the community.
  • HRSDC officials have worked with Access to Information and Privacy (ATIP) representatives to develop an explanatory letter and consent form for parents, to better describe the PIDACS data collection and obtain consent for their child to complete the direct assessment activities. In consultation with ATIP, a data sharing process has also been developed to allow recipient organizations to request their EDI and PIDACS project data files, for use under the conditions of the agreement.
  • Communication with project recipients has been greatly enhanced. In 2007, HRSDC officials met individually with the final set of UEY projects to familiarize them with HRSDC processes and requirements, and to enable the organizations to better plan their budgets and workplans. These site visits were also used as an opportunity to discuss the school boards' key role in the data collections and the importance of transmitting information early in the project life cycle. Two information sessions were held in Ottawa to inform the 2007 projects about the UEY research; one session was held before the data collections began so that the projects would better understand the process and their role.

3. Consider streamlining parent interviews.

The program agrees with this recommendation:

  • HRSDC officials reviewed and streamlined the parent questionnaire for the PIDACS data collection in the 2007 UEY community projects. This process took into account feedback received from the 2005 UEY community projects. The length of the parent interview has been shortened from 90 minutes to 60 minutes on average.

4. The linkage of research data from EDI and PIDACS may better enable research on possible relationships between early childhood outcomes and a number of family and community factors. As well, there is a need for program officials to better communicate with community coordinators and others in the community about the cohort used for the PIDACS and the usefulness of these data. The continued concern expressed by community representatives in this area indicates the need to validate the PIDACS results once they are all available.

The program agrees with the recommendation regarding the linkage of the EDI and PIDACS data. HRSDC officials may incorporate this element into the design of future initiatives to enable research on the possible relationships between developmental outcomes assessed by the EDI and family and community factors examined by the PIDACS.

  • In the context of the current national UEY Initiative, data linkage was not part of the program design and both data sets are valuable to communities in their own right.
  • The EDI and PIDACS data for the 2005 projects cannot be linked because they were collected for two different cohorts of kindergarten children (2006 and 2007). The data collected for the 2007 UEY projects will not be linked as HRSDC will not be a recipient of the EDI data. The community projects are acquiring their EDI data through direct contracts with the organization responsible for the collection and analysis or through their provincial EDI collections.
  • Taken together, without linking, the different perspectives on young children, and their family life provided by the EDI and PIDACS data make the results rich. The EDI provides kindergarten teachers' assessments of children's outcomes in major developmental areas, while PIDACS makes available the results of three direct assessments of children's cognitive development and parents' perspectives on their children's well-being. PIDACS also provides parents' perspectives on a range of family characteristics and family and child experiences.

The program agrees with the recommendation about the need for better communication about the cohort used for the PIDACS and its usefulness.

  • Departmental officials and research contractors met directly with representatives of each community project to explain that the PIDACS data, collected on a different cohort of children than that for the EDI, does not compromise the objectives of these two surveys: to find out how young children in a UEY community are faring in major developmental domains such as physical health, cognitive development and social behaviours (EDI), and to better understand characteristics of their families and experiences (PIDACS).
  • The community research reports produced by the PIDACS contractor based on both the PIDACS and EDI data, from two different cohorts of kindergarten children, presented a comprehensive picture of the young children growing up in a community. The reports were well received and useful to communities in their planning activities.
  • Departmental officials acknowledge that there may still be some degree of misunderstanding about the usefulness of the data given that it was collected on a different cohort of children. The data provided to the UEY projects are complex and, for most, it is the first time that communities have had access to such comprehensive and timely information about their young children and families. To support the 2007 projects, HRSDC officials hosted a two day session in January 2009, to further explain the research and how it can be used in UEY communities. Departmental officials will endeavour to help the projects to use the PIDACS data when it becomes available, notably through data sharing agreements.

The program agrees that the PIDACS data must be validated.

  • The PIDACS researcher validated the PIDACS data for the 2005 projects was representative of each community prior to writing each community's research report. The PIDACS data for the 2007 projects will be validated by checking that they represent the kindergarten population in each UEY community project. The same approach will be followed for the 2007 community projects.

5. It would be helpful to further assist communities in completing the Community Mapping Report. This could be accomplished in different ways (e.g. additional in-person training sessions, an on-line training module or a helpline).

The program agrees with this recommendation. HRSDC officials have undertaken the following activities to support UEY projects to accomplish this activity:

  • Introductory mapping workshops have been delivered to representatives from both the 2005 and 2007 UEY projects. Training resources included participant workbooks and a follow-up web cast for the 2007 community projects.
  • HRSDC officials have encouraged all UEY projects to recruit researchers who have some expertise using geographic information systems (GIS) software to map local data. As well, HRSDC officials with mapping expertise have been available to offer additional support.

6. To better support the summative evaluation, the Initiative needs to consistently collect data at the community level on how knowledge gained within the UEY communities is being applied regarding child development. Guidance will need to be provided to the communities on how best to collect these data.

The program agrees with this recommendation:

  • HRSDC officials have met directly with representatives from all of the UEY projects to provide an overview of the Initiative's desired outcomes and key indicators, to provide guidance on how these data could be collected at the community level, and to explain the activity and final reporting templates. As well, UEY HRSDC performance reporting expertise are available to offer additional support in determining methods for collecting and reporting on identified performance indicators.
  • The Initiative's intermediate outcome "increased awareness regarding child development" has been revised to state "families and communities understand the importance of children's development". Targeted indicators have also been developed as part of the Initiative's revised performance measurement framework.

7. To better support the summative evaluation, the UEY Initiative and the UEY sites should ensure that outcomes are linked to measurable indicators and that the necessary data can, and are, consistently collected. Consideration should be given to improving the activity report template to better facilitate the quantification and analysis of UEY project progress/performance. As well, there is a need for improved quantitative baseline data for indicators of intermediate and longer-term outcomes.

The program agrees with this recommendation:

  • As noted previously, departmental officials undertook a redesign of the Initiative's RMAF/RBAF, including the logic model, performance measurement framework, evaluation strategy, risk assessment and audit requirements.
  • To support the collection of baseline data on indicators to measure progress on intermediate and longer-term outcomes, HRSDC officials have developed a reporting system to ensure that projects' progress and reported results are being collected and linked to measurable indicators.
  • To better facilitate the quantification and analysis of UEY project progress and performance, HRSDC officials have revised both the quarterly activity and final report templates that are completed by each project.
    • Two interactive one-day workshops were held in 2008, to inform UEY project representatives about HRSDC's approach to performance measurement and evaluation, to ensure that project reports meet HRSDC requirements and at the same time are beneficial for the recipient organizations.
  • Departmental officials will be working with the Evaluation Directorate, to finalize the design of the summative evaluation and the quantitative indicators that will be employed in the summative evaluation to measure UEY's intermediate and long-term efforts or impacts.

Conclusion

The conclusions outlined in the formative evaluation include several positive findings as well as areas for improvement. These observations will provide sound advice for senior management and they have helped to inform implementation and management of the final set of 15 UEY projects which commenced in 2007. Changes have been made to the Initiative since the study period to address these observations, and HRSDC officials are committed to continuous improvement to the Initiative.

1. Program Overview and Evaluation Context

1.1 Purpose and Organization of the Report

This report presents the findings from the formative evaluation of the Understanding the Early Years (UEY) Initiative. The formative evaluation was undertaken to assess the development of the Initiative by examining issues of design and implementation, early progress in achieving results (i.e. progress towards achieving the immediate objectives), and issues related to accountability. This report also provides some preliminary guidance/suggestions on the design of a summative evaluation of the UEY Initiative. The information provided in this report consists of six sections:

  • Section 1 provides an overview of the UEY Initiative, plus the evaluation context and methodology;
  • Section 2 examines issues of design and implementation;
  • Section 3 examines progress toward immediate objectives;
  • Section 4 examines issues related to accountability;
  • Section 5 identifies some suggestions for the design of a summative evaluation;
  • Section 6 presents the main conclusions

1.2 Overview of the UEY Initiative

1.2.1 Program Description

The UEY Initiative helps communities gather and use local information about their children to:

  • identify gaps in services and programs for young children and their families; and
  • foster partnerships among community groups and individuals to make informed decisions about the best programs and services for young children to thrive.

The UEY Initiative was announced in 2004 as a seven-year national initiative (2004-05 to 2010-11) to support up to 100 communities across Canada. The Initiative built on the success and lessons learned in 12 UEY pilot projects, which began in 1999 and ended in 2007. The objective of the pilot projects was to enhance knowledge about community factors that influenced the early development of children.

The UEY Initiative provides three years of contribution funding to sponsoring organizations which act in concert with their communities to generate local information (as discussed in Section 1.2.3 and Section 1.2.4). The information allows members of funded communities to work together to identify gaps in services and programs for young children and their families, and to develop a Community Action Plan to guide community decisions on how best to support families with young children. The Initiative includes rural and urban areas and promotes the participation of communities with children from diverse backgrounds including First Nations, Inuit and Métis children, children in immigrant families and official language minority communities, and children in low-income families.

1.2.2 UEY Design and Delivery

The UEY Initiative is delivered through National Headquarters of Human Resources and Skills Development Canada (HRSDC) and is managed by the Community Development and Partnerships Directorate. The UEY team consists of project analysts who oversee the contribution agreements held by sponsoring organizations in the UEY communities, and research staff who are responsible for the management of two large data collections and related activities. Service Canada officials provide assistance with outreach activities and are part of the internal review of proposals submitted as part of the UEY community selection process.

UEY communities are selected through a Call for Proposals process. The submitted proposals are first screened to determine whether they meet the Initiative's mandatory requirements.

  • The applicant organization must be a legally incorporated Footnote 3 not-for-profit organization and be actively pursuing social development issues.
  • The proposed community must be located within a certain geographical location defined by boundaries understood by residents. The geography must be continuous or contiguous. People within these boundaries share a sense of belonging and identify with all, or parts, of the geographic community. Footnote 4 This requirement is designed to encourage ownership and community involvement around the children's early development.
  • The proposed community must have an existing community coalition with experience and a record of accomplishments in dealing with social issues. Footnote 5
  • Applicants are required to identify (in advance) all participating schools/school boards (or their equivalent) willing to participate, Footnote 6 because school boards play a key role in facilitating the collection of some of the local data.
  • Within the community, there must be suitable potential candidates (with appropriate skill sets and leadership qualities) for the UEY position of community coordinator.
  • The proposed community must have at least 300 five-year old children entering senior kindergarten (or equivalent) in the upcoming school year. Footnote 7

Proposals that meet the mandatory requirements undergo an internal assessment by National Headquarters of HRSDC and regional staff plus an external assessment by provincial government representatives. Following the internal and external assessment process and Ministerial approval, HRSDC enters into a three-year contribution agreement with each not-for-profit organization that has a successful application on behalf of its community. Each of these organizations becomes the project sponsor for the UEY community.

With the UEY funding, the sponsoring organization hires a community coordinator to manage all aspects of UEY. The funding may also be used to hire a community researcher to carry out the research on community programs and services and to interpret and map the research results. The sponsoring organization and community coordinator work with a UEY community coalition (which consists of parents, educators, service providers and others interested in the well-being of children). Together, they engage the community around the importance of early childhood development and the development of a Community Action Plan (as discussed in Section 2.1.3, Section 2.1.4 and Section 2.1.5). Also, the UEY community staff organize and coordinate information sharing and dissemination activities to promote UEY activities within their community and strengthen partnerships among the various stakeholders (e.g. parents, schools, service providers and the business sector) to address the needs of young children.

In the fall of 2005, 21 community projects across the country were selected to be UEY sites (see Appendix A). These projects are the subject of this evaluation. Under the three-year UEY contribution agreements, the UEY sites received on average between $350,000 and $375,000 in funding for their community project. As well, UEY communities received training and technical support (e.g. orientation meeting, community mapping workshop, EDI workshop). A second group of 15 communities is being added to the Initiative from the 2006 Call for Proposals.

1.2.3 UEY Research

UEY research generates several types of local information that is made available to each UEY project. The local information includes parents' and teachers' perspectives on the development of kindergarten children, direct assessments of kindergarten children's cognitive abilities, parents' perspectives on family circumstances and children's experiences, information on local programs and services, and local socio-economic characteristics. Each of these types of information is discussed in more detail below.

Teachers Complete the Early Development Instrument (EDI)

Kindergarten teachers complete the Early Development Instrument (EDI) for all children in senior kindergarten (or the equivalent) in schools participating in the UEY project.

  • The EDI is a population-based measure used to assess the development of kindergarten children as a group. It provides teacher ratings of children's behaviour and capacities in five areas: physical health and well being, emotional maturity, social competence, language and cognition, and communication and general knowledge.
  • The EDI was developed under the direction of Dr. M. Janus and Dr. Dan Offord at the Offord Centre for Child Studies at McMaster University. Under contract to HRSDC, the Offord Centre was responsible for the EDI assessments for the 21 UEY projects, including site-specific reports for each community.

Parent Interviews and Direct Assessments of Children Survey (PIDACS)

The Parent Interviews and Direct Assessments of Children Survey (PIDACS) has a number of components. The parent interview component provides information on family background and family processes, including socio-economic status, parenting style, and use of community resources. Parents also report on their kindergarten child's development and experiences, and activities at home and in the community. Direct child assessments components examine receptive vocabulary, early literacy and general cognitive skills, and early numeracy.

  • The instruments used by the PIDACS are based on those used by the National Longitudinal Survey of Children and Youth (NLSCY) since 2000. The instruments are administered to a sample of kindergarten children in each community and the person most knowledgeable about each child (usually a parent).
  • Under contract to HRSDC, R.A. Malatest and Associates was responsible for the PIDACS data collection in the 21 UEY projects, including a Community Research Report for each community.
  • A standard letter of information and consent form for parents was provided by HRSDC and used in each UEY community (with some local information added if applicable), to ensure that parents were adequately informed about the PIDACS research and consented to their local UEY project receiving a data file for use on behalf of young children in the community.

Inventory of Programs and Services

Each UEY project develops its own inventory of local programs and services available to young children and their families. Relevant programs and services include health clinics, schools, family resource centres, recreational facilities (including parks and pools), and child care facilities.

Socio-economic Data

Each project also compiles data on community socio-economic characteristics such as levels of employment, education and home ownership. This information is usually based on neighbourhood-specific data from the latest Census of Canada, but may also include local administrative information. HRSDC has provided 2001 Census data to sponsoring organizations on request through its licensing agreement with Statistics Canada.

1.2.4 Using and Disseminating Information

The Initiative's objectives include increasing the communities' understanding of how young children in the community are doing and how the community is supporting those children. Certain reports are developed by each UEY community to help use and share the results from its UEY research.

  • The Community Research Report presents findings on the development of kindergarten children as provided by their parents, teachers and the children. This report also presents information about families and the community provided by the parents. Each community-specific research report is developed by the research firm responsible for the PIDACS data collection.
  • The sponsoring organization in each funded community creates a Community Mapping Report to illustrate the UEY research findings for community members. The report includes maps that provide a "picture" of the relationships between community factors (e.g. location of parks, libraries and child care centres), community socio-economic characteristics (e.g. rates of employment and home ownership), and children's developmental outcomes (e.g. social competence). This information comes from the inventory of programs and services, neighbourhood-level Census data, and teacher data on children's development.
  • A Community Action Plan is developed by the sponsoring organization in collaboration with community organizations and individuals. The Action Plan is based on the findings from the UEY research. It identifies measures to improve services and programs for young children. It also identifies measures to encourage community members to work together to address the needs of young children. As well, it proposes ways to address gaps in community supports for children and serves as the community's blueprint for action.

The process of developing the Community Action Plan and the events and activities to disseminate the research information (to parents, service providers, educators and others) are used by the UEY sponsoring organization and community coalition to engage the community around ways to enhance the importance of early childhood development.

1.2.5 Key Players in a UEY Project

In addition to HRSDC, a variety of key players are involved in implementing a UEY project.

  • The community sponsor/sponsoring organization is responsible for the management and delivery of the project (as the responsible agent in the contribution agreement with HRSDC). The sponsoring organization has overall responsibility for project deliverables and results. It also has overall responsibility for providing financial and activity/progress reports and a final report to HRSDC. As well, it is also responsible for hiring the community coordinator and the community researcher (if one is hired).
  • The community coordinator manages all aspects of UEY-related activities within the UEY site and is responsible for working with the researchers (internal and independent) and with the community coalition. The community coordinator reports to the sponsor/sponsoring organization and is responsible for the completion of project deliverables, relationship building, and developing community interest in UEY activities and the importance of early childhood development. The coordinator works with the sponsoring organization, community researcher and community coalition to develop the evidence-based Community Action Plan, and is responsible for facilitating data collection, local research, and information sharing. The community coordinator also trains the teachers to participate in the EDI.
  • The community researcher Footnote 8 compiles the Inventory of Community Programs and Services, interprets data, liaises with data collection contractors on reports, and produces the Community Mapping Report.
  • The community coalition has an advisory role. It works with the community coordinator and sponsoring organization to understand the UEY data, engage the community, and develop the Community Action Plan.
  • The independent contractors collect the EDI and PIDACS data. They also produce the EDI reports, Community Research Reports, and the EDI and PIDACS data files.
  • Provincial and territorial officials, while not key players in a UEY project, participate in the external review of proposals and are involved in some community coalitions.
  • The independent contractors collect the EDI and PIDACS data. They also produce the EDI reports, Community Research Reports, and the EDI and PIDACS data files.
  • Provincial and territorial officials , while not key players in a UEY project, participate in the external review of proposals and are involved in some community coalitions.

1.3 Evaluation Scope and Methodology

The formative evaluation was focused on examining the development of the UEY Initiative by examining issues of implementation and design, progress in achieving immediate outcomes, Footnote 9 and issues related to accountability. In addition, the evaluation team was asked to provide preliminary guidance on the design of the summative evaluation.

The formative evaluation focused specifically on the 21 communities selected for UEY funding in 2005 (i.e., the 2005 round of UEY sites). It was conducted approximately half way into the three-year funding agreements with these communities.

1.3.1 Evaluation Approach

The approach developed for the formative evaluation was based on the use of multiple lines of evidence drawn from three main sources.

  • The review of documents and files included existing UEY documents and the administrative files for each of the 21 communities selected in 2005. The majority of reviewed documents were provided by HRSDC and included: background on the UEY Initiative, information on UEY data collection and support to sites, community proposals, contribution agreements, evaluation and results-based reports, and administrative files. Also included was the Early Implementation Review produced by HRSDC Internal Audit. Footnote 10
  • A literature review was conducted to assess how research and evaluations of similar programs could inform the UEY summative evaluation. As well, the literature review examined similar community-based programs in Canada and other countries to help identify ways to augment the UEY design, data collection and analysis methods. Also included was a review of recent research literature on the role of communities in fostering early childhood development. Footnote 11
  • In-depth interviews with key informants (n=44) were conducted in the spring and summer of 2007 (approximately half way into the three-year funding agreements with the 21 UEY communities) with representatives of the following groups:
    • UEY program management and staff (n=6);
    • independent contractors, including persons from firms working under contract with HRSDC who are involved in data collection and analysis (n=3);
    • an Access to Information and Privacy expert with knowledge of the UEY Initiative (n=1); Footnote 12
    • UEY community representatives (n=24) (i.e. UEY community sponsors/sponsoring organizations (n=9), UEY community coordinators (n=9), and UEY community coalition members (n=6));
    • community applicants whose UEY application was not funded (n=5); and
    • representatives from provincial and territorial government departments, and experts in the field of community influences on childhood development (n=5).

Ten of the 21 UEY communities were selected to be part of the key informant interviews. Footnote 13 These communities were selected to ensure representation of an urban/rural, large/small, region and ethnic mix of the population. All key informant interviews were conducted by telephone. Footnote 14

To provide relative weight to the findings presented throughout the evaluation report, a set of guidelines were used to describe how key informants responded to a given question:

  • None: no key informants expressed a particular view.
  • A few: one or two key informants have a particular view.
  • Some: between one-quarter and half of the key informants hold the same view.
  • Most: a majority (more than half) but not all key informants holds the same view.
  • All: all key informants expressed the same/similar views.

1.3.2 Methodological Challenges and Limitations

Although the combined use of these three sources of information provides a useful information base for the formative evaluation, certain limitations should be noted.

  • Although some data on outcomes was drawn from the file review process, these data should be treated with caution because some definitional challenges were encountered in aggregating the data across projects. For example, the reporting of quantitative data/information by UEY sites is, to some extent, open to each site’s interpretation of the indicators specified by the UEY Initiative. As well, problems in determining what constituted a partnership made it difficult to interpret community partnership data.
  • Due to some turnover among staff in the UEY communities, it was sometimes difficult to reach the most suitable key informants.
  • To ensure that one opinion did not unduly influence the findings, a ‘representative approach’ was used to quantify the key informant results. A limitation of this approach, however, is that it can minimize the views of individuals who may have unique knowledge of an issue (based on their specified area of expertise). Therefore the evaluation team made every effort to ensure that this type of unique knowledge was factored into the evaluation results.
  • Another potential limitation is that key informant interviews were conducted with representatives of only ten of the 21 UEY sites. The effects of this limitation were minimized, however, because the majority of data collection focused on all 21 UEY sites. For example, program files were reviewed from each of the 21 sites. As well, the other categories of key informants provided a broader perspective of the UEY Initiative.

2. Findings on Design and Implementation

This section examines issues in the following areas:

  • UEY objectives;
  • flexibility and project selection;
  • roles and responsibilities of UEY players;
  • the strengths, challenges and suggested improvements in the case of UEY design and
  • key research tools; and
  • lessons learned (specifically, what has worked well for the 2005 round of UEY sites,
  • where are changes needed, and what refinements have been made).

2.1 UEY Objectives

This part examines whether UEY objectives are clear and achievable. It also examines whether the 21 UEY sites have experienced any difficulties in understanding or making progress towards their project’s immediate objectives and expected results for the UEY Initiative.

2.1.1 Are UEY Objectives Clear and Achievable?

UEY program documentation indicates that the broad objectives of the UEY Initiative are: to strengthen the ability of communities to use quality local data to help them make decisions to enhance children’s lives; and to enable community members to work together to address the needs of children. The intended program outcomes (objectives) are shown in Table 2.1.

Table 2.1 - Summary of Intended Program Outcomes
Type of Outcome Intended Outcome
Immediate outcomes (objectives) anticipated within one year of implementation
  • increased support for communities
  • increased availability and sharing of information on the role of families and communities in child development
  • increase awareness and use of parent and community knowledge regarding child development
Intermediate outcomes (objectives) anticipated within two to three years of implementation
  • increased community ability to form partnerships across sectors torespond to child development issues
  • increased ability of communities to help parents make choices for their children based on local knowledge and data
Longer-term outcomes (objectives) anticipated in four or more years
  • enhanced support system within communities based on successful and sustainable community initiatives
  • increased accountability of social policies and programs through the incorporation of local knowledge and data
Ultimate goal of the UEY Initiative
  • strengthen Canadian families with children by providing choice and support that recognizes the diversity of their needs
  • Source: Document review.

Finding: The mandate, objectives and intended outcomes of the UEY Initiative are consistently communicated.

Findings from the Early Implementation Review concluded that the various elements of strategic planning for the UEY Initiative – including mandate, objectives, and intended outcomes – support the vision, goals and priorities of the department. Files and documents reviewed for the formative evaluation supported the conclusion that the mandate, objectives and intended outcomes of the UEY Initiative are all consistently communicated.

Finding: The focus and measurability of some of the Initiative’s intended outcomes could be improved, particularly in the case of the longer-term and ultimate outcomes.

Program officials interviewed as key informants had mixed opinions on whether the Initiative’s outcomes (immediate, intermediate and longer-term) are well-defined and achievable.

  • A few key informants felt that the longer-term outcome of “increased accountability of social policies and programs through the incorporation of local knowledge and data” is difficult to attribute to UEY, and therefore UEY cannot be expected to have a measurable impact in this area.
  • Similarly, a few key informants noted that the ultimate goal of “strengthened Canadian families with children by providing choice and support that recognizes the diversity of their needs” is very broad and should be more focused.

It should be noted that these comments related to improvements/refinements rather than a redesign of the Initiative. They are also consistent with revisions of the UEY logic model underway at the time of the formative evaluation.

2.1.2 Participants’ Understanding and Early Experience

Finding: Participants appear to have a clear understanding of their project’s immediate objectives and expected results for the UEY Initiative.

Key informant interviews with community representatives (i.e. community sponsors, coordinators and coalition members) and program officials provided evidence that each of the UEY sites has a clear understanding of the community’s immediate and expected results for the UEY Initiative. All community key informants were able to provide a description of their project’s objectives, which were reflective of the outcomes of the UEY Initiative.

Finding: The evidence indicated that all 21 UEY communities have made progress in implementing their project and were making progress towards their immediate objectives and expected results for the UEY Initiative.

The file review found that all 21 communities have made progress in implementing their UEY project and were making progress towards the immediate objectives and expected results. For example, at the time of the evaluation, all communities had completed their EDI data collection and had received the EDI results report from the independent contractor. All sites had developed information sharing products on UEY for their community, most communities had started or completed their Inventory of Programs and Services, and over half had initiated activities for the preparation of their Community Mapping Report.

Most key informants who were program officials believed that UEY communities should be able to make good progress on achieving their expected results. At the same time, however, some key informants noted that certain communities had made more progress than others.

Finding: Most sites experienced some early difficulties, which were largely the result of challenges within individual communities (e.g. buy-in challenges and staffing) rather than due to flaws in UEY design.

Although the 2005 round of UEY sites were found to be on-track to achieve their immediate outcomes, there was evidence that most sites experienced some early difficulties. These difficulties reflected the realities/challenges within each community rather than design or delivery problems with the Initiative itself. The following early challenges were identified by key informants and supported by the document and file review.

  • Geographic and/or demographic factors: some communities faced challenges associated with the composition of the population or their geographic location. For example, evidence shows that, due to language and cultural differences, incorporating immigrant and Aboriginal groups into the project can be challenging. Some sites reported that travel time and distances between communities posed a challenge for project implementation.
  • Collaboration of schools/school boards: a few projects experienced difficulties getting buy-in on the part of schools and/or school boards. Sometimes the schools were involved in a number of projects not directly related to the UEY Initiative, which resulted in overload and confusion for teachers. In other cases, the UEY project involved a number of schools with no single governing board. This meant community sponsors had to work through levels of bureaucracy or decision-makers for each school. In a few communities, schools or school boards with ethnic or multicultural populations were reluctant to participate in data collection processes (EDI and PIDACS). They did, however, participate in activities stemming from the initiative (e.g. school breakfast programs). Further details on the participation of schools/school boards are presented in Section 2.4.2 and Section 2.4.3.
  • Community buy-in: some community representatives noted that it was initially difficult to get community members and organizations to buy into the project when they did not understand the purpose or potential benefits of the UEY initiative. Engaging various types of stakeholders, such as profit and non-profit organizations and local politicians, was also identified as an early challenge.
  • Staffing: a few community representatives mentioned staffing challenges within their communities, such as staff turnover or finding the appropriate individuals with the necessary knowledge and skills (i.e. to be a community coordinator or community researcher). Some identified staff turnover at HRSDC as a challenge for communities because new HRSDC project analysts needed time to become familiar with UEY projects. Although some staff turnover should be anticipated, given the three-year duration of UEY projects, UEY management has been working to minimize the impact of HRSDC staff changes on communities.
  • Delays: were often cited as early challenges. Frequent examples included delays in collecting, receiving or sharing data. Other delays occurred because of difficulties in obtaining the collaboration of school boards, staffing challenges within the community (e.g. in hiring a community coordinator), or obtaining community buy-in. In some cases, the start of the project was stalled by a delay between the application stage and funding. This had further consequences when staff turnover during the waiting period resulted in a need to obtain replacements (causing additional delays). There was also a substantial delay in getting the PIDACS data collection started (as discussed Section 2.4.1 and Section 2.4.3).

It should be noted that the document and file review found that the selection process/Call for Proposals specifically screens for factors that are expected to strongly contribute to a community’s ability to successfully meet its UEY project objectives. The document and file review also revealed that the UEY’s Call for Proposals process encourages communities to anticipate and resolve some of these challenges at the application stage. For example, communities are required to plan and clearly identify the geographic boundaries of their UEY project (as noted in Section 1.2.2). Similarly, communities are required to provide evidence of community and school board buy-in and participation in the project. The experience of the 2005 round of UEY sites suggests that it would be useful to give additional consideration to identifying ways to alert potential applicants to the various types of early challenges and to inform them of solutions that have worked well in other UEY communities.

2.2 Flexibility and Project Selection

This section examines whether the UEY model is flexible in accommodating differences across communities and whether it is inclusive of diverse communities. This section also examines whether the proposal review process included adequate regional representation. It also examines the experience of non-funded applicants.

2.2.1 Flexibility and Inclusiveness of the Model

Finding: The UEY model is inclusive of the broad range of communities and is sufficiently flexible to accommodate their participation. Inclusiveness is balanced, however, against the Initiative’s mandatory requirements (which help to ensure that communities receiving UEY funding are able to successfully meet their objectives).

The file and document review indicated that the UEY Initiative is committed to providing funding to a broad spectrum of communities. The Call for Proposals was widely promoted across Canada, and communities had many opportunities to learn about the Initiative and submit a proposal. In addition, the criteria for funding encouraged the participation of a diverse range of communities. As well, proposals were screened with a view to achieving regional representation as much as possible, to ensure that urban and rural communities were among the successful candidates. Effort was also made to include children from marginalized or disadvantaged communities in the selected projects.

The profile of the 21 sites selected for the 2005 round of UEY funding indicates that these communities are very diverse in terms of their population density, socio-economic status, ethno-cultural composition, languages and number of new Canadians. Eight of the 21 sites are urban, six are rural and the other seven are mixed. Population information available for 16 of the 21 sites indicate that their populations range from 12,000 to 774,072 people. The regional distribution of the 21 sites is as follows: Ontario (seven sites); British Columbia (six sites); Nova Scotia (three sites); Quebec (two sites); and Saskatchewan, Manitoba and New Brunswick (one site each).

Program officials who were key informants were unanimous in their view that UEY is inclusive of diverse communities across the country, and most provincial/territorial government representatives and experts agreed. Community members who were key informants strongly supported the perceptions of program officials. For example, while some UEY communities, particularly rural sites, are described as being quite homogeneous, others are described as quite culturally and socially diverse. Similarly, some communities have a significant proportion of low-income families or single parent families, while others are quite affluent overall.

Most key informants from funded communities and key informants from non-funded applicants (i.e. communities that applied for, but did not receive UEY funding) agreed that the UEY model is inclusive of children and families of diverse backgrounds and circumstances. They illustrated this by describing the groups included in the Initiative (e.g. Aboriginal, Francophone, ethnic/minority, rural, lower income). Some community key informants also described a broad range of efforts undertaken by their community to ensure inclusiveness such as translation of data collection tools, using ethnic or multicultural media (i.e. not English or French) for communication, and conducting parent interviews in languages other than English or French.

Despite evidence of diversity, the Early Implementation Review (2007) identified some gaps in regional coverage in the 2005 Call for Proposals. The formative evaluation found that some program officials and provincial/territorial government representatives felt that more vulnerable communities and Aboriginal communities may be excluded because of not meeting the mandatory requirements. Footnote 15 This concern was also noted by some non-funded applicants (see Section 2.2.3). These key informants acknowledged, however, that communities not meeting the mandatory requirements of the Call for Proposals are unlikely to be able to successfully implement a UEY project.

2.2.2 Adequacy of Regional Participation in Selection Process

Finding: The proposal review process was generally considered to work well. The process was seen to adequately include the participation of regional representatives and experts, and to incorporate their feedback fairly and appropriately.

The Early Implementation Review concluded that the assessments conducted by provincial/territorial government representatives and regional early childhood development experts are intended to ensure objectivity in decision-making and minimize the risk of error, abuse or fraud.

The document review conducted for the formative evaluation indicated that the proposal review process used to select the 2005 round of UEY sites included internal and the external assessments performed by provincial government officials, Department staff and regional representatives. Footnote 16 During the assessment process, regional representatives of Service Canada and provincial/territorial government representatives contributed their knowledge of regional and local non-profit organizations that applied for UEY funding. Upon completion of the assessments, each proposal was assigned an overall weighted score (using a 60 percent weighting for internal assessments and a 40 percent weighting for external assessments). Finally, environmental and risk assessments of each proposal were conducted, and the recommendation rationale was prepared.

Both the proposal review process and the participation of regional representatives are viewed positively. Key informants familiar with the process (i.e. program officials, provincial/territorial government representatives and experts in early childhood development) noted there was a good balance between UEY staff and provincial/territorial staff, with each providing input within their areas of expertise. They also suggested that changes to the assessor review guide have made the process more user-friendly, have reduced duplication and are well aligned with the proposal. They also noted that the template for the proposal assessment is well thought out, and results in a better proposal and a more efficient review process.

Finding: Although participation of regional representatives in the proposal review process was viewed positively, a lack of information on the overall context and the potential for workload/scheduling problems were identified as two challenges in the external assessment process.

While there is little evidence of challenges associated with the internal proposal assessment process, provincial/territorial government representatives and experts identified two key challenges with the external assessment process.

  • Lack of information on context : each province/territory conducts the assessments in isolation. As a result, reviewers do not have a good sense of the balance (in terms of communities selected) across the country. For example, reviewers indicated that they did not know how many UEY sites in total would be approved for each province and, therefore, did not know how many proposals to recommend for funding by province or territory.
  • Workload/scheduling : external reviewers participate in the proposal review process in addition to their regular duties. A few key informants noted that this adds to their usual workload and can result in scheduling difficulties, although they acknowledged that the UEY Initiative has little control in this area.

2.2.3 Experience of Non-Funded Applicants

Finding: The experience of non-funded applicants was mixed, reflecting the unique situation and challenges faced by each community. Their more detailed feedback suggested that the mandatory requirements (which are intended to ensure the capacity of the community to be involved in the Initiative and the integrity of the data) were not consistently well understood.

Based on their experience with the UEY application process, key informants who were non-funded applicants (n=5) were asked to comment on what they felt worked well. Most explained that the UEY proposal process helped them to bring the community together and, as a result, they were able to form networks that continue to do work on early childhood issues (although without funding from the UEY Initiative). Some key informants also reported that the networks they formed would help them to develop ideas and to submit a stronger proposal next time.

The non-funded applicants were also asked what lessons they learned about the UEY Initiative, based on their own unique experience during the application process.

  • One key informant felt that there was a lack of information available during the application process, leaving little opportunity to address potential shortcomings in the community’s proposal. Another community representative, however, offered a different perspective and explained how feedback from UEY officials during the process allowed the community to adjust the proposal and include a stronger role for the school board and, as a result, submit a successful (second) application.
  • Another interviewee explained that, during the application process they discovered that they had misinterpreted the purpose of the UEY Initiative. They withdrew their proposal when they realized that they had already completed a similar initiative in their community.
  • Another key informant explained that during the application process they came to the conclusion that the UEY Initiative was better suited to large urban communities and concluded that their community was too small to qualify as a UEY site.

Non-funded applicants were also asked to comment on what they felt did not work, and needed to be changed.

  • Mandatory requirements : most had concerns about the requirements they had to meet in the application process (i.e. most would like the Initiative to be more inclusive). For example, one key informant explained how the community chose an umbrella organization as a sponsoring agency, but was rejected because the sponsoring organization had not been in operation for the required minimum of three years. Similarly, another key informant talked about how the criteria excludes small communities (e.g. with less than 300 kindergarten children) and needs to be revised to be relevant to First Nations communities.
  • Explaining their community : one key informant felt the application template did not allow for a full explanation (when more details might have helped at the review stage). Another key informant found it difficult to explain how their community worked using the application template, and felt HRSDC officials should be more flexible.

2.3 Roles and Responsibilities

This section examines whether the roles and responsibilities of key players in the UEY Initiative are clearly understood and being implemented as intended.

Finding: Evidence indicates that roles and responsibilities of the key players were well understood at the time of the evaluation and were implemented as intended, although there had been some clarification in response to initial challenges.

The specific roles and responsibilities of the key players were described Section 1.2.5. According to UEY program officials and program documentation, the roles and responsibilities of key players were explained and clarified through UEY orientation sessions and through written communication. Most key informants who were community representatives, program officials, and independent contractors felt that roles and responsibilities were well understood at the time of the evaluation. As with other aspects of UEY, there was some variation across communities in how the roles and responsibilities were implemented. However, that variation remained within the parameters of the UEY Initiative.

The document and file review and key informants indicated some initial challenges were experienced (generally site-specific and reflected the variations across the UEY sites). The evidence also indicated that those challenges have been largely overcome. The initial challenges are listed below for each key player.

  • In a few cases there was a need to clarify HRSDC’s expectations with respect to the level of involvement of the community sponsors/sponsoring organizations . A program official key informant noted that this clarification was done during the process of implementing the contribution agreement in each community, which resolved any early confusion.
  • There was an early need to clarify the role of community coordinators with respect to their relationship to the independent contractors (i.e. that the coordinators do not work for the independent contractors). Independent contractors who were key informants noted that some community coordinators had stronger working relationships within the community than others and were able to leverage their relationships and play a stronger leadership role in their community.
  • In a few cases there was a need to clarify that the community coalition had an advisory role and not a lead role in the community’s project. As well, a few community coordinators who were key informants noted an appreciable decline in the numbers of coalition/community members attending meetings over the course of the project. Independent contractors noted that there was variation across communities, with some community coalitions playing a larger role than others in the UEY projects.
  • Not all of the 21 UEY sites hired a community researcher . In communities where a community researcher was hired, there was no evidence of a lack of understanding or clarity with respect to the role of the researchers.
  • There was an early need to clarify the roles and responsibilities of the independent contractors in the EDI and PIDACS data collection and reporting process. Sponsors and coordinators from a few communities reported a misunderstanding on the part of independent contractors regarding what was required of them. There were a few cases of problems resulting from miscommunication between the independent contractors and the school boards, but one key informant noted that these cases could have been due to staff/management changes at the boards. There was also evidence of coordination challenges, and a few communities felt the independent contractors expected an unreasonable level of support from the community.

Finding: Opinions were mixed on the clarity and implementation of the Department’s role, largely due to staff turnover at HRSDC .

While some sponsors and coordinators reported having positive experiences with HRSDC, others reported significant problems related to staff turnover and delays. According to some community representatives, staff turnover at HRSDC meant that new project analysts had to “catch up”, which led to communication difficulties between HRSDC and the communities. At the same time, however, other community representatives praised the professionalism and assistance provided by HRSDC project analysts. The independent contractors also had mixed opinions of the Department, and one also noting the challenges to the communities associated with staff turnover at HRSDC.

2.4 UEY Design and Key Research Tools

This section begins by examining the key strengths and major challenges/limitations of the design of the UEY Initiative. It also identifies some suggestions for improving UEY design. The rest of the section presents the findings on the strengths, limitations/challenges and suggested improvements for the EDI, the PIDACS and the Community Mapping Report. Areas where changes/refinements had already been made at the time of the evaluation are discussed in Section 2.5.2.

2.4.1 Assessment of the Overall Design Model

Perceived Strengths of the UEY Design

Finding: Key design strengths are considered to include collaboration, community focus, credibility and communication.

There is limited evidence from the document and file review of the specific strengths and limitations of the Initiative’s design. Evidence from the Early Implementation Review, however, indicated that the UEY model works best when communities use a collaborative, inclusive approach (government and non-for-profit organization involvement), have strong leadership, and produce accessible information products.

Key informants who were program officials and provincial/territorial government representatives and experts were asked what they considered to be the key overall strengths of the UEY design. Although there was no clear consensus, the following four categories of strengths were cited:

  • Collaboration , which includes:
    • bringing together the community’s resources and strengths with federal resources;
    • the relationship between UEY and provincial partners (e.g. assessment of proposals and federal/provincial/territorial committees), which helps to ensure efforts are complementary; and
    • HRSDC staff and the support (e.g. software) and training (e.g. for mapping) provided to the sites;
  • Community , which includes:
    • the use of dedicated positions (e.g. community coordinator and community researcher) for each UEY site rather than using existing positions within each community, an approach which ensures an alignment of skills with the demands of the UEY Initiative;
    • community participation in data gathering (allows the community as a whole to develop a better understanding of the factors that help or hinder childhood development and promote change in the community); and
    • community capacity building through the community’s work on data collection, preparation of inventory of programs/services, and working with Census data to conduct their community mapping;
  • Credibility , which includes:
    • a sound Call for Proposals and assessment process;
    • sound data collection instruments and approaches (e.g. EDI, PIDACS, Community Mapping) and using independent contractors to ensure standardized data collection for EDI and PIDACS, while allowing for sufficient flexibility to adapt to community-specific needs;
    • provision of credible research data from multiple sources as a catalyst for community action and the development of a Community Action Plan; and
    • for the UEY Initiative as a whole, good (credible) data that has come from a wide array of sources and communities across Canada;
  • Communication , which includes:
    • raising awareness of the importance of early childhood development and encouraging a focus on children within the community; and facilitating knowledge transfer.

Perceived Challenges/Limitations and Suggested Changes for the UEY Design

Finding: The main design concern identified by key informants was a lack of sufficient linkage between research data from the EDI and PIDACS.

The number and breadth of the strengths of the UEY design noted by key informants indicates the ongoing successes and general satisfaction with the Initiative among the UEY sites. At the same time, key informants felt there was insufficient linkage between research data from the EDI and PIDACS. The UEY design initially called for the EDI and PIDACS to be conducted for the same cohort of children. Due to administrative delays in contracting, however, data collection for the PIDACS was stalled for almost a year. As a result, a different cohort was used for the EDI and PIDACS. Some key informants were skeptical that the information obtained from two different cohorts will provide useful and accurate data. Section 2.4.3 examines the use of two different cohorts in more detail.

Finding: The suggested improvements to the UEY design focused on improving or refining processes, rather then a redesign of the Initiative. Many of the suggestions made by the key informants (e.g. improving coordination with provincial/territorial early childhood development programs and projects) have been a focus of the UEY Initiative from the beginning. The literature review suggested taking a longer-term approach and encouraging communities to also plan for the needs of older children.  

Key informants who were program officials and provincial/territorial government representatives and experts were asked whether they felt any changes were needed in the design of the UEY Initiative. For the most part, the areas they identified for improvement coincided with areas that have been a focus of improvement efforts from the beginning of the UEY initiative. Their suggestions included:

  • improve the proposal review process by devoting more effort to assessing the capacity of UEY applicant sites to ensure that funding goes only to communities that have the necessary capacity to undertake a UEY project;
  • improve coordination with provincial/territorial early childhood development projects and programs; and
  • explore options for improving knowledge transfer  (e.g. involve the business community in the UEY Initiative).

The literature review found the general characteristics of the UEY Initiative to be similar to many community initiatives being implemented in other countries. These characteristics also serve as operating principles and strategies that have been found to work well across those community-driven early childhood development initiatives. For example, all of the initiatives:

  • are locally controlled and responsive to local needs, as perceived by the people in the community;
  • place a priority on a community-driven process that defines program goals, intervention strategies, and specification of local project objectives;
  • appear to have approached their communities in a similar way (i.e. open invitation to participate, clear specification of participation requirements, an expectation that there will be key agency involvement, and the use of responsible local sponsors); and
  • support some form of evaluation.

The literature review also found that program differences reflect value choices or government priorities rather than deficiencies (as indicated in Table 2.2).

Table 2.2: Summary of Program Differences Across Other Programs
Program Key Features
Communities for Children (Australia) and Success by Six (United States)*
  • Focused on young children and responds to the importance of enriching experience for children in the preschool years.
  • Policy materials tend to refer to the importance of brain development and the provision of stimulating environments.
Better Beginnings, Better Futures (Canada)
  • Focused on two groups: very young children (birth to age four), and older children (age four to eight).
Children’s Trusts (England)
  • Concerned with all children and youth.
  • Also concerned with certain conditions (e.g. child abuse and neglect, poor school achievement, and anti-social behaviour in adolescents).
Vibrant Communities (Canada) and Social Inclusion Partnerships (Scotland)
  • Whole community strategies aimed at improving the living conditions for all community members.
  • Aims to enhance the lives of children, adults and whole families.

* Note that the UEY Initiative would be a part of this group of programs.

Source: Literature review.

  • * Note that the UEY Initiative would be a part of this group of programs.
  • Source: Literature review.

Regarding suggestions for improving UEY design, the literature review noted that the Initiative (which is focused on younger children) varies from many similar types of programs in other jurisdictions in that it establishes a boundary between younger and older children. This suggests that the logic of the UEY Initiative might be enhanced by including a more explicit treatment/consideration of the issue of developmental continuity. For example, UEY communities could be asked to anticipate and explicitly plan for the needs of children after they pass beyond the early years.

2.4.2 Assessment of the EDI

As noted in Section 1.2.3, the EDI is a population-based measure used to rate the readiness to learn of four and five-year old children when they are in kindergarten. Data are collected from kindergarten teachers on all kindergarten children (year prior to grade one) in schools participating in the UEY project. Footnote 17 Funds are required for replacement teachers while participating teachers attend training sessions about the completion of EDI questionnaire and while participating teachers complete the questionnaire for each child. The funds for teacher replacements are managed by the sponsoring organization, through the contribution agreements.

Participation of School Boards in the EDI

Finding: Some key informants noted that some communities experienced initial problems in obtaining buy-in from the school district staff administrations (e.g. some schools/school boards were concerned about the possible misuse of results).

  • As noted in Section 1.2.5, schools/school boards are key players in implementing the EDI. Therefore each community’s UEY proposal had to include letters of support from each participating school board indicating their willingness to: a) provide lists of children and contact information for their families; and b) facilitate the collection of UEY data in their schools with their senior kindergarten (or equivalent) children. By requiring signed letters of commitment, the UEY Initiative sought to ensure that school boards were fully engaged in the UEY Initiative.
  • As noted in Section 2.1.2, some community representatives and independent contractors indicated there were initial problems in some communities getting buy-in from the school district staff administrations. For example, some school boards were concerned that the results could be misused/misconstrued and reflect poorly on specific schools or boards. As well, schools in some of the provinces where the EDI had already been implemented (British Columbia, Manitoba and Ontario) questioned why the EDI had to be repeated. A few key informants also noted early challenges with respect to compensating schools/boards for teacher replacement costs associated with conducting the EDI.

Perceived Strengths of the EDI

Finding: A key strength of the EDI is that it is a standardized instrument that is nationally used and recognized. Other perceived strengths included processes put in place to help support the quality of the collected data.

Key informants who were community representatives, independent contractors, and program officials were asked to comment on the strengths of the EDI. They indicated that a key strength is that the EDI is a standardized instrument that is nationally used and recognized. Footnote 18 Certain other strengths were also identified by key informants and the project file review.

  • Training: providing training to teachers on how to complete the EDI was credited with increasing the comfort level of participating teachers, providing the basis for a more standardized approach for completing the EDI and mitigating (to some extent) the potentially subjective nature of the EDI.
  • Compensation : reimbursing schools/school boards for the time spent by teachers in completing the EDI was credited with increasing the level of buy-in and participation on the part of school boards and individual schools.
  • Research staff: having an independent contractor responsible for the EDI was credited with providing support to the communities and making results available in a timely fashion.
  • Analytical methods: the mapping of the EDI results was credited with helping to better illustrate the data, and increase community understanding of the data.

Perceived Challenges/Limitations and Suggested Changes for the EDI

Finding: Most of the challenges/limitations identified for the EDI were related to the EDI process (i.e. certain areas of communication, reporting, and data sharing) rather than related to the EDI instrument itself. The suggested improvements largely reflected the challenges encountered with administering the EDI and were, in some cases, specific to the experience of the key informant’s community.

Key informants who were community representatives, independent contractors, and program officials were asked to provide feedback on the limitations/challenges with the data collection, analysis methods and data quality of the EDI. They were also asked whether they felt any improvements needed to be made to the EDI data collection and analysis.

In general, the EDI was considered to be a credible instrument and its overall implementation was generally viewed to have been successful. The feedback also indicated that some communities experienced few challenges in implementing the EDI, while others experienced significant hurdles. Table 2.3 shows the main challenges/ limitations related to the EDI process that were identified by key informants and the project file review. The suggestions offered by key informants for improving the EDI generally corresponded with these areas (as shown in Table 2.3).

Table 2.3: Challenges/Limitations and Suggested Changes for the EDI
General Area Challenges/Limitation Suggested Changes
Communication
  • Between community coordinator and independent contractor : there was insufficient interaction and a lack of direct contact between the independent contractor and community coordinators because communication was channeled through HRSDC.
  • Improve communications: to ensure a shared understanding among all key players of how the EDI is going to be implemented. Specifically the roles and responsibilities of each key player and the data collection process need to be communicated to everyone involved.
Reporting
  • Initial dissatisfaction due to poor communication : a few community coordinators indicated that their insufficient interaction with the independent contractor resulted in initial dissatisfaction with the EDI reports.
  • Not sufficiently user-friendly : the early EDI reports were not user-friendly, and were difficult to understand. The reports have since been modified to the satisfaction of community coordinators.
  • Improve clarity of reporting: the overall clarity of the EDI reports should be improved. Also the reports should provide more explanation to support data tables so the reader fully understands the findings.
  • More information is needed on what communities can do with the EDI data/report (i.e., to help identify need and provide services).
Participation
  • Participation issues for some groups some communities’ schools/school boards had ethnic or multicultural populations who were reluctance to participate in the EDI. These populations do, however, participate in local UEY project activities.
 
Data sharing
  • Data could not initially be shared:   the letter to parents explaining the EDI collection was not as comprehensive as it might have been with respect to uses of data.  Initial departmental review of the documentation and legislation regarding disclosure of personal information resulted in the projects being told that they would not have access to the personal-level EDI data files.  A subsequent review of the legislations determined that in specific and limited circumstances the personal-level data can be disclosed.  This decision resulted in a data sharing proposal and agreement process which allowed projects to receive their own EDI data file for the duration of their UEY project to do specific analysis as outlined in their proposal. 
  • Clarify data sharing consent: the letter to parents (respondents) must be clear with respect to what is being collected and how the data will be used so that their participation in the research is well informed and participating communities can more easily access the personal-level data for community-specific analyses.
Length of assessment
  • Assessment is too long : a few community key informants felt that the EDI assessment (i.e. length of time needed to administer the instrument) was too long.
 
Credibility
  • Some research-related concerns : some key informants were concerned that the EDI was conducted on only one cohort of children and felt that it was open to kindergarten teachers’ subjective views of individual children’s abilities. Program documentation indicated, however, that the EDI was selected for use based on the best advice of recognized academics in the field of early childhood development.
  • Address concerns about geographic and cultural biases : a few community representatives felt that the EDI has an urban and Anglo bias, which led them to question the validity of the results with respect to rural or ethnically diverse communities.
  • Better identification/ inclusion of special needs children: a few key informants felt that special needs children were being largely left out of the EDI.
 
  • Source: Key informants and file review.

2.4.3 Assessment of the PIDACS

As noted in Section 1.2.3, the parent interview component of the PIDACS provides information on family background and family processes, including socio-economic status, parenting style and use of community resources. The direct child assessments examine receptive vocabulary through the Peabody Picture Vocabulary Test - Revised, early literacy and general cognitive skills through the “Who am I?” instrument and early numeracy with the Number Knowledge Test. The PIDACS was intended to be applied to a sample of the EDI cohort of children in each UEY community. Due to delays in implementing the PIDACS, however, it was applied to a different cohort.

Participation of School Boards in the PIDACS

Finding: Key informants indicated large variations in the level of cooperation across school boards.

Key informants who were independent contractors and program officials described large variations in the level of cooperation across school boards. The independent contractors felt that the engagement and cooperation of school boards had a direct impact on the participation rate of parents. For example, cooperative school boards were more pro-active in engaging parents in the process (i.e. encouraging parental participation). By following up with parents, they increased the response/participation rate for the community.

Perceived Strengths of the PIDACS

Finding: A key strength of the PIDACS is that it is a validated, standardized instrument based on the National Survey of Children and Youth (NLSCY). Another key strength is that the instrument and data collection can be tailored and customized to accommodate differences across UEY sites.

Key informants who were community representatives, independent contractors and program officials felt a key strength is that the PIDACS is a validated, standardized instrument based on the National Longitudinal Survey of Children and Youth (NLSCY). Another key strength is that the instrument can be tailored and customized to accommodate the differences among the 21 UEY sites (e.g. language of interviews, use of in-person interviews). Footnote 19 As a further strength, some key informants cited the fact that PIDACS collects data on both parents and children, and measures community and family factors.

As well, certain factors were identified as facilitating the implementation of the PIDACS.

  • Flexibility in accommodating differences across communities: the process was adapted to meet the needs of the community (i.e. parent interviews were conducted in the family’s language of choice, interviews were conducted in-person when a telephone interview was difficult).
  • Using local people to conduct the direct assessments with the children helped to increase community buy-in and helped to meet the various linguistic needs of the community. Local interviewers were also used for the in-person interviews in one community.
  • Training and workshops : community representatives found the UEY workshop on the PIDACS to be extremely helpful. Assessor/interviewer training was considered to be standardized and comprehensive.

Perceived Challenges Encountered With PIDACS and Suggested Improvements

Finding: The most frequently cited challenge was the delayed implementation of the PIDACS. Community representatives and independent contractors also expressed concern about some of the consequences of that delay.

Most of the community representatives who were asked to comment on challenges mentioned the delay in conducting the PIDACS data collection. The delay was caused by administrative delays in implementing the UEY Initiative and in selecting the contractor responsible for collecting the PIDACS data. Community representatives and independent contractors also noted certain consequences of the delay.

  • The PIDACS and EDI were conducted on different cohorts: one consequence of the delay was that the EDI and PIDACS were conducted on different cohorts of children. UEY officials sought methodological advice on this issue and the evidence shows that all results from the EDI and PIDACS are valid (regardless of the use of different cohorts). In spite of this, comments by some key informants indicated that they remain sceptical and this suggests need for further dialogue with communities. 
  • Collaboration with schools/school boards: confusion occurred on the part of some schools/school boards about their agreement to participate. Due to the delay, people who had agreed at the application stage had left the school/school boards by the time the project started. New people had to be brought up to speed on the project and the advantages of participation. This often involved a significant investment in time on the part of community coordinators.

The following limitations were also identified: 

  • Respondent burden : the parent interview is seen as too lengthy. One key informant reported that the parent interviews were initially about 90 minutes in length. This was subsequently reduced to 60 minutes, but was still seen to be too long.
  • Data sharing: UEY officials reported that a base letter was approved by HRSDC to inform parents about the data collection, uses and sharing of the data, and to ask for consent for their children to do the assessment activities and agreement to share their data with the sponsoring organization.  It was possible for the sponsoring organizations to change sections of the letter that were not related to data release and sharing.  In a few cases, the inability to revise the complete letter was an issue for some community projects.

Finding: Some of the suggested improvements for the PIDACS tended to be similar to the suggestions made for the EDI (e.g. reducing the length of the interview, clarifying data sharing consent, eliminating bias, providing more lead time and improved communication).

Key informants suggested a number of changes to the PIDACS that were similar to those suggested for the EDI.

  • Increased brevity: a few community representatives felt the interviews were too long.
  • Clarify parental approval for data sharing: use an improved/in-person administration of the consent form to ensure that families are not missed and parents understand how the information will be used. A few key informants felt that some parents may not have understood the consent forms that were sent, even when they were sent in the family’s language of choice.
  • Address concerns about geographic and cultural bias : a few key informants felt that the PIDACS was culturally biased and a few felt it had an urban bias.
  • More lead time, better planning and improved communication of the roles of key players: including a strategy that outlines roles and expectations for key players, timelines for implementing the PIDACS, and what contributes to success.

Finding: The suggested improvements for the PIDACS also included certain process changes that related specifically to the PIDACS (e.g. adding further flexibility to accommodate differences across communities, and hiring local/known people as PIDACS administrators).

Key informants also suggested process changes related specifically to the PIDACS.

  • Flexibility in accommodating differences across communities: in-person interviews should continue to be held in the family’s language of choice so as to increase parents’ comfort and understanding. This is particularly important with ethnic and new immigrant communities. This approach has proven to be effective in communities where languages other than English or French are used.
  • Hiring local/known people: a few respondents felt that having PIDACS administrators who are known and trusted in the community increased participation rates. This approach, which was implemented successfully in every community, was also reported to increase community buy-in and sense of ownership of the UEY project.

2.4.4 Assessment of the Community Mapping Report

As noted in Section 1.2.4, the UEY Initiative requires each community to conduct a community mapping exercise to illustrate the UEY research findings for community members. The Community Mapping Report provides information on the linkages between the community’s characteristics (including socio-economic data, programs and services) and child outcomes.

Perceived Strengths of the Community Mapping Report

Finding: Key strengths of the Community Mapping Report are the provision of useful data and information, and the development of local capacity to undertake research.

At the time of the formative evaluation, most of the UEY communities had begun the process of community mapping, but had not completed their community mapping exercise. Therefore, not all community representatives felt able to comment on the Community Mapping Report. Those who felt able to comment, along with program officials, cited the following strengths related to the Community Mapping Report:

  • providing useful information : the Mapping Report provides practical and useful information on programs/services in the community, provides an assessment of program reach, and assists with the identification of overlaps and gaps in services and programming; and
  • capacity development: the mapping process has allowed UEY sites to develop capacity with respect to human resource skills and knowledge. In some cases, community representatives reported that UEY sites have become a “knowledge centre” for mapping in the region.

Perceived Challenges/Limitations and Suggested Changes for the Community Mapping Report

Finding: Challenges identified for the Community Mapping Report related to capacity issues within the community (i.e. a limited capacity to develop the report) and challenges in obtaining relevant data.

Although there is evidence that the community mapping exercise was challenging for some communities, there was also a high level of satisfaction with the resulting reports  (particularly the ability to identify available programs and services and gaps in programming). Certain challenges were identified within communities that had either completed or were in the midst of conducting the community mapping exercise.

  • Limited capacity: many UEY communities reported that they had limited capacity to collect and analyze data and develop the Mapping Report. Accessing the expertise required to do mapping was identified as a key challenge by community representatives and corroborated by the file review. This challenge was particularly apparent for small communities. Footnote 20 Necessary skills had to be recruited/hired from outside the community, which sometimes took longer than anticipated. There was no evidence, however, that UEY communities were unable to recruit the necessary capacity to undertake the mapping exercise.
  • Data: some data points in the Mapping Report do not apply to rural areas (e.g. playgrounds – children in rural areas tend not to play in playgrounds). In addition, there were challenges in obtaining the necessary data and geographic information.

Finding: A few key informants suggested the community mapping exercise could be improved by encouraging information sharing among UEY communities and by providing more/better guidance and assistance when communities are preparing the Community Mapping Report.

A few key informants offered suggestions for improving the Community Mapping Report, based on their experience to date.

  • Mandatory requirements : improve the identification of neighbourhoods of a suitable size for UEY (i.e. so that data is not suppressed due to small sample sizes).
  • Information: improve/encourage information sharing among UEY communities.
  • Guidance/assistance: provide more guidance/assistance when communities are preparing the Mapping Report. Many communities lack the necessary capacity to undertake the mapping exercise on their own. A few community representatives noted that the mapping workshop provided by HRSDC was not particularly helpful/did not provide them with sufficient training/knowledge.

2.5 Lessons Learned

This section examines lessons learned by examining what has worked well for the 2005 round of UEY sites. It also examines areas identified for change and some refinements made to date for the 2006 round of UEY sites. The lessons learned can be considered to apply to all 21 sites because the majority of data collection focused on all of these sites, including the document and file review and the key informant interviews with program officials and independent contractors.

2.5.1 Aspects of UEY That Have Worked Well

Finding: Key informants identified five main aspects of the UEY Initiative that they felt have worked well: providing guidance and training, use of project officers, emphasizing community relationships/partnerships, having a community coordinator, and emphasizing knowledge transfer and communication activities.

Key informants who were community representatives and program officials were asked to identify the key lessons they have learned to date from their experience with the UEY Initiative. Some reflected on the lessons they have learned with regard to the Initiative’s implementation, while others commented on results. Overall, five main aspects of the UEY Initiative were identified as having worked well (with a few key informants identifying each one).

  • Providing guidance/training : the orientation, training, and guidance provided by HRSDC to UEY communities was credited with providing the opportunity for sites to learn about the Initiative and meet UEY program management, staff and representatives from other UEY sites. This encouraged relationship building among key players involved in UEY at the national level.
  • Use of project officers : having an HRSDC project officer dedicated to each UEY site who is knowledgeable about the UEY Initiative was considered to be a good approach.
  • Emphasizing community relationships/partnerships : the emphasis placed on, and the effort put into, community relationships/partnerships for UEY sites was considered to be a good approach.
  • Having a community coordinator:  having a dedicated position and a person who is know in the community as project coordinator/manager was considered beneficial.
  • Emphasizing knowledge transfer and communication activities : the emphasis and activities undertaken by UEY sites in these areas including booklets, Websites, learning sessions, workshops, newspaper articles, as well as data sharing with community partners.

2.5.2 Areas Identified for Change

Finding: Key informants identified certain process-related areas for change (e.g. more time is needed to plan and prepare for data collection, delays need to be avoided/minimized, clearer consent for data sharing, and clearer communication with school boards). Most of these are consistent with areas where some refinements have been made for the 2006 round of UEY sites.

Table 2.4 shows the areas where key informants felt that changes needed to be made.

The review of program documents indicated that plans and program delivery mechanisms for the UEY Initiative are revisited regularly, with risks and opportunities for improvement identified and action taken as needed. Examples of improvements to UEY design and delivery are also shown in Table 2.4, based on the document and file review and key informant interviews with program officials. Overall, there is good evidence that the UEY Initiative is pro-active in responding to challenges or limitations in the design and delivery of the UEY Initiative as they arise, and in modifying aspects of program design or delivery as necessary.

Table 2.4: Areas Identified For Changes and Refinements Made to Date
General Area Suggested Changes Refinements Made to Date
Planning/timelines
  • More time is needed to plan and prepare for EDI and PIDACS data collection
  • Delays need to be avoided/minimized : the three-year UEY timeline will be difficult to meet due to delays in project implementation Because of the delay, some buy-in was lost, momentum was lost in a few communities, and schools and teachers had to be re-engaged.
  • Refinements to timelines include:
    • EDI and PIDACS data collection are to occur during the second year of the project (previously EDI occurred in year 1, while PIDACS occurred in year 2);
    • the inventory of community programs and services is to be completed in year 1 of the project (previously started in year 2); and
    • the development of the Community Action Plan starts in year 2 of the project.
  • Change in mandatory requirements : the applicant organization (potential project sponsor) must be legally incorporated.
Contact with HRSDC
  • More contact and communication with HRSDC project analysts : a few key informants indicated that communication with their project analysts was insufficient.
  • Increased support to funded projects: providing extensive support to projects through templates and training, in addition to frequent contact for project monitoring.
  • Increased guidance : a financial reporting component has been added to the orientation session provided for new UEY sites.
  • Meeting with successful project sponsors: some applications appeared to be written by contractors with little input from the individuals who would be responsible for implementing the UEY project. Therefore, a decision was made to meet with successful project sponsors before entering into the next round of contribution agreements. This was to allow UEY staff to familiarize the 2006 round of UEY project sponsors with HRSDC processes and requirements. This was also to better enable project sponsors/ sponsoring organizations to realistically plan their project’s financial requirements and work plans

 

Communication/understanding/participation/buy-in
  • Data could not initially be shared: The letter to parents explaining the EDI collection was not as comprehensive as it might have been with respect to uses of the data.  Initial departmental review of the documentation and legislation regarding disclosure of personal information resulted in the projects being told that they would not have access to the personal-level EDI data files.  A subsequent review of the legislation determined that in specific and limited circumstances the personal-level data can be disclosed.  This decision resulted in a data sharing agreement process which allowed projects to request their own EDI data file for the duration of their UEY project to do specific analysis as outlined in their proposal.  
  • Clearer communication with school boards with respect to what is expected of them in terms of participation/ involvement in EDI and PIDACS.  
  • Increased clarity/guidance/tools :  tools and workshop components have been developed to educate and respond to confusion over data collections requirements. Specific tools (developed or improved) include templates for acomprehensive letter and consent form for parents involved in PIDACS data collection.

 

  • Source: Document and file review and key informants.

3. Progress Toward Immediate Objectives

This section examines the progress made by the UEY Initiative towards its immediate objectives at the time of the evaluation (i.e. approximately 18 months into the community projects). It also examines unanticipated impacts. As indicated in Section 2.1, the Initiative has three immediate objectives

  • increased support for communities;
  • increased availability and sharing of information on the role of families and communities in child development; and
  • increased awareness and use of parent and community knowledge regarding child development.

3.1 Increased Support to Communities

Finding: The evidence suggests that progress has been made in providing communities with funding support. Although it is too early to assess the ability to attract additional support, the UEY has enabled community coalitions to identify priorities for grants, incorporate UEY into funding proposals, and apply for and secure funding.

Key informants who were UEY program officials felt that progress has been made in providing communities with funding support. Although the UEY objectives provide little indication of the anticipated sources of increased support, the quarterly activity reports examined by the project file review indicated that the UEY Initiative had enabled community coalitions to identify priorities for grants, incorporate UEY into funding proposals, and apply for and secure funding.

While the ability to attract additional funding is important, most key informants who were community representatives indicated that it was too early to determine the impact of such funding. Some key informants stated that they were still gathering data, while others were optimistic that there would eventually be increased financial support stemming from the UEY Initiative.

Finding: The key informant interviews indicated that progress has been made in providing communities with technical assistance, with community mapping being cited as a main catalyst for this progress.

Key informants who were community representatives and program officials felt that progress has been made in providing communities with technical assistance.

Most key informants who were community representatives reported that there was increased technical assistance for their communities as a result of their involvement with the UEY Initiative. They cited community mapping as the main catalyst for this progress. Progress in some communities included consultation on data collection and use, and knowledge dissemination. In other communities mapping was cited as raising awareness of the need for technical support. The evidence also suggests that the community mapping exercise, while challenging for a few communities, has resulted in the development of capacity within communities.

3.2 Increased Availability and Sharing of Information

3.2.1 Partnerships

Finding: Evidence from the file review and key informant interviews indicated that UEY communities have been successful in forming or strengthening partnerships as a result of the Initiative.

Key informants who were community representatives and program officials generally believed that partnerships have increased and strengthened as a result of the UEY Initiative. The review of project files also indicated that UEY communities have been successful in forming or strengthening partnerships. For example, information drawn from the project files indicated that all UEY sites have formed partnerships, ranging from 19 partnerships in one community to one partnership (with an average of ten). As noted in Section 1.3.2, however, the available community partnership data should be treated with caution because problems in determining what constituted a partnership makes it difficult to interpret these data.

Finding: The most frequent partnership reported was with schools/school boards/school districts, followed by municipal and provincial governments, community service providers, and health and social service networks.

Table 3.1 provides a summary of the types of organizations involved in partnerships with UEY communities. Table 3.1 also shows the number of UEY communities that reported a partnership with each type of organization. These data were collected from the UEY project files and should be treated with caution (as noted above). The most frequent partnership reported was with schools/school boards/school districts (reported by 17 UEY sites), followed by municipal and provincial governments, community service organizations, and health and social service networks. Other types of organizations reported to be involved in partnerships included other not-for-profit/non-governmental organizations and other UEY sites and colleges. Footnote 21

Table 3.1: Organizations Partnering with UEY Sites
Type of Organization Number of UEY Sites Reporting Partnership
Schools/school boards/districts 17
Municipal government 16
Provincial government 15
Community service providers 14
Health or social service networks 13
Other not-for-profit/non-governmental 11
Other UEY sites 10
Colleges 9
Researchers/research institutes 7
Aboriginal organizations 6
Universities 4
Religious organizations (e.g. local church) 3
Parents/parents groups 3
Private sector 3
Federal government (other than HRSDC) 2
Other (specify) 5
  • Source: UEY Project files, 2007.

While there is evidence that partnerships have increased, at the time of the evaluation there was no evidence of an impact on the level of support to families with children.

3.2.2 Networks and Mechanisms to Share Information

Finding: Evidence from the file review suggests that networks and mechanisms have been developed to share information between HRSDC and UEY communities.

Evidence from the file review indicates that there has been increased sharing of information between HRSDC and UEY communities. According to the project files, 15 of the 21 UEY sites reported participating in networks or mechanisms to share information with HRSDC.

Finding: Evidence from the document and file review and key informants suggests that the UEY Initiative has increased the number of opportunities and tools for knowledge sharing and networking within communities on the role of families and communities in child development, with the types and number of activities varying across UEY sites.

The UEY Initiative provided assistance and guidance on sharing information within the community. For example, the Initiative has developed a guide for communities entitled Sharing the Knowledge: Maximizing the Impact of Your Project.

Most key informants who were community representatives and program officials agreed that there has been progress towards more opportunities to share knowledge and network within communities. One community sponsor commented that “I think this has been the biggest bang for the UEY buck so far.” Key informants cited a variety of mechanisms through which knowledge sharing and networking were taking place (e.g. task forces, Websites, presentations, workshops, various collaborations and the local media).

The file review indicated that all UEY sites reported either developing or participating in networks/mechanisms to share information within the community. The file review also indicated that information sharing activities have varied across sites (see Table 3.2), but often included attending conferences (reported by 19 UEY sites), presentations to school boards/school districts (reported by 15 UEY sites), and participation in relevant local committees (reported by 15 UEY sites).

Table 3.2: Types of Information Sharing Activities
Information Sharing Activity Number of UEY Sites Reporting
Participation in Information Sharing
Activity
Conference attended 19
Presentations to school boards/districts 15
Participation in relevant local committee 15
Presentations on UEY 11
Sharing of EDI data with school boards/districts 11
Participation in community planning sessions 11
Community consultations 10
Participation in community events 10
Briefings of community leaders on UEY 9
Other (specify) 8
Presentation at a conference 7
Community information session/open house 5
Site visits/presentations to local communities 5
Hosting public event related to UEY 3
Meet with/present to children and family 3
Interviews with media 2
Distribution of UEY research reports 1
Presentation/guest lecture at university 1
  • Source: UEY project files

Most key informants who were community representatives were able to cite a variety of networks, mechanisms and tools that had been created in the community to share information on child development since the beginning of the UEY project. Examples included Websites, community meetings, reports and research communiqués showing results, and surveys. Although the PIDACS and EDI were frequently identified as being key to the increased knowledge and data, key informants also identified other documents and information that had been developed and shared with the community at large.

The file review also indicated that UEY communities have produced a variety of knowledge dissemination tools (see Table 3.3). The most frequently cited tools were resource guides/asset lists (reported by 15 UEY sites), Websites or Web pages to present UEY information (reported by 14 UEY sites), and PowerPoint presentations (reported by 12 UEY sites). The document review and key informant interviews indicated that much of the work done to date can be considered groundwork (i.e., getting the project and those involved known in the community at large). In the first quarter of 2007, a number of activities related to knowledge products began to take place.

Table 3.3: Types of Knowledge Dissemination Tools
Knowledge Dissemination Tool Number of UEY Sites Reporting Production of Tool
Resource guide/asset list 15
Website/webpage to present UEY information 14
PowerPoint presentation 12
UEY handout/brochure 10
UEY newsletter 10
Press release 10
EDI handout/brochure 9
UEY article 8
Social marketing plan/communications strategy 8
Posters 8
Letterhead 5
EDI reports 4
Research communiqué 3
Community Mapping Report 3
Resource library 1
Other (specify) 14
  • Source: UEY project files

3.3 Increased Awareness of Child Development

Finding: The evidence suggests that the UEY initiative has increased parental and community awareness and knowledge regarding child development to some extent, although it was too early for this progress to be fully evident.

Most key informants who were program officials believed there has been progress in terms of increasing parental and community awareness and knowledge regarding child development. They cited the creation of brochures and presentations at workshops as examples of progress. Most key informants who were community representatives indicated there has been at least some progress in parental and community awareness, although some felt it was too early to assess this area. Footnote 22 Many cited increased dialogue with parents as an example of progress.

Finding: At the time of the formative evaluation, the evidence was inconclusive on whether there was increased use of knowledge regarding child development.

Most key informants who were program officials did not know if progress was being made towards increased parental and community use of knowledge regarding child development. Some said it was too early to tell. These key informants did see progress in facilitating the exchange of information on child development, however. As examples, they cited information sharing events and brochures.

Key informants who were community representatives had mixed opinions on whether there has been increased parental and community use of knowledge. They felt that this was difficult to determine and that it was still early in the local UEY projects’ lifespan. A few noted that parents are seeking out more information, which suggests that parents are using this information.

The evaluation team noted that program documentation and files for this indicator were limited. The number of requests for UEY information on early childhood development does not appear in project files. As well, as indicated in Section 3.2.2, the types and number of tools and activities for knowledge transfer/information sharing varied across UEY sites.

3.4 Unanticipated Impacts

Finding: Key informants identified a number of unanticipated impacts.

Key informants who were program officials, community representatives, independent contractors, and provincial/territorial government representatives and experts identified the following positive unanticipated impacts:

  • community coordinators and coalitions have assisted with other social development activities within the community beyond the scope of early childhood development;
  • issues of child care, in general, have been brought to the forefront;
  • some community coordinators are now seen as experts in the community in other areas relevant to UEY;
  • there is broader awareness of and interest in the early childhood issues (e.g. more requests for information) than expected;
  • there is increased ability within the community to engage local leaders;
  • there has been knowledge sharing with other communities involved in the UEY Initiative who have the same motivation and vision; and
  • non-funded applicants often felt that the UEY proposal process helped them to bring the community together and form networks that continue to do work on early childhood issues even in the absence of funding.

The key informants also identified the following negative unanticipated impacts:

  • because of delays, EDI and PIDACS data collections involved different cohorts of children leading some to question the credibility of results;
  • demands on community coordinators’ time (e.g. to make presentations to other audiences) are a significant burden;
  • the late recognition of the implications of privacy legislation meant that communities were initially offered access to aggregate data only (which limits local analysis and understanding).  A subsequent review of the legislation determined that in specific and limited circumstances the personal-level data can be disclosed.  This decision resulted in a data sharing proposal and agreement process which allowed projects to request their own EDI data file for the duration of their UEY project to do specific analysis as outlined in their proposal; and
  • requests for administrative assistance in conducting the PIDACS overwhelmed some school boards.

The document and file review did not provide any evidence of unanticipated impacts (either positive or negative).

4. Accountability Issues

This section examines:

  • the adequacy of program safeguards to avoid duplication;
  • the appropriateness of reporting strategies; and
  • the adequacy of reports for program monitoring.

The availability of suitable data for the summative evaluation is examined in Section 5.

4.1 Adequacy of Safeguards to Avoid Duplication

Finding: Despite the array of provincial programs aimed at children, the program scan conducted for the formative evaluation found no evidence of overlap or duplication with the UEY Initiative.

The literature review included a scan of programs in Canada. The scan did not reveal any universally available programs that would be jeopardized by the UEY Initiative. For example, programs identified as having a similar intent or approach (Success by Six, Better Beginnings, Better Futures, and Vibrant Communities) were available only on a very limited basis or were dependent on voluntary funding sources or ‘soft’ money. Most other early childhood initiatives (such as Best Start in Ontario) take quite different approaches and focus on different objectives. Footnote 23

Finding: A number of safeguards have been put in place to avoid or lessen the potential for overlap and duplication.

A number of steps have been taken to avoid or reduce the potential for overlap and duplication.

  • The Federal/Provincial/Territorial (FPT) Committee on Early Childhood Development Knowledge Information and Effective Practices  meets twice annually and helps to ensure that FPT activities are coordinated and that duplication is avoided.
  • UEY sites are required to inform the program analyst if they receive funding from any other federal, provincial or municipal sources that duplicate UEY funding (i.e. cover the same activities). If the other funding is received, the site’s UEY funding is adjusted to prevent duplication.
  • The external review process includes asking external assessors if they are aware of any UEY-like or UEY-related activities planned for the community such as the collection of data similar to UEY, data mapping work, and coalition building through other projects.

As well, UEY officials were confident that their own activities through networks and conferences help to keep them informed about other programs or initiatives being discussed or developed. All key informants who were program officials expressed confidence that the safeguards are adequate and do not need improving.

Finding: Evidence indicates that miscommunication was at the root of situations where some overlap and duplication may have occurred. This suggests a need for better communication between community representatives and program officials prior to project implementation.

Unlike their federal counterparts, most key informants who were provincial/territorial government representatives and experts were unaware of the safeguards in place to prevent overlap and duplication. Some cited examples of conflict. For example, one key informant explained how a provincial evaluation study competed directly with a UEY project for the time and resources of teachers and parents. Footnote 24 As noted above, however, community proposals for UEY funding are expected to include information about other programs or projects that may compete with or duplicate the UEY Initiative. However, most key informants noted that one key way to avoid overlap and duplication is to maintain clear communication channels (especially in times of staff turnover).

4.2 Appropriateness of Reporting Strategies

Finding: The general view among key informants is that the results-based reporting strategies for the UEY Initiative are appropriate and compatible with federal government accountability requirements.

Based on a suggestion in the Early Implementation Review, a Results-based Management Accountability Framework (RMAF) has been developed for the UEY Initiative. All key informants who were program officials expressed confidence that the required reporting strategies are appropriate and effective, and are compatible with federal government accountability requirements for contribution agreements. Most key informants who were community representatives believed that the results-based reporting strategies are appropriate and effective.

Finding: Most key informants who were community representatives expressed confidence in their community’s capacity to undertake the necessary reporting, although most also expressed the desire for more clarity and feedback from HRSDC as they moved forward with reporting on the results of data collection.

All UEY communities are required to submit quarterly activity reports in order to receive funding and to be reimbursed for project activities. In addition, the Initiative has developed a training session for reporting on UEY project results. This training session/guide is intended to encourage consistency in project reporting.

Key informants who were community representatives were asked to comment on what reporting they have done on their project’s results for UEY. Some indicated that they would not be able to proceed with reporting on the results of data collection until the Community Research Report was made available to them in January 2008.  Most, however, listed a number of reporting activities that went beyond the required quarterly activity reports. Most also expressed confidence in their community’s capacity to undertake the necessary reporting.

Key informants who were community representatives also viewed the reporting strategies (quarterly reports) as effective in terms of helping to track their own progress, and in providing HRSDC with oversight. At the same time, most expressed the desire for more clarity and feedback from HRSDC as they moved forward with reporting on the results of data collection. This was especially true for those facing timeline challenges. That group also suggested that it would be helpful to have HRSDC provide clearer and more reasonable reporting expectations.

4.3 Adequacy of Reports for Performance Monitoring

Finding: Although all key informants who were program officials felt that the performance monitoring reports of UEY sites were being sent in a timely manner, the project file review found that two-thirds of project files were missing at least one quarterly report.

All key informants who were program officials felt that the reports for ongoing performance monitoring of UEY sites were being received and usually in a timely manner. The project file review found, however, that two-thirds of UEY project files were missing at least one quarterly report.

The program officials were satisfied that the information in the reports is useful and sufficient for performance monitoring. One explained that HRSDC staff follow up and get the necessary information (i.e. a narrative description of progress and financial information) if the initial information does not appear to be useful.

5. Guidance for a Summative Evaluation

This section provides some preliminary suggestions/guidance for the design of a summative evaluation of the UEY Initiative. These suggestions could serve as input to the development of the evaluation framework for the summative evaluation. This section also provides some data-related suggestions to better support the summative evaluation.

Finding: A number of features of the UEY Initiative (e.g. a multi-site funding framework with community-level interventions) were identified as important considerations for conducting a summative evaluation.

The evaluation team conducting the formation evaluation identified a number of key features of the UEY Initiative that can be expected to have a strong bearing on what approaches can be used to conduct a summative evaluation.

  • The UEY Initiative is not typical of the types of programs referred to in the program evaluation literature. Instead, it is a funding framework designed to provide communities with local data and the capacity to develop information as a way to identify gaps in services and programs and to foster partnerships pertaining to child development.
  • The UEY Initiative recognizes that families and children in different communities face different challenges. The project activities that result from the Initiative will be uniquely designed to meet the specific needs of each community, but may still show a high degree of common content across communities.
  • The UEY Initiative is designed to permit a high degree of local control over program features and operation. This makes it difficult, on an a priori basis, to specify what the particular outcomes will be and how they ought to be assessed.
  • The Initiative is grounded in current psychosocial theory in seeing child and family outcomes as the product of complex social system processes where many cultural and community effects combine to determine outcomes for individuals (children).

These features of the UEY Initiative indicate that the approach developed for the summative evaluation will need to take into account the uniqueness of each community and the impacts of the UEY Initiative on each community as well as the overall impacts of the Initiative. This strongly implies that evaluation strategies will need to be implemented at the level of each community in anticipation of the summative evaluation to ensure that outcome data are available at the community level as well as at the level the overall Initiative.

Finding: After taking into account the features of the UEY Initiative and the uniqueness of each community, a theory of change approach Footnote 25 appears to be the most appropriate approach for the summative evaluation.

The literature review identified evaluation models appropriate for multi-site national human service programs that share many of UEY’s features. Footnote 26 Seven different approaches to conducting evaluations were considered to be particularly relevant to the needs of multi-site evaluations: cluster evaluation, developmental evaluation, extended-term, mixed method evaluation, five-tiered approach, initiative evaluation, systems approaches in evaluation, and theory of change. Based on her review, Dr. Lilley concluded that a theory of change approach was most appropriate for national, multi-site human service program initiatives and particularly for those involving community-level interventions with a strong degree of local control over the interventions implemented at the community level.

Finding: The evaluation team also suggested ways to better support the summative evaluation (e.g. do more to ensure that baseline data are collected consistently).

The evaluation team examined the availability of data for conducting a summative evaluation based on findings from the preliminary consultations and the review of data collection capacity conducted to help plan the formative evaluation. They found that not all of the necessary baseline data is being collected consistently. For example, there is a need to more clearly define information/data being requested from the UEY sites (e.g. clarify what is meant by partnerships). As well, baseline measures identified in the Performance Measurement Framework in the RMAF for the UEY Initiative should ideally be designed to be on-going, rather than providing a single snapshot.

The evidence also indicated a need to refine the logic model and more clearly articulate a set of measurable outcomes and indicators. Expected outcomes should be sufficiently focused so that changes may be attributed to the Initiative. As well, there is evidence that some outcomes are too broad in scope to be influenced by UEY (as noted in Section 2.1).

6. Overall Conclusions

This section highlights the overall conclusions provided by the formative evaluation. It also highlights the main areas identified for consideration/improvement.

6.1 UEY Design and Implementation

The formative evaluation found the design of the UEY Initiative to be generally sound . The evidence indicated that the UEY model is inclusive of a broad range of communities and is sufficiently flexible to accommodate their participation. Key strengths of the UEY design are considered to include collaboration, community, credibility and communication.

The objectives and intended outcomes of the UEY Initiative are consistently communicated, and participants appear to have a clear understanding of their project’s immediate objectives and expected results for the UEY Initiative. At the same time the focus and measurability of some of the Initiative’s intended outcomes could be improved, particularly in the case of the longer-term and ultimate outcomes.

The main design concern identified by key informants was a lack of sufficient linkage between research data from the EDI and PIDACS. The UEY design initially called for EDI and PIDACS to be conducted for the same cohort of children. Due to delays in contracting, however, data collection for the PIDACS was stalled for almost a year. As a result, a different cohort was used for the EDI and PIDACS. Although HRSDC has assured the communities that this will not affect the usefulness of the data, some key informants continue to be skeptical that the information obtained from two different cohorts will provide useful and accurate data.

The suggested improvements to the UEY design focused on improving or refining processes (e.g. improving coordination with provincial/territorial early childhood development projects and programs) rather than a redesign of the Initiative . The literature review suggested that the Initiative could be improved by taking a longer-term approach and encouraging communities to plan also for the needs of older children.

All 21 UEY communities have made progress toward implementing their project and are making progress towards their immediate objectives and expected results for the UEY Initiative. Although most sites experienced some early difficulties, those difficulties were largely a result of the challenges/characteristics within individual communities (e.g. staffing challenges and challenges in obtaining buy-in from certain groups and school boards), rather than a result of specific flaws in the UEY design.

The roles and responsibilities of key players in the UEY Initiative were well understood at the time of the evaluation, and have been implemented as intended, although there was some need for clarification early on. Some evidence of variability in the implementation of roles and responsibilities across UEY communities was noted, but those variations were within the parameters set out by the UEY Initiative.

The EDI, PIDACS and the Community Mapping Report are generally positively regarded by all key players in the UEY Initiative . At the same time, however, some process-related challenges/limitations were identified for each of these UEY research tools, along with some suggestions for change.

  • For example, as noted above, there was concern about the use of different cohorts of children for the EDI and PIDACS.
  • As well, the late recognition of the implication of privacy legislation and lack of clarity in parental information letters meant that access to EDI data could not occur initially as originally anticipated. At the time of the interviews, community representatives continued to express disappointment over the lack of access to individual level data. However, a subsequent review of the legislation determined that in specific and limited circumstances the personal-level information can be disclosed.  The decision resulted in a data sharing proposal and agreement process which allowed projects to request their own EDI file for the duration of their UEY project to do specific analysis as outlined in their proposal. 
  • Some communities’ schools or school boards had ethnic or multicultural populations who were reluctant to participate in the data collection process (EDI and PIDACS). These populations do, however, participate in local UEY project activities.
  • Some communities have experienced challenges in finding the necessary capacity to develop the Community Mapping Report, although there is good indication that all communities either have or will be able to recruit the necessary capacity.

The document review indicated that the UEY Initiative has been pro-active in responding to challenges stemming from the design and delivery of the UEY Initiative. This has included working to resolve process-related issues in the areas of planning/delays, clarity/guidance/tools and communication.

6.2 Progress Toward Immediate Objectives

The early evidence suggests that progress is being made toward the three immediate objectives of the UEY Initiative.

Increased Support for Communities

Evidence from key informants and the file review suggests that progress has been made in providing communities with funding support . Although it is too early to assess the ability to attract additional support, the file review indicated that the UEY Initiative has enabled community coalitions to identify priorities for grants, incorporate UEY into funding proposals, and apply for and secure funding. The evidence also indicated that progress has been made in providing communities with technical assistance , with community mapping being cited as a main catalyst for this progress.

Increased Availability and Sharing of Information

Both the project file review and key informants indicated that UEY communities have been successful at forming or strengthening partnerships as a result of the Initiative. As well, evidence from the document and file review and key informants suggests that the Initiative has increased the number of opportunities and tools for knowledge sharing and networking within communities on the role of families and communities in childhood development . There was also evidence of the sharing of information among UEY communities.

Increased Awareness Regarding Child Development

Most key informants who were program officials felt there has been progress in terms of increasing parental and community awareness and knowledge regarding child development , although it was too early for this progress to be fully evident. Cited examples of progress included the creation of brochures and presentations at workshops.

6.3 Accountability Issues

Despite the array of provincial programs aimed at children, the program scan conducted for the formative evaluation found no evidence of overlap or duplication with the UEY Initiative. In addition, a number of safeguards have been put in place to avoid or reduce the potential for overlap and duplication (e.g the Federal/Provincial/ Territorial Committee on Early Childhood Development Knowledge Information and Effective Practices).

The general view among key informants is that the results-based reporting strategies for the UEY Initiative are appropriate and compatible with federal government accountability requirements. Most key informants who were community representatives expressed confidence in their community’s capacity to undertake the necessary reporting, although most also expressed the desire for more clarity and feedback from HRSDC as they move forward with reporting on the results of data collection. The file review found, however, that two-thirds of project files were missing at least one quarterly report.

6.4 Guidance for the Summative Evaluation

Certain features of UEY were identified as important considerations for the design of the summative evaluation. For example, the Initiative is not typical of the types of programs referred to in the program evaluation literature. Instead it is a funding framework designed to provide communities with local data and capacity to develop information as a way to identify gaps in services and program and to foster partnerships pertaining to child development. As well, the Initiative is designed to permit a high degree of local control over project features and operation.

These aspects of UEY Initiative indicate that the approach developed for the summative evaluation will need to take into account the uniqueness of each community and the impacts of the Initiative on each community as well as the overall impacts of the UEY Initiative. This strongly implies that evaluation strategies will need to be implemented at the level of each community in anticipation of the summative evaluation to ensure that outcomes data are available both at the community level as well as at the level of the overall Initiative.

6.5 Main Areas for Consideration/Improvement

The formative evaluation has identified the following main areas for consideration/ improvement:

  • Some of UEY's long-term objectives may be too broad relative to the scope and size of the UEY Initiative. These objectives should be clarified or modified to better link them to the resources and scope of the UEY Initiative.
  • The UEY Initiative is working to resolve process-related issues (e.g. related to planning, delays) and this work should continue.
  • Consider streamlining parent interviews.
  • The linkage of research data from EDI and PIDACS may better enable research on possible relationships between early childhood outcomes and a number of family and community factors. As well, there is a need for program officials to better communicate with community coordinators and others in the community about the cohort used for the PIDACS and the usefulness of these data. The continued concern expressed by community representatives in this area indicates the need to validate the PIDACS results once they are all available.
  • It would be helpful to further assist communities in completing the Community Mapping report. This could be accomplished in different ways (e.g. additional in-person training sessions, an on-line training module or a helpline).
  • To better support the summative evaluation, the Initiative needs to consistently collect data at the community level on how knowledge gained within the UEY communities is being applied regarding child development. Guidance will need to be provided to the communities on how best to collect these data.
  • To better support the summative evaluation, the UEY Initiative and the UEY sites should ensure that outcomes are linked to measurable indicators and that the necessary data can, and are, consistently collected. Consideration should be given to improving the activity report template to better facilitate the quantification and analysis of UEY project progress/performance. As well, there is a need for improved quantitative baseline data for indicators of intermediate and longer-term outcomes.

Appendix A: UEY Communities

Table A:1 The 21 UEY Community Projects Funded in the Fall of 2005
Province UEY Community
BC UEY Greater Victoria
  UEY Mission
  UEY Okanagan Similkameen
  UEY Sunshine Coast
  UEY Campbell River
  UEY North Shore (North and West Vancouver)
SK UEY Northeast Saskatchewan
MB UEY Division scolaire franco-manitobaine
ON UEY Niagara Region
  UEY Ottawa
  UEY Northern Region of Ontario
  UEY Kawartha Lakes & Haliburton County
  UEY Lower Hamilton
  UEY Northumberland County
  UEY Milton
QC UEY Pointe-de-l’île
  UEY Montréal Chassidic and Orthodox Community
NB UEY Greater Saint John
NS UEY Cumberland County
  UEY Western Nova Scotia
  UEY Halifax West and Area
  • Source: Document and file review.

A second group of 15 communities is being added to the Initiative from the 2006 Call for Proposals.

Report a problem or mistake on this page
Please select all that apply:

Thank you for your help!

You will not receive a reply. For enquiries, contact us.

Date modified: