Page 8: Evaluation of the Aboriginal Head Start in Urban and Northern Communities Program at the Public Health Agency of Canada

Appendix A. Technical annex

Methodology

Approach and design

This ex-post (retrospective) evaluation was conducted by Evaluation Services, a group that is internal to the Public Health Agency but that is not involved in the program area responsible for administration of the Public Health Agency’s AHSUNC activities.

The overall approach to this evaluation integrated two conceptual evaluation models: goals-based and process-based.

A goals-based model allowed the evaluators to explore whether the program was meeting its stated goals and objectives as set out in the original program mandate. Given that the AHSUNC program has not been evaluated at the national level since 2006 (prior to the release of the Policy on Evaluation from the Treasury Board of Canada Secretariat), the initial purpose of the evaluation was to look at (a) program relevance and whether or not program goals should be changed, as well as (b) how program goals should be established in the future.

The other purpose of the evaluation was to fully understand how the AHSUNC program is actually working today, so a process-based evaluation model was also appropriate. While the program has evolved since its inception in 1995, this evaluation provided an opportunity to explore its delivery.

Evaluation Services used a non-experimental cross-sectional design to take a retrospective look at this long-established program. Since little baseline information was available, this design allowed evaluators to analyze information at one point in time from multiple sources. The evaluation covered the performance of and “value added” by the program, as well as the alignment of program activities with the Public Health Agency’s roles and priorities.

The timing of this evaluation was driven by the requirement under the Treasury Board of Canada Secretariat’s Policy on Evaluation that all ongoing programs of grants and contributions be evaluated every five years, as required by section 42.1 of the Financial Administration Act.^{Footnote 3535}

There were a number of evaluation risk factors. In particular, the long history of this program (over 15 years) and the scarcity of evaluative analysis at the national level related to program relevance undertaken over the life of the program were determining factors in the evaluation methodology selected.

In addition to the perspectives of a broad cross section of program management and staff, major stakeholders’ perspectives were also considered. In particular, the evaluators sought out the perspectives of recipients of the services provided through the AHSUNC program, including the National Aboriginal Head Start Council and program site coordinators. Other government departments and organizations were also consulted.

Methods

The evaluation triangulated data from multiple lines of evidence.

Literature review

A literature review was conducted to explore the nature of early childhood development in Canada and internationally. The review addressed evaluation issues related to relevance. The purpose of this review was to obtain information regarding the need for the program, including whether it represents a legitimate and necessary role for the Government of Canada, as well as to discover best practices that may suggest potential improvements to the design and delivery of the AHSUNC program.

In total, 93 documents were reviewed. These documents included all of the available literature suggested by program staff, including grey literature, as well as documents identified through Internet searches, surveys from Statistics Canada and bibliographic key word searches. Evaluators conducted bibliographic key word searches for recent (last five years) academic publications in English and French, focusing on literature about the health and social needs of the off-reserve Aboriginal population in Canada, the concept of school readiness and the effects of early childhood education.

The literature review also included a summary of early childhood development programs in three other countries: the United States, New Zealand and Australia.

Document review

The evaluation team undertook a review of available AHSUNC program documentation. Many documents were identified by program management and staff. This review was used to address evaluation issues related to both relevance and performance. In total, 94 documents were reviewed by the evaluation team.

The types of documents reviewed included:

• regional evaluations of the AHSUNC program completed over the last five years
• previous audits and reviews of the AHSUNC program
• records of decision from various internal committee meetings
• correspondence and communication related to the AHSUNC program
• program performance measurement data, including National Administrative and Process Evaluation Surveys, program site mapping and the Brigance Preschool Screen report.

Interviews

The evaluation team conducted interviews with key senior managers and staff from the Public Health Agency, members of the National Aboriginal Head Start Council and AHSUNC contacts with other federal government departments and organizations.

The purpose of the interviews was to obtain a further description of how various elements of the program have been implemented as well as receive input on program relevance and performance.

The interviews were conducted using structured interview guides. The interview guides were tailored to each group of interviewees.

AHSUNC program management and staff identified key informants for the evaluation. The sampling approach was purposive. Evaluation Services notified key informants of the evaluation and requested their participation. From September 15, 2011 to February 07, 2012, 18 one-hour interviews were conducted with 49 interviewees. Several interviews involved more than one key informant.

Case studies

For information on how the case studies were conducted please see Appendix B. A complete technical report on the case studies is available upon request from the Public Health Agency’s Evaluation Services.