Page 4: Evaluation of the Aboriginal Head Start in Urban and Northern Communities Program at the Public Health Agency of Canada

1. Introduction

1.1 Purpose and scope of evaluation

The Aboriginal Head Start in Urban and Northern Communities (AHSUNC) program was last evaluated by the Centre for Health Promotion in 2005-06. The Financial Administration Act^{Footnote 22} and the Treasury Board of Canada Secretariat’s Policy on Transfer Payments require departments to conduct a review every five years of the relevance and effectiveness of their grants and contributions programs, and the 2009 Treasury Board of Canada Secretariat’s Policy on Evaluation requires comprehensive evaluation coverage of all program spending on a five-year cycle. Consequently, the AHSUNC program has been re-evaluated for the period 2006-07 to 2010-11.

Evaluation questions address the relevance (continued need for this program and alignment with federal government roles and priorities) and performance (achievement of expected outcomes and demonstration of efficiency and economy) of the entire AHSUNC program, including its grants and contribution component.

The purpose of the evaluation is to explore and recommend program changes to help improve effectiveness in achieving desired outcomes for Aboriginal children, families and communities. In its 15-year history, AHSUNC’s stated objectives and program scope have remained fairly consistent. The evaluation includes a review of the original intent and plans for the program to determine whether the program is being delivered as planned, whether there are better ways to deliver the program, and whether there should be a shift in the delivery of the program to better address emerging Government of Canada and Public Health Agency priorities. The evaluation provides the opportunity to learn from current experiences, best practices and alternative program delivery models.

1.2 Methodology

The evaluation was conducted by Evaluation Services, a group that is internal to the Public Health Agency but is independent of the AHSUNC program.

Evaluation Services analyzed information from multiple sources:

• key internal documents, including:
  • regional-level evaluations of the AHSUNC program as well as previous audits and reviews of the AHSUNC program
  • records of decision from various internal committee meetings
  • correspondence and communication related to the AHSUNC program.
• case studies of various AHSUNC sites across Canada, as well as non-AHSUNC sites; a complete technical report on the case studies is available upon request from the Public Health Agency’s Evaluation Services (evaluation@phac-aspc.gc.ca)
• interviews with key senior managers and staff from the Public Health Agency, selected other government departments, non-governmental organizations and the National Aboriginal Head Start Council
• literature reviews including lessons learned on international experiences.

This evaluation triangulates multiple lines of evidence and a combination of qualitative and quantitative measures to ensure a balanced analysis of the relevance and performance of the AHSUNC program. For a detailed description of methodology, see Appendix A. Appendix B elaborates on the methodology used for the case study component. Two evaluation matrices that outline evaluation questions, lines of evidence and findings can be found in Appendix C.

Limitations

Most evaluations are confronted with constraints that may have implications for the validity and reliability of evaluation findings, conclusions and recommendations. This section identifies the limitations with respect to the design and methods for this particular evaluation. Also listed are the mitigation strategies put in place by the evaluation team to ensure that the evaluation findings can be used with confidence to guide program planning and decision making.

Finite data collection

The scope of the data collection process was defined in accordance with available resources and time allocated and therefore a limited number of interviews were conducted with external stakeholders (i.e. provincial and territorial representatives and external experts). More interviews with those stakeholders could have provided greater insight into alignment of roles, and program delivery. However, through literature and document reviews, as well as case studies and internal interviews, the evaluators were able to gain a general understanding of these issues.

Validity of qualitative data sources

The interviews and focus groups were partly retrospective. The validity of this information could be questioned as a result of such things as staff turnover and challenges with memory or recall. In some cases, program staff turnover limited knowledge of the history of the AHSUNC program. To mitigate the challenge of staff turnover, the evaluators conducted interviews with staff and external stakeholders who were involved when the program began.

Lack of common framework for regional evaluations

Although the secondary data analysis incorporated multiple regional-level AHSUNC program evaluation reports into the assessment of program performance, the design and intent of the various regional evaluations was inconsistent. This lack of consistency limits the degree to which findings from individual regional evaluations can be extrapolated to the national program as a whole.

Despite these potential methodological limitations, the evaluation team implemented a number of strategies to ensure the validity of the findings, particularly the use of multiple sources of evidence. In addition, there was national consistency in the framework used to measure school readiness and national data to inform cultural literacy findings. Therefore Evaluation Services is confident that senior management can rely on this report for planning and decision-making purposes.

Scope of case studies

The case study data collection in communities was limited to actual project sites and did not extend to other community sources. Although case studies were not conducted at sites administered by the Northern Secretariat (Health Canada), two projects that reach Inuit children were included in the sample and all Public Health Agency regions were represented.