Language selection

Government of Canada / Gouvernement du Canada

Search

Page 10: Evaluation of the Aboriginal Head Start in Urban and Northern Communities Program at the Public Health Agency of Canada

Appendix C. Data source and evidence matrices

Figure 21: Data source matrix
Issue Question Lines of evidence Data sources
Issue #1: Continued need for program What needs does the program intend to address? Do these needs exist?
  • Program document review
  • Literature review
  • Secondary data analysis
  • Key informant interviews
  • Program files and peer reviewed journals, grey literature, the
    Government of Canada’s reports
  • AHSUNC principles and guidelines
  • Statistics Canada data (e.g. Aboriginal Children’s Survey 2006 and Aboriginal Labour Force Analysis)
  • Public Health Agency’s program staff
Where (geographical setting, age, gender, etc.) are the needs greatest?
  • Program document review
  • Literature review
  • Secondary data analysis
  • Administrative data review
  • Key informant interviews
  • Project files review
  • Statistics Canada data
  • Regional and environmental scans
  • Project level evaluations and regional evaluations
  • Program files and peer reviewed journals, grey literature, the Government of Canada’s reports and performance studies
  • Public Heath Agency’s program staff, key stakeholders and program recipients
  • AHSUNC sites and project level data
  • Program staff
Have needs changed over time and has the program design changed to accommodate these needs?
  • Program document review
  • Literature review
  • Secondary data analysis
  • Key informant interviews
  • Case studies
  • Group interviews
  • Program files and peer reviewed journals, grey literature, Government of Canada’s reports, Statistics Canada data (e.g. census)
  • Public Health Agency’s program staff
  • AHSUNC principles and guidelines
  • Program authorities, program evaluation reports
  • Key stakeholders and program participants
Are there further changes that should be implemented?
  • Program document review
  • Literature review
  • Secondary data analysis
  • Key informant interviews
  • Group interviews
  • Program files and peer reviewed journals, grey literature, Government of Canada’s reports
  • Public Health Agency’s program staff
  • Key stakeholders and program participants
Issue #2: Alignment with government priorities What are the connections between AHSUNC and other Public Health Agency, other federal departments and provincial programming?
  • Program document review
  • Secondary data analysis
  • Key informant interviews
  • Budget and Throne speeches, program files, Government of Canada’s reports, program authorities
  • Public Health Agency’s program staff
  • National Administrative Process Evaluation Survey data (in-kind contributions, partnerships, etc)
Who else delivers similar services to this population?
  • Program document review
  • Secondary data analysis
  • Key informant interviews
  • Case studies
  • Group interviews
  • Budget and Throne speeches, program files, Government of Canada’s reports, federal, provincial/territorial and community reports and websites
  • Program evaluation reports, program participants, the Public Health Agency’s staff, federal government stakeholders and community representatives
Is there overlap and/or duplication or complementarities in service delivery?
  • Program document review
  • Secondary data analysis
  • Key informant interviews
  • Case studies
  • Group interviews
  • Budget and Throne speeches, program files, Government of Canada’s reports, federal, provincial/territorial and community reports and websites
  • Program evaluation reports, program participants, the Public Health Agency’s staff, federal, government stakeholders, community representatives
Are there opportunities for alternative service delivery mechanisms (e.g., provincial or local delivery / partnerships)?
  • Program document review
  • Literature review
  • Key informant interviews
  • Case studies
  • Group interviews
  • Program files, peer reviewed journals, grey literature, Government of Canada’s reports, federal, provincial and territorial and community reports and websites
  • Program participants, the Public Health Agency’s staff, federal government stakeholders, community representatives
Is there a common understanding among stakeholders of the objectives of the program and the role of PHAC?
  • Program document review
  • Key informant interviews
  • Case studies
  • Group interviews
  • Program files and reports.
  • Program participants, the Public Health Agency’s staff, federal government stakeholders, community representatives
Issue #3: Alignment with federal roles and responsibilities Should the federal government be delivering the services provided by AHSUNC?
  • Program document review
  • Literature review
  • Key informant interviews
  • Budget and Throne speeches, Government of Canada’s reports, Health Canada and the Public Health Agency’s annual reports, Report on Plans and Priorities and Strategic Plans
  • Public Health Agency’s managers and staff, Policy division, Legal services
Is the AHUSNC program consistent with the current PHAC mandate and strategic priorities?
  • Program document review
  • Secondary data analysis
  • Key informant interviews
  • Budget and Throne speeches, the Government of Canada’s reports, the Public Health Agency of Canada’s Strategic Plan and Reports on Plans and Priorities, Departmental Performance Reports
  • Public Health Agency’s managers and staff, Policy division, Legal services
Is there overlap and/or duplication or complementarities in service delivery?
  • Program document review
  • Secondary data analysis
  • Key informant interviews
  • Case studies
  • Group interviews
  • Budget and Throne speeches, program files, the Government of Canada’s reports, federal, provincial and territorial and community reports and websites
  • Program evaluation reports, program participants, the Public Health Agency’s staff, federal government stakeholders, community representatives
Issue #4: Achievement of expected outcomes Has the program been implemented as planned?
  • Program document review
  • Literature review
  • Secondary data analysis
  • Project files review
  • Administrative data review
  • Key informant interviews
  • Case studies
  • Group interviews
  • Program files, site visit reports, program authorities
  • Program participants and community stakeholders, the Public Health Agency’s staff, federal government stakeholders, community representatives
  • Data trends in program reach over the last five years
  • Financial data
  • Process Survey
What are the challenges and barriers to program delivery and how can the design be improved to address these challenges and/or barriers?
  • Program document review
  • Literature review
  • Key informant interviews
  • Case studies
  • Group interviews
  • Program files, routine site visit reports by program consultants, program authorities
  • Program participants, the Public Health Agency’s staff, federal government stakeholders, community representatives
  • National Administrative Process Evaluation Survey data
  • Regional Evaluations
Are there opportunities for more effective program delivery?
  • Key informant interviews
  • Case studies
  • Group interviews
  • Program participants, the Public Health Agency’s staff, federal government stakeholders, community representatives
Is the AHSUNC program achieving its expected outcomes and what are the requisite conditions for success?
  • Program document review
  • Secondary data analysis
  • Project files review
  • Administrative data review
  • Key informant interviews
  • Case studies
  • Group interviews
  • May be addressed by the following: National Administrative Process Evaluation Survey data, program files, regional office interviews, performance studies, community stakeholders and case studies
Does the program reach the intended target population (s)?
  • Program document review
  • Literature review
  • Secondary data analysis
  • Project files review
  • Administrative data review
  • Case studies
  • Program authorities, program files for intended demographic
  • Spatial analysis of the AHSUNC sites in relation with target population.
  • Project level data and site level data
  • National Administrative Process Evaluation Survey data
What are the impacts and/or value added of the program, on individuals, families and the community? What works for whom and under what circumstances?
To what extent has the program built or increased capacity in urban and northern communities since its inception?
Issue #5: Demonstration of efficiency and economy How were funds used to support project outputs and planned outcomes?
  • Program document review
  • Project files review
  • Key informant interviews
  • Case studies
  • Program authorities
  • Financial documents
  • Program files
  • PHAC staff
  • AHSUNC Sites
  • Meeting minutes (National Aboriginal Head Start Council and program networks)
What is the cost per site (salary, operation and maintenance, capital)?
  • Program document review
  • Project files review
  • Administrative data review
  • Case studies
  • Financial documents
  • Program files
To what extent has the program leveraged in-kind resources and volunteers?
  • Project files review
  • Administrative data review
  • Key informant interviews
  • Case studies
  • AHSUNC Sites
  • National Administrative Process Evaluation Survey data
  • Regional reports
  • Case studies
How are non grants and contributions-related funds (e.g. salary and operation and maintenance) used within the program? Is there a variance between planned vs. actual expenditures?
  • Program document review
  • Administrative data review
  • Key informant interviews
  • Program files
  • Public Health Agency’s staff
  • Financial data
Is program performance measurement adequate?
  • Program document review
  • Project files review
  • Key informant interviews
  • Program authorities
  • Public Health Agency’s corporate risk assessment, program risk assessment, audits
  • Performance measurement and evaluation tools used at project, regional and national levels
  • AHSUNC staff, regional program consultants, project level staff, managers and directors
Figure 22. Findings and evidence matrix
Relevance questions: What needs does the program intend to address? Do these needs exist? Where (geographical setting, age, gender, etc.) are the needs greatest? Have needs changed over time and has the program design changed to accommodate these needs? Are there further changes that should be implemented?
Findings Evidence
There is a continued and growing need for the Aboriginal Head Start in Urban and Northern Communities (AHSUNC) program. Population growth rates are higher for Aboriginal people than for the general Canadian population, and a shift towards urban settings has increased the need for early childhood development programming off-reserve. Seventy per cent of young Aboriginal children live off-reserve and of those, 80 per cent live in urban centres. A young and growing population is also contributing to a strong need for the program in northern Inuit communities.

Aboriginal children are at higher risk for poor developmental and health outcomes due to several factors. In Canada, Aboriginal children:
  • are overrepresented in the child welfare system
  • experience higher levels of moderate and severe food insecurity (33 per cent) than non-Aboriginal populations (nine per cent)
  • are twice as likely to experience poverty as the general Canadian population
  • are two to three times more likely than non-Aboriginal Canadians to be raised by young, single parents.
Continued need for the program is also evidenced by the long wait lists for participation. Over the last four years, there was an average of 1,300 children per year waiting for a spot in the AHSUNC program. This high level demand likely reflects the limited number of free and culturally sensitive programs in Canada.
Relevance questions: Should the federal government be delivering the services provided by AHSUNC? Is the AHSUNC program consistent with the current PHAC mandate and strategic priorities? What are the connections between AHSUNC and other PHAC programs (for example: FASD, Healthy Living, Mental Health, CPNP, CAPC, Modernization of Children Programs, Northern Health Portfolio, PHAC First Nations, Inuit and Métis Public Health Policy Framework) and Health Canada, First Nations and Inuit Health Branch? Who else delivers similar services to this population? Is there overlap/ duplication or complementarities in service delivery? Are there opportunities for alternative service delivery mechanisms (e.g. provincial or local delivery/partnerships)? Is there a common understanding among stakeholders of the objectives of the program and the role of PHAC?
Findings Evidence
Delivering the AHSUNC program is an appropriate role for the federal government and the Public Health Agency. The Government of Canada is not prohibited from investing in areas that are primarily within provincial and territorial jurisdiction and thus, often makes policy decisions to do so. For example, while provinces and territories have the primary responsibility for early childhood development, the Government of Canada plays a supporting role.

Typically, when investing in an area of shared jurisdiction with the provinces and territories, the federal government's role is to:

  • address an issue of national scope that it is beyond the capacity of any particular province or territory to address by itself
  • fill gaps for a vulnerable population
  • collaborate with provinces and territories to complement provincial or territorial directions.
The AHSUNC program meets all of these criteria.

Address an issue of national significance:
There are well-documented inequalities between Aboriginal people and the general population of Canada and action is required to close the gap and reduce disparities in well-being.

Fill gaps for a vulnerable population:
The AHSUNC program is a targeted initiative for a vulnerable population and was recognized as such by the Chief Public Health Officer's Report on the State of Public Health in Canada (2009).

The AHSUNC program supplements provincial and territorial investments in the area of early childhood development programming. Despite increased provincial and territorial investments that have resulted in enhanced availability and accessibility of early childhood development programming in Canada, there is still more demand for these programs than can be met at this time. Information collected from case studies indicates that Aboriginal parents may choose not to access provincial early childhood development programs available to them due to cultural irrelevance. When children attend either poor or no preschool programs, the outcome can be weaker cognitive, social, language and motor skills than children who attend these programs.

Complement provincial and territorial programs:
Most AHSUNC projects align with provincial and territorial standards and regulations for operating early childhood development centres. At the site level, AHSUNC project staff partner with provincial and territorial staff in public health, social services and education to ensure the delivery of complementary, efficient and effective programming.

The Public Health Agency's mission is to promote and protect the health of all Canadians, including Aboriginal peoples, through leadership, partnership, innovation and action on public health. The Public Health Agency fulfills this mission in a number of different ways, one of which is the funding of community-based public health programs through grant and contribution transfer payments. Given the alignment of the AHSUNC program with the federal role described above, delivering this community-based program is a legitimate and appropriate role for the Public Health Agency. However, funding community-based programs is not the Public Health Agency's only role, as the Public Health Agency is also expected to provide leadership, collaboration and coordination on matters of national public health significance.
The AHSUNC program aligns with the Public Health Agency of Canada's mandate and strategic priorities, particularly with respect to Northern and Aboriginal public health. The Public Health Agency's mandate includes health promotion.

As well, the Public Health Agency has an established role to strengthen intergovernmental collaboration on public health and facilitate national approaches to public health policy and planning.

AHSUNC aligns with the Public Health Agency's strategic priorities and mandate, by directly addressing social determinants of health in a known vulnerable population.

Aboriginal public health, including the North, was determined to be one of the Public Health Agency's top six strategic priorities in an Executive Committee retreat discussion on policy priorities in January 2011.

Aboriginal public health was subsequently emphasized as a top priority in a Public Health Agency Resources, Planning and Management committee meeting.
Performance questions: Has the program been implemented as planned? What are the challenges/barriers to program delivery and how can the design be improved to address these challenges/barriers? Are there opportunities for more effective program delivery? How were funds used to support project outputs and progress to achieving planned outcomes? What is the cost per site (salary, O&M, capital)? To what extent has the program leveraged in-kind resources and volunteers and support from other funders? How are non G&C-related funds (e.g. salary and O&M) utilized within the program? Is there a variance between planned vs. actual expenditures? Is the AHSUNC program achieving its expected outcomes and what are the requisite conditions for success? Does the program reach the intended target population(s)? What are the impacts/value added of the program, on individuals, families and the community? What works for whom and under what circumstances? To what extent has the program built or increased capacity in urban and northern communities since its inception? Is program performance measurement adequate?
Findings Evidence
The program has had a positive effect on school readiness, specifically in improving children's language, social, motor and academic skills. Performance results have also demonstrated effectiveness in improving cultural literacy and enhanced exposure to Aboriginal languages and cultures. There are also positive effects on health promoting behaviours such as children's access to daily physical activity as well as determinants of health, such as access to health and dental care. The program has not systematically assessed results related to knowledge development and exchange, collaboration with Aboriginal early childhood development programs or support to parents and families. AHSUNC has had a measurable, positive effect on participating children's language, social, motor and academic skills and cultural literacy. In addition, the culture and language component has increased participants' exposure to Aboriginal languages and cultures; this component viewed as key to fostering a sense of pride and identity for parents and children. The program has positive effects on children's access to daily physical activity and health and dental care. In addition there is some evidence that participation increases the frequency with which children consume healthy foods. Parental engagement varies from project to project and the impact on parents has not been consistently measured. However several lines of evidence indicate that some parents have experienced benefits such as new knowledge, social support and access to employment. Although the program's impact on communities has not been consistently tracked, there is evidence that some of the AHSUNC project sites have become known as the community 'hub', creating a sense of community for Aboriginal children and their families. Although there are a few regional examples of alignment with provincially or territorially funded early child development systems or the school system, overall there was limited evidence of this outcome. While knowledge development and exchange is underway at the community level, this has not been measured. There was limited evidence of the dissemination of knowledge to influence policy or practice within the Public Health Agency or among other federal, provincial and territorial government departments, or national Aboriginal organizations. Some notable exceptions were identified. There was limited evidence of integration with provincially run early child development programming and Health Canada's Aboriginal Head Start On-Reserve program and links to training and research institutions, thus limiting the potential reach and influence of AHSUNC and potential enrichment of AHSUNC sites
There are a variety of models across the country for delivering the AHSUNC program. Three delivery models were observed: (1) a project site operating independently, (2) a project that operates within (co-located with) an early childhood education activity hub, (3) project activities and funding are completely integrated within a broader early childhood development program.
Over the last five years, implementation of the National Strategic Fund has been challenging. The evaluation re-viewed formal documents and conduct an analysis of the financial data related to this fund over the last five year, and found that the fund allocation was not totally spent for any of the five years covered by the evaluation.

Interviews and program documents revealed the difficulties for the recipients to produce propositions to be funded that would meet the criteria for the funding of a project.

Document review confirmed that the objectives of the National Strategic Fund are clearly enunciated.

Multiple sources confirmed that there is a debate about opening the Fund to other National Aboriginal Organizations and point to diverging opinions on this subject.

While the Regional Enhancement Strategic Fund has been assessed to measure the impact of the initiatives funded over the last five years as well as the management of the fund, the National Strategic Fund was never assessed.
Resource leveraging and partnerships at the site level are strengths of the program. However, short-term funding agreements and staff turnover, as well as enrolment levels have led to some inefficiencies in program delivery. Several elements of the program have been implemented as planned and were identified as strengths, including: the six program components, community-based projects delivered by Aboriginal organizations, some projects' alignment with schools, projects' responsiveness to local needs and the provision of transportation.

The contribution funding allocated for funding AHSUNC projects (both ongoing projects and time-limited enhancement projects) was spent in entirety each year of the evaluation timeframe, with the exception of funding for national strategic projects. Additional contribution dollars were transferred into the program to supplement available funds. This demonstrates that the program is able to effectively spend its allocated funding and absorb additional funds.

Two aspects of program delivery that contribute to efficiency are the presence of committed champions at all levels of the program and the projects' abilities to leverage funding and in-kind resources. Conversely, four aspects of the program that reduce efficiency are the use of short-term contribution agreements with late funding announcements, varying enrolment levels, high project staff turnover and concerns related to the National Aboriginal Head Start Council.
The advisory function that supports program governance could be improved. Originally, the National Aboriginal Head Start Council was intended to provide advice to the Public Health Agency on policies, procedures, services and programs related to AHSUNC (curriculum development, standards for program performance, research priorities and cultural and traditional values). Evaluation findings indicate that the National Aboriginal Head Start Council is not currently serving the role that was originally intended.

Management and staff at the Public Health Agency also expressed concerns about the lack of clarity of the National Aboriginal Head Start Council mandate as well as concerns about the effectiveness of the current advisory function.

Possibly contributing to the lack of clarity around the role of the National Aboriginal Head Start Council is the composition of the Council. Several findings suggest that this composition may not be ideal.
Reach is limited; this may be due in part to the design of the program and the delivery model. AHSUNC is reaching the intended population (First Nations, Inuit, Métis preschool children) and in 2010-11 the program reached approximately eight per cent of the off-reserve population of three to five-year olds.

Information from multiple sources suggests that the program has the potential to reach more children. While some sites are not full, in 2009-10 close to 50 per cent of AHSUNC sites had a waiting list, which suggests that there is higher demand in some communities than is currently being met. It appears that site locations and the delivery model have not been reviewed to explore options to adapt to shifting needs and demand.
Coordination with similar early childhood development programs within the Public Health Agency is good, but coordination needs to be strengthened with other federal departments and the provinces and territories. The evaluation found that there are varying levels of coordination within the program.

The AHSUNC program appears to be well coordinated with other children's programs within the Public Health Agency, Centre for Health Promotion.

There does not appear to be any formal operational connection between the two streams of Aboriginal Head Start programming within the Health Portfolio (Head Start on- and off-reserve). No formal coordinating mechanisms were found to be in place at any level.

The evaluation found no evidence of formal links to knowledge development partners within the Health Portfolio, such as the Canadian Institutes for Health Research, or the National Collaborating Centre for Aboriginal Health (which receives transfer payments from the Public Health Agency).

The evaluation revealed no evidence of systematic coordination between the Public Health Agency and other federal departments delivering similar programs, namely Aboriginal Affairs and Northern Development Canada and Human Resources and Skills Development Canada. Within Public Health Agency's Regional Operations, provincial or regional advisory committees are established to provide advice and guidance on the implementation of the AHSUNC projects. Federal partners may be included in some of these committees, but this is not consistent across the country, nor systematically coordinated at the national level to ensure federal dialogue and exchange at the program level. Furthermore, there is no evidence of the Public Health Agency taking part in partnership tables chaired by other federal departments.

The evaluation found no evidence of systematic coordination between the Public Health Agency's AHSUNC program and other provincial or territorial government programs. The exception to this would be the inclusion of provincial and/or territorial representatives in some, but not all, provincial and regional AHSUNC advisory committees.

At the site level however, there is evidence of consistent partnering with local service providers, health authorities and other early childhood development stakeholders.
Performance measurement for this program is extensive, but could benefit from more consistent and comprehensive pan-regional initiatives and streamlined approaches and tools. Extensive performance information is available about project-level activities, processes and reach to children, however there was limited evidence of dissemination of this information internally and externally.

The measurement of children's short- and medium-term outcomes for school readiness is consistent and of high quality.

Performance measurement work does not appear to be guided by an overarching framework that guides periodic studies and demonstrates a strategic approach to assessment of intermediate and long-terms outcomes. Program outcomes related to knowledge development and exchange, support to parents and families and collaboration with early childhood development programs have not been systematically assessed. There is a need to explore the conditions that promote desired outcomes (e.g. what contexts and factors promote parental involvement and what impact does this have of child and family outcomes). As a result, there is a lack of knowledge about what works for whom and under what conditions.

Page details

Date modified: