ARCHIVED - Chronic Diseases in Canada

Volume 30, no. 2, March 2010

Alberta Aboriginal Head Start in Urban and Northern Communities: longitudinal study pilot phase

A. M. dela Cruz, MSc, BN (1); P. McCarthy, Associate Degree in Applied Science (2)

https://doi.org/10.24095/hpcdp.30.2.02

Author References

1. Evaluation Consultant, Public Health Agency of Canada, Alberta Region
2. Chair, Alberta Aboriginal Head Start Provincial Committee

Correspondence: Añiela M. dela Cruz, Public Health Agency of Canada, Alberta Region, 620 Harry Hays Building, 220 – 4th Avenue SE Calgary, Alberta T2G 4X9, Tel.: (403) 292-6714, Fax.: (403) 292-6696, E-mail: aniela.dela.cruz@phac-aspc.gc.ca or adelacru@ualberta.ca

Abstract

Aboriginal Head Start in Urban and Northern Communities (AHSUNC) is a federally funded, national, early childhood intervention strategy that addresses the needs of Aboriginal preschool children and their families. A pilot study, based on principles of community-based research, evaluated an Alberta Aboriginal Head Start (AHS) program offered off-reserve in Alberta. Overall, the results pertaining to children having followed an AHS program are positive. This phase 1 of a broader longitudinal evaluation study of all AHS sites in Alberta has led to the creation of several recommendations, which reinforce this type of evaluation and look to mitigate the limitations encountered in phase 1 (around available data, tools and context).

Keywords: Aboriginal Head Start, longitudinal evaluation study, community-based research, early childhood intervention strategy, Alberta

Introduction

Aboriginal Head Start in Urban and Northern Communities (AHSUNC) is a federally funded, national, early childhood intervention strategy that addresses the needs of Aboriginal preschool children and their families living in urban centres and in northern communities. It strives to instil a sense of pride and confidence, to foster a desire to learn and emotional and social development, to provide parenting skills and to improve family relationships. In Canada, Aboriginal Head Start (AHS) programs are administered both on- and off-reserve. Funded by the Public Health Agency of Canada (PHAC), there were (as of June 2008) 16 sponsors who were implementing 21 AHS programs across Alberta.

AHS programs in Alberta typically provide a preschool experience for children to prepare them for their school years, and provide opportunities for spiritual, emotional, intellectual and physical development. From a population health perspective, early childhood intervention strategies contribute to the health and well-being of children in the short- and long-term; they affect children’s school readiness and health, and ultimately their health and well-being as adults. In contrast, poor childhood development can result in poor physical and mental health, including chronic diseases, throughout life.1

All AHS sites provide programming in six core areas: education and school readiness, Aboriginal culture and language, parental involvement, health promotion, nutrition, and social support. Class sizes range from about 10 to 20 students, and programs are implemented either as half-day (morning or afternoon) or full-day programs. Also, AHS directly involves parents and the community in the management and operation of projects; parents are supported in their role as the child’s first and most influential teacher, and the wisdom of Elders is valued.

All AHS sites in Alberta are members of the Alberta Aboriginal Head Start (AAHS) Provincial Committee. This committee includes all AHS sponsors in Alberta and representatives from the provincial and federal governments, including regional PHAC staff. In 2007, the AAHS Provincial Committee identified the need to conduct an evaluation of AHS programs in Alberta,² and a consultant was hired to develop different options and approaches to conduct a longitudinal evaluation study. In 2008, further discussion took place within the AAHS Provincial Committee regarding the opportunity and potential to conduct a longitudinal evaluation study that would examine the impact of the AHS program by comparing development among children who attend AHS programs and those who do not, thus determining whether the program is, in fact, working as intended.

The committee decided on a community-based research approach to ensure that the relevance of the research topic was identified and verified by community members; that the resources for research (finances, expertise and personnel) were shared with community members and key stakeholders, particularly those most affected by the study topic; that the research process recognized and utilized the expertise that community members have; and that the research process and results were accessible to and understood by community members.* Such principles are also aligned with those created by the National Aboriginal Health Organization (NAHO): ownership, control, access, and possession (OCAP).3a

Purpose and objectives of the AAHS longitudinal study pilot phase 2007/2008

Currently, Alberta AHS sites are involved in evaluation-related activities that help inform programming improvements. However, these evaluations focus more on process-oriented measures. Here we report on the results of a pilot study that will help plan for a broader longitudinal study in Alberta that would examine the impacts of the AHS program.

The purpose of this study was to conduct a pilot longitudinal evaluation study in one Alberta AHS site to assess the feasibility of longitudinal evaluative studies of AHS programs in Alberta and to plan for future studies.

There were three objectives of the AAHS Longitudinal Study Pilot Phase 2007/2008: to develop and implement an evaluation plan for the longitudinal pilot study that assesses areas such as school achievement, social skills and school readiness among Alberta AHS graduates; to develop an evaluation plan that is consistent with a community-based approach; and, based on the pilot results, to develop and present an evaluation plan for a broader longitudinal study of the Alberta AHS program for 2008–2012, including an estimate of the budget and resources required to complete a full study.

This pilot study was part of Phase 1 of the longitudinal evaluation study of the AHS program in Alberta. It is anticipated that a broader evaluation study will be conducted across all AHS sites in Alberta in the next three to five years.

Methods

A research team was hired, based on their extensive experience and knowledge of Canadian AHS programs and Aboriginal communities, to assist in the implementation of the pilot study. Their recommendation to use a post-test design to conduct the pilot study was based on budget limitations, time constraints (the pilot study was funded by the Alberta Regional PHAC office and thus all pilot study activities had to be completed within the fiscal year, as well as within the AHS school year), and the lack of baseline or pre-test data, which meant that pre-test and post-test outcome results could not be compared.

Sample and participants

All Alberta AHS sites were given the opportunity to participate in the pilot study; some chose not to due to conflicts with school activities and an inability to commit to the particular timeframe. This AHS site was chosen based on the school’s voluntary participation as well as the support for the pilot study offered by the school administration.

The AHS pilot study was based in a small rural off-reserve community of approximately 550 people 400 kilometres northwest of Edmonton, Alberta, where the AHS site has been in operation for approximately 11 years. Situated in the local school, the AHS site shares the school’s resources such as gymnasium space and food programs. The program is child-centered with emphasis on language development, socio-emotional well-being and learning through play. At the time of the pilot study, there were 20 children aged 3 and 4 years attending the AHS program. Nearly all children who attend the AHS program continue with their education at this school.

Of the 64 children enrolled at the AHS site, 48 (75%) participated in the pilot study. Reasons for non-participation of students included non-attendance and lack of informed consent.

The average age of the Early Childhood Services (ECS) children who participated in the pilot study was 5.7 years for children who completed AHS and 5.2 years for those who did not attend AHS, or Non-Aboriginal Head Start (N-AHS) children. In Grade 1 the average age was 6.5 years for AHS children and 6.6 years for N-AHS children; in Grade 5 the average age was 10.5 years for AHS children (there were no N-AHS in Grade 5); in Grade 6 the average age was 11.6 years for AHS children and 12 years for N-AHS children. An equal number of girls and boys participated in the pilot study.

Measuring instruments

The measures used in the pilot study were selected based on the need to sample children’s literacy, achievement in school, prosocial skills and receptive vocabulary (a measure of learning potential as well as verbal skills and emerging literacy); on their use with culturally distinct communities, including other AHS sites in Canada; and on budget.

Three standardized and norm-referenced measures were used: the Wechsler Individual Achievement Test (WIAT-II-A),4 which assesses word reading, math calculation and spelling; the Social Skills Rating Scale (SSRS),5 a rating scale tool used by teachers to identify prosocial behaviours and problem behaviours; and the Peabody Picture Vocabulary Test (PPVT-IV),6 which assesses children’s literacy potential through non-verbal measurement.

Design

Data collection was scheduled for three school days, February 26, 2008, through February 28, 2008. This timeline, chosen in cooperation with the school and the AHS coordinator, had few scheduling conflicts such as holidays or report card timelines.

School staff members were briefed about the pilot study and adjusted their class schedules to accommodate the 10 to 30 minutes that each participant student would be away from the class. A group of three to four students were taken from each classroom at a time; two would work with two researchers, while the other pair waited their turn. This format provided for efficient use of time with each researcher, as children would take different lengths of time to complete the tasks.

A classroom assistant helped with the continuous and smooth transition of participants from their classrooms to the data collection room, a practical and necessary part of the data collection procedures; a liaison person explained the procedure to the students to help them accept these, and assisted them as necessary; school administrative personnel confirmed receipt of parental consent forms and provided students’ dates of birth for data analysis.

Child-friendly data collection methods (encouraging, fun, game-like) were used to maximize data collection and to make the experience non-threatening for the participants. The data were collected by researchers with applicable training and skill in the measurement tools. Further, the involvement and assistance of community members during data collection ensured that evaluation activities were culturally relevant and meaningful, and that they understood the process and methods, and could explain these should any questions arise.

Data analysis

The researchers ensured that data analysis adhered to the ethical and evaluative principles and sanctions of established evaluative and psychological governing bodies, such as the Canadian Evaluation Society and the Canadian Psychological Association.^† Analysis included the use of the statistical software, Statistical Package for the Social Sciences (SPSS).⁷ The statistical tests included a full range of descriptive statistics, sample size determinations, one-tailed t-tests to determine differences between groups, case-matched differences and one-tailed t-tests, ANOVA for grade differences, and two-tailed t-tests for gender and social skills differences. Qualitative data such as contextual factors, observations and process findings from the data collection were analyzed and summarized by the researchers.

Ethical considerations

A number of ethical considerations were of central importance throughout the pilot study. Participation was completely voluntary. All data collection activities required informed consent: parents/guardians were given a consent form to complete and had the right to refuse permission. Parents also had the opportunity to discuss any questions or concerns about the pilot study with school staff and the researchers. They could also seek out their individual child’s results, which would be explained to them by the researchers. Confidentiality was maintained throughout, and the identities of the children were protected; participant results were not shared or even filed in school records. Group results were reported without revealing the identity of the participating children or their families.

Finally, the researchers integrated the OCAP^3b principles into the pilot study in order to ensure that the work was respectful and consistent with community values and traditions. Specific procedures in terms of de-briefing the staff of the pilot site, providing parents/guardians access to their child’s results and ensuring the ownership of data were integrated in this project’s methods. At the end of the pilot study, the researcher and her team provided a preliminary report of their findings to the site, the principal and the AHS coordinator. This ensured that results were given back to the local community for their use and for future program planning.

Results

Results of the pilot study were described in two main ways: the results of the outcome measures taken among children who had graduated from the Aboriginal Head Start program at the pilot site, and results pertaining to the feasibility of the methodology used and the feasibility for future longitudinal evaluation studies in Alberta.

The researchers found encouraging results in relation to the outcome measures completed among AHS graduates and observed that the AHS program is a highly beneficial intervention. Pilot study participants across all four grades scored in the average range (mean = 98.3) on the PPVT-IV, as compared to age-matched peers, indicating that the AHS children are performing as well as their age-equivalent peers on tasks that measure verbal abilities and language development (Table 1).

The pilot study participants scored in the average range on the WIAT-II-A for the three measures of academic skills, reading, numeracy and word writing (Table 2). The mean scores for all study participants across all four grade samples were within the average range compared to the norm-referenced group of age-matched peers (as outlined from the WIAT-II-A standardization sample of children), indicating that the AHS children are performing as well as their age-equivalent peers on academic tasks.

Performance across all grade levels in the AHS study pilot phase was at or above the average or expected range for language skills (receptive vocabulary) and school achievement. A series of analyses of variance (ANOVA) was performed for each of the achievement tests and the PPVT-IV. This analysis provided significant differences between groups on performance in WIAT-II-A word reading at different grade levels (F(3,40) = 3.69, p < .05, Cohen’s d = .44). The Grade 5 and Grade 6 groups demonstrated stronger word reading skills than their younger counterparts, despite age-adjusted differences. The ECS class performed significantly lower on the word reading achievement test, compared to both the Grade 5 (p = .007) and grade 6 (p = .02) groups. The Grade 1 class showed a trend towards fewer word reading compared to Grade 5 (p = .08) and Grade 6 (p = .09). Despite these differences, the AHS children achieved equivalent or above average levels for the PPVT-IV and WIAT II-A, suggesting that these children are performing at or above their grade level (Figure 1).

Slight differences were found between the AHS and N-AHS participants in the pilot study, with the AHS group showing better performance on both word reading and math operations, though both the AHS and N-AHS participants displayed average scores on the PPVT-IV and WIAT-II-A, indicating good language and academic achievement. To determine the difference between the small control sample of N-AHS students and the larger group of AHS children, we used a case-matched analysis. The four control N-AHS participants were matched with AHS participants in both age (within 3 months) and gender. Differences on one-tailed t-tests revealed no significant group differences on the PPVT-IV (t(6) = .14, p = .45, Cohen’s d = .10); WIAT-Word (t(6) = .45, p = .34, Cohen’s d = .32); WIAT-Math (t(6) = .34, p = .37, Cohen’s d = .24) or the WIAT-Spell (t(6) = .06, p = .48, Cohen’s d = .04). Figure 2 outlines the case-matched differences between AHS and N-AHS participants.

Discussion

Limitations

Key limitations of the pilot study included the small sample size and lack of pre-test (baseline) data. The small sample size was due to the rural, northern location of the pilot site. However, despite the small size, the overall sample was representative of both male and female students at different ages. The pilot study did not use a randomized group design so as to not exclude any students from participating in the pilot study. The lack of pre-test (baseline) data precluded analysis of the effect of the AHS program on the study participants. To date, there is no standardized process in place to collect ongoing baseline information among children who enter AAHS programs. It would be ideal to test participants prior to their entering and on their completing an AHS program.

Several other limitations were identified: there was no analysis of special needs children; there was no analysis of the community’s socioeconomic demographics; the measures used were chosen in consideration of the needs of a culturally distinct sample; the measures used were abbreviated ones and were norm-referenced and standardized with North American children, which may or may not be representative of the pilot study sample. Also, when looking at the sample, children who participated in the pilot were essentially self-selected, i.e. those whose parents provided consent for their child to participate. These children may differ from those whose parents did not allow their children to participate.

Recommendations

Despite these limitations, the researchers and AAHS Evaluation Subcommittee determined that there was potential for longitudinal studies and made the following recommendations to the AAHS Provincial Committee. First, baseline data collection should be collected from children before they start and then after they complete an AHS program, to help assess its effects. Further, standardized baseline data collection procedures should be integrated into current AHS site-level data and administrative reporting systems.

Second, the quality of the AHS program should be assessed as this can influence the graduates, and this is an important variable to measure in future evaluative studies.

Third, the workloads of AHS staff should be considered when planning for future studies. AHS program staff members implement daily and weekly program activities within full schedules, and future AHS evaluation studies need to consider realistic time commitments from program staff so as to not compromise AHS program operations.

Fourth, sufficient lead time for evaluation planning, implementation and informed consent processes must be considered, as well as the resources required to assist with project liaison, scheduling, follow-up, administration, and any external challenges that arise. In some communities lead time may be as much as one year to allow for consultation with regional boards of education or other key stakeholders or administrators. Also, parents/guardians need to be briefed on the purpose, goals and procedures of the evaluation study in order to maximize understanding of the evaluation study.

Also recommended is a step-wise approach to the evaluation study: breaking down the longitudinal evaluation activities into different priorities each year may be a more practical and pragmatic way to accommodate budget and organizational capacity limitations of AHS sites. Further, program evaluation skills at the AHS sites should be developed. Finally, a greater and more varied sample of AHS sites in Alberta should be used to ensure a balance between urban and rural AHS programs.

Figure 1
PPVT-IV and WIAT-II-A scores across grade levels from AHSUNC pilot – AHS students

Figure 2
Case-matched differences of academic achievement and receptive vocabulary – AHSUNC longitudinal pilot study, Alberta (2008)

Next steps

The results of the pilot study were discussed with the AAHS Provincial Committee (i.e. all Alberta AHS sites) and with regional Alberta PHAC staff at the June 2008 AAHS Provincial Committee meeting. All partners agreed that the results demonstrate potential for a full longitudinal program evaluation study of the AHS program in Alberta. There are considerations to be taken in planning a full study, as outlined in this article. Also critical is ongoing support from the Alberta AHS Provincial Committee and government partners to ensure an adequately resourced study over the next three to five years.

Conclusion

This pilot study demonstrated positive results for future studies. There are many program and evaluation strengths to build upon within AHS programs in Alberta, and these strengths can drive ongoing program development and outcome evaluation activities. Future studies in Alberta would help demonstrate the impact of AHS on children and the resulting impact on their health and well-being. Also important is the integration of the principles of community-based research into the evaluation of a community-based program such as AHS.

Acknowledgements

The opinions expressed in this article are those of the authors and do not necessarily reflect the opinions of the Public Health Agency of Canada. The authors would like to thank and acknowledge the following people: the teachers, parents and students who participated in the pilot study; Dr. Jennifer H. Chalmers, Dr. Marjan Saghatoleslami and Dr. Cheryl Bradbury for their expertise and the completion of this work; and the Alberta Aboriginal Head Start Evaluation Subcommittee for overseeing this project. We also thank Sarah Barber, Mary Jane Yates and Ann Pasquill for their comments on the manuscript. The project described in this article was done in collaboration between the Alberta Aboriginal Head Start Provincial Committee and the Public Health Agency of Canada, Alberta Region.

Footnotes

^†Further information on the principles and values of community research can be found at http://www.communitybasedresearch.ca/Page/View/Principles

^*More information on the Canadian Evaluation Society can be found at http://www.evaluationcanada.ca and on the Canadian Psychological Association at http://www.cpa.ca

References

1. ^ Canadian Institute for Health Information. Improving the health of Canadians. Ottawa (ON): Canadian Institute for Health Information; 2004.
2. ^ Public Health Agency of Canada. Aboriginal Head Start in urban and northern communities (AHSUNC). An exploration of models of longitudinal and retrospective analysis. Edmonton (AB): Public Health Agency of Canada, Alberta Region; 2007.
3. a,b First Nations Centre. OCAP: Ownership, control, access and possession [Internet]. Ottawa (ON): National Aboriginal Health Organization; 2007 [cited 2009 Oct 28]. Available from: http://www.naho.ca/firstnations/english/documents/FNC-OCAP_001.pdf
4. ^ Wechsler, D. The Wechsler individual achievement test, 2nd edition abbreviated. San Antonio (TX): Pearson Education Inc.; 2001.
5. ^ Gresham, FM, Elliot, SN. Social skills rating scale (SSRS) for elementary level teacher ratings. Toronto (ON): Pearson Canada Assessment Inc/NCS Pearson, Inc.; 1990.
6. ^ Dunn, LM, Dunn DM. Peabody picture vocabulary test, 4th Edition. Toronto (ON): Pearson Canada Assessment Inc/NCS Pearson, Inc.; 2006.
7. ^ SPSS for Windows Version 11. Chicago (IL): SPSS Inc.; 2001.