Evaluation of the Language Instruction for Newcomers to Canada (LINC) Program

2. Methodology

2.1. Document review

The overall purpose of the document review was to enable the evaluators to learn about the program and its context and to collect pertinent information on the program. Key documents included: LINC pamphlets, an annual report, policy documents, earlier evaluation frameworks and evaluations of LINC, iCAMS reports on LINC data, and more general literature dealing with assessment of language proficiency and language programs.

2.2. Administrative data review

Analysis of administrative data was used to develop a statistical profile of the program, to ensure the program is (or can be) properly monitored, to address pertinent evaluation issues, and for preliminary outcome analysis. Pertinent databases were iCAMS^{Footnote 5} and HARTs^{Footnote 6}. HARTs is the system used in Ontario to collect the iCAMS data.

2.3. Key informant interviews

Interviews with key stakeholders were crucial to assess program implementation and operation; to explore interviewees' perceptions of the success of LINC in achieving its immediate and long-term objectives; to examine communications and promotion activities; to investigate the interaction between LINC and provincial ESL programming; and to gather suggestions for improving the program. Key informants were identified and interview guides were designed to govern the interviews. Interviews were held with national, regional and local CIC officials. Interviews lasted between 25 minutes and two hours.

Table 2-1: Summary of interviews

2.4. Surveys

The evaluation included surveys of SPOs, teachers, LINC learners, a comparison group of newcomers who never enrolled in LINC and a group of former LINC learners.

LINC administrators

Surveys were sent by email to all 155 LINC providing organizations in Canada (outside of Quebec, Manitoba and BC), to be completed by the LINC program administrator at each SPO.

The survey was administered by email, with a response of 141 returned surveys, for a rate of 91.0%.

Learners

It was determined that the best option was to rely on instructors to conduct the survey in their classroom. This attenuated the issue of English/French comprehension because teachers could help lower-level learners through the survey.^{Footnote 7}

Multistage random sampling was used to select the sample (see Appendix B: for explanation).

It turned out that one of the 70 classes selected was a literacy class: it had to be excluded because of the extreme difficulty surveying such students. Sixty-eight survey packages were returned. This represents a response rate of 98.6%. In total, 651 surveys are included in the analysis (in several classes there were fewer than 10 learners in class on the day of the survey; all those surveys were included in the sample). This represents about 2.3% of LINC learners in the country. Home Study learners represent 3.1% of the survey respondents, about the same as their proportion of the LINC population. Non-response was negligible so it should not be a source of bias.

As expected, the characteristics of the LINC sample closely mirror those of the population (Table 2-2 ). Based on the sex, LINC level, age, time in Canada and education variables, it is safe to conclude that inferences drawn from the survey findings should be valid. Note that for age and time in Canada the differences between the two groups reach statistical significance. The distributions are very similar but the large number of cases makes the statistical tests very sensitive.

Table 2-2: Survey respondents compare closely to the LINC population

The case study sample is also reasonably close to the population on all characteristics shown in the table. In the case study sample, women and older age groups are somewhat over-represented. LINC level 3 is also over-represented.

Teachers

An instructor’s survey collected teacher opinions and feedback on the program. In addition, a class information form (CIF) was also completed by the teachers, collecting administrative information on the classes. The CIF survey collected administrative data, including enrolment dynamics, class level, class schedules, class focus, teaching materials used, and methods of assessment used. The evaluators surveyed teachers of classes selected for the learner survey. This was a random sample, although the number of surveys was limited to 68 (with a 98.6% response rate). This represents approximately 4% of LINC teachers nationwide. All class information forms were returned.

Teacher surveys and class information forms, along with the learner surveys, instruction sheets were sent to 56 of the 70 classes selected at random. The other 14 instructors were involved in the case studies.

In addition, four Cours de Langue pour les Immigrants au Canada (CLIC) classes were selected at random for the survey. All teacher and CIF forms were returned. Twenty-eight CLIC learner surveys were completed as well.

Comparison group

Surveying the comparison group of newcomers not enrolled in LINC was carried out at the time of the outcomes testing of the selected newcomers. They were handed a paper copy of the survey for completion. The assessors who administered the outcomes testing did their best to help those with low-level English skills understand the survey.

Former LINC learners

The evaluation attempted to reach former LINC clients in order to determine their reasons for discontinuing classes, and overall perceptions of the LINC program. A difficult respondent group to reach due to often outdated contact information, the response was low, with 91 surveys returned, for a rate of 17%. However, the findings from this group do provide good indications regarding several key questions.

2.5. Case studies

Fifteen case studies were conducted: 14 LINC classes were randomly selected from the 70 selected for the survey, in addition to a randomly chosen CLIC class. Each case study visit consisted of: facility/classroom observations, completion of the learner survey, completion of the CLBA^{Footnote 9} by class participants and a learner focus group discussion. Prior to the visit the LINC instructor completed and submitted a teacher survey and a class information form.

2.6. Literature review

A literature review of existing research on language training in Canada and other countries was conducted to provide additional perspectives and evidence regarding the impact of various designs and models of language instruction for newcomers. It on the following topics: the impact of language proficiency on settlement and integration; language instruction program design; and, best practices in language training delivery. The literature review findings^{Footnote 10} were incorporated into the report.

2.7. Outcomes testing of comparison group

In order to measure changes in language proficiency, the CLBA was used in a pre/post-test approach. To help isolate the impact of randomly selected LINC learners from all other possible influences on language acquisition, the evaluation included a comparison group of newcomers (who were initially assessed but never took LINC). Initial assessment scores were then compared against re-test results, using the CLBA tool. There is no standardized exit test available for LINC; while the CLBA was not specifically devised for this purpose, it was determined that it was the best available tool to use for the purposes of the evaluation.

The challenge was locating those who had been assessed but did not enroll in LINC classes; the sample totaled 53 people. The Ontario database, HARTs (History of Assessments, Referrals and Training), was the electronic source for selecting a comparison group,^{Footnote 11} which includes all newcomers to Ontario whose language abilities were assessed by language assessors. The challenge was to find those who were assessed but had not taken LINC. A random sample of newcomers in Ontario and Edmonton who were assessed but did not enroll in LINC improved the representativeness of the comparison group. For newcomers selected at random who had not attended LINC, the evaluation surveyed and assessed language capability via CLBA testing. To facilitate outcomes testing, the selection of the comparison group was limited to newcomers originally assessed at one of three assessment centres, one in Toronto, one in Halton/Peel and one in Edmonton. The newcomers were contacted with the assistance of the assessment centres (the evaluators did not have access to personal identification information). Each newcomer was given an honorarium for time spent testing.

2.8. Limitations

There are several limitations to this evaluation:

• LINC program data is collected through two systems – HARTs in Ontario (which represents approximately 82% of LINC clients) and iCAMS in the other regions. The data used in this study corresponds to all reporting SPOs in the iCAMS and HARTs data systems. There may instances where SPOs have not reported LINC training in these systems for various reasons.
• LINC data contained in iCAMS (and HARTs in Ontario) are largely administrative in nature, providing client and program profile information, but does not provide adequate information on client outcomes. Development of a systematic approach to collection of outcome information (such as exit scores) would permit further analysis of client progress.
• In an ideal environment, it might be possible to test the effectiveness of LINC against a control group who had not received language training. It would, however, be difficult to isolate the impact of LINC, on LINC learners, from other influences on their language acquisition. In this study, in an effort to provide a more quantitative assessment, a small sample group (those assessed but not enrolled in LINC) was selected and a pre-test/post-test approach was used to compare gains scores.
• A related challenge to assessing language progression was the absence of a standardized exit test available for LINC. The evaluation used the CLBA to measure changes in language proficiency. Though it was not devised for this purpose, it was determined that CLBA was the best available tool to use for the purposes of the evaluation, and it allowed a direct comparison of assessment and ‘exit’ scores.
• The comparison group selected for outcomes testing is representative of the Greater Toronto Area (GTA) and Edmonton only, (but Ontario represents approximately 80% of LINC clients). Many who were selected could not be contacted; others refused to participate. The final number of individuals in the comparison group (53) is modest, but is enough for reliable statistical analysis. Multiple regression analysis controls for observable differences between the LINC client and comparison groups but there may be other differences (e.g., motivation) that cannot be controlled for.
• There was difficulty contacting former LINC learners, resulting in a response rate of 17% (96 returned surveys), thus the associated results, while a good indication for this group, should be taken as approximations.
• Because course content can vary between SPOs, there was difficulty in assessing the extent to which LINC participants acquired knowledge of Canada and of Canadian civics and the degree to which the program has assisted with settling in Canada. Using already validated theme-based tools to test thematic (content) information^{Footnote 12} yielded unreliable results. Thus content gains were assessed with surveys and focus groups which provided defensible conclusions about content gains attributable to LINC.

• Previous
• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8 (current)
• 9
• 10
• 11
• 12
• 13
• 14
• Next