Evaluation of the Language Instruction for Newcomers to Canada (LINC) Program

5. Program impact and cost-effectiveness

5.1. Language proficiency

LINC is designed specifically to improve the language proficiency of newcomers in a Canadian context. Accordingly, one focus of the evaluation was to assess the impact of the program in terms of language gains. In an ideal environment, it might be possible to test the effectiveness of LINC against a control group who had not received language training. It would, however, be difficult to isolate the impact of LINC, on LINC learners, from other influences on their language acquisition. Similarly, for a control group, it is difficult to identify the impact of unobservable characteristics (e.g. motivation, diversity of social networks, etc.) on their language acquisition, outside of a LINC environment.

In this study, in an effort to provide a more quantitative assessment, a small sample group (those assessed but not enrolled in LINC) was selected and a pre-test/post-test approach was used to compare gains scores, measuring changes in language proficiency using the CLBA tool. To further isolate the impact LINC classes from all other possible influences on language acquisition, the scores of a comparison group of newcomers (who were initially assessed but never took LINC) were analyzed using a regression model^{Footnote 36}.

LINC levels completed

On average, LINC students had completed 1.0 level. Just over half the LINC students in the case studies had completed at least one LINC level^{Footnote 37} (refer to Table 5-1 ). The mean number of hours to complete each level is presented in Table 5-2^{Footnote 38}. The large standard deviations suggest that many individuals deviate substantially from the mean at every level. Across all students (in the case studies) the mean number of hours to complete a LINC level was 347.4. Using iCAMS datafor all LINC students in late spring 2009, the typical learner took 389.4 hours to complete a level. Table 5-2 compares the LINC population to the case study sample by level. The sample is reasonably close to the population except at level 5 (where there were only four cases in the sample)^{Footnote 39}.

For the case study cases, number of hours was examined by using HARTs data and information solicited directly from the case study SPOs outside of Ontario. Table 5-1 includes iCAMS data on the population and data from the case studies; the proportions are very close, confirming that the sample well represents the population.

Table 5-1: LINC levels completed

Table 5-2: Mean hours to complete LINC level by level

N=126

Language proficiency gains - Mean difference scores

Table 5-3 compares language proficiency gains without considering possible influencing factors or observable differences. The table below illustrates the mean difference in language proficiency gains by subtracting the current assessment score from the client’s initial assessment score. The entries under “LINC” and “Comparison” represent the difference between the current and original benchmark score for both groups. Most notably, LINC clients improved their reading skills by 1.21 benchmark levels, and experienced gains in all skill areas. While the comparison group (those assessed but not enrolled in LINC) improved their proficiencies, the gains were more modest.

Table 5-3: Comparing mean difference scores (uncontrolled)

All the results in the “Difference” column are in a positive direction – that is, LINC students improved more than the comparison group – but the gains were not enough to reach statistical significance for listening, speaking and writing. Note that a simple pre/post-test design (using only the results in the LINC column) demonstrates that LINC brought about significant gains in all four skill areas. However, this does not consider the differences in the two groups nor attempt to attribute the gains to the LINC program. Some of these key differences considered were: education completed before immigration, age, gender, language distance^{Footnote 40}, LINC level at initial assessment, employment status and length of time since initial assessment.

Language proficiency gains - Regression/observable differences

Because LINC client and comparison group individuals differ, observable differences were controlled for by using multiple regression analysis in order to attempt to isolate the effect of LINC^{Footnote 41}. Introducing statistical controls, the analysis supports the findings from Table 5-3.

The column labeled ß is the regression coefficient, which indicates the unique (independent) contributions of the “group” variable (LINC group vs. comparison group) to explaining the total variance in the assessment score, Table 5-4 displays the results of the analysis for each language skill.

Table 5-4: Regression analysis^{Footnote 42} – group variable only

Although LINC students appeared to advance about 21% of a benchmark level more in speaking than the comparison group, and 23% listening (see the ß column in Table 5-4 above), the gains were not enough to reach statistical significance once the differences between the groups were controlled.

Gains for reading (88% of a benchmark level) and writing (51% of a benchmark level) reached statistical significance for the group variable^{Footnote 43}. The regression coefficients for “group” may be interpreted as the change in benchmark with a unit change in Group (from comparison to LINC) on the assumption that all other values for the remaining regressors are held constant. Thus, once observable differences between the groups are accounted for, gains of 0.9 benchmark in reading and half a benchmark in writing were most likely attributable to LINC^{Footnote 44}. The analysis cannot make more definitive conclusions as it is not possible to control for unobservable differences (such as motivation and native intelligence).

The only variable that significantly influences listening and speaking is length of time since initial assessment. The more time spent since the initial assessment – that is the more time spent in Canada immersed in English – the more listening and speaking improved. This variable also positively influenced reading and writing gains. Note that none of the other independent variables – age, sex, education, language distance, LINC level, employment status, enrolment in non-LINC ESL – significantly affected any of the four skill areas.

Number of hours in LINC makes a considerable difference. The next figure shows that as the number of hours in LINC rises, the impact of LINC rises. While benchmark levels increase moderately from 1 to 750 hours, a more significant impact is realized as more time is spent in LINC classes: When students attend LINC classes for 1000 hours or more, the gains likely attributable to LINC increase to 1.3 benchmark for listening, 1.2 for reading and 1.7 for writing.

Figure 5-1: Benchmark gains over time – Gains versus comparison group

While the language gains ascribable to the program are higher in certain skills, there are elements of language acquisition that cannot be captured in the evaluation approach.

5.2. Course content

LINC is also intended to improve students’ knowledge of Canada and of Canadian civics and to introduce students to concepts they need to integrate in Canada by providing information on the Canadian workplace, job search techniques and tools and so on. Because LINC does not have a mandated curriculum it is almost impossible to create validated instruments to assess what has been learned in LINC classes beyond proficiency in English. Content gains were examined using findings from the surveys and focus groups.

Since content gains should take into account what is taught in the classes the class information form asked teachers to specify what subjects their class on and to pinpoint the main focus. As Figure 5-2 shows, the two main foci of LINC – English for daily life and settlement/integration – were covered in almost all LINC classes. This corresponds to the dual purpose of LINC. Asked to specify the main focus from among those listed, 63% of teachers said English for daily life and 31% said settlement/ integration (many said both – their responses were evenly distributed between the two categories).

Figure 5-2: Class focus

N=65

Source: Class Information Form

The focus groups got more specific about the subjects covered in class. The list of what students had learned about Canada was considerable and, for the most part, consistent across classes selected for the case studies: History; Geography; Culture/multiculturalism; Government /politics; Customs and traditions; Weather/climate; Procuring documents and learning how to get access to key services; Transportation; Natural resources; Medical system; Emergency services; Laws; Family life; Sports and activities; Housing; Taxes; Shopping; Education system; Industry; Immigration; Holidays; Music; Banking; Women’s rights in Canada.

For most case study classes, topics related to employment were also cited as a crucial facet of LINC. When asked, focus group participants consistently responded that a wide array of job search and work place skills and concepts were taught in LINC classes.

Settling in Canada

The ability to settle in Canada was assessed (with LINC client and comparison group surveys) to determine the extent to which newcomers were able to gain access to basic services. LINC was said to help most in those areas where there is more of an interaction than merely applying for something, like a bank account, SIN or health card. It helped most with making friends – likely to include classmates.

Table 5-5: LINC students settling in Canada

Column 1 represents responses for all survey cases, thus the slightly different percentage from Table 5-6 below

These questions were asked of the comparison group as well, enabling a test of the incremental benefit of LINC. Table 5-6 suggests that LINC has been of little incremental benefit for these elements, as comparison group responses indicate the same level of settlement without attending LINC classes.

Table 5-6 : Comparison group settling in Canada

Column 1 represents the case study respondents, thus the slightly different percentage from Table 5-5 above.

Regression was used to control for observable differences between groups. The conclusions are the same: for none of these variables did LINC make a notable difference. Comparison group members were more comfortable going alone for health appointments, likely because they had better English skills on average than the LINC group. For several aspects of settlement, newcomers are likely to require them immediately upon arrival before even enrolling in language training. Many students indicated in the focus groups that they had bank accounts, SIN and health cards before taking LINC so LINC could not be expected to help.

5.3. Cost-effectiveness

This section examines LINC program expenditures and the key areas where investments were made.

5.3.1. LINC funding

LINC funding has increased considerably in recent years (see Table 5-7). In the five year period beginning in 2004-05, LINC spending increased by 83%. Because integration spending increased by 178% during the same period, LINC accounts for a smaller proportion of total integration spending as settlement funding increased.

Table 5-7: LINC expenditures

Source: LINC Factsheet with updates from CIC. Excludes a grant to Quebec and funding arrangements with Manitoba and British Columbia

5.3.2. LINC expenditures

For the last 10 years, investment in teacher salaries has accounted for the largest portion of LINC spending. Under a revised settlement funding model for the period of 2000-01 to 2005-06, this category ranged from 69% to 76% of total program expenditures and was relatively constant, ranging from $64.7M to $68.5M during that time^{Footnote 45}. Combined, child minding and transportation expenses have risen from approximately 2% in 1998-99 to 18% of total LINC expenditures in 2008-09.

Table 5-8: LINC Program expenditures by category, 1998-99 – 2008-09 - Part 1
^{Footnote 46}

Table 5-8: LINC Program expenditures by category, 1998-99 – 2008-09 - Part 2

As a result of the increased funding, the cost per LINC student has risen substantially. As program expenditures rose from $94 million in 2004-05 to $173 million in 2008-09, the number of learners rose from about 52,000 to about 55,000. As a result, the cost per LINC student had risen from about $1800 to approximately $3130^{Footnote 47}.

The substantial increase in the average cost per LINC client reflected the need for CIC to invest in program renewal following several years of static funding prior to 2006-07. During this period, LINC payments to SPOs had fallen behind the actual cost of delivering the service. The 2004 LINC Evaluation confirmed that program funding levels had not kept pace with rising costs, that funding deficits were associated with long wait lists, and that new monies would be required to implement program improvements suggested by the evaluation. With an influx of new funds in 2006, CIC invested significantly in the following areas^{Footnote 48}:

• Program renewal: New program funding has been used to provide more, and more diverse, course offerings to ensure that newcomers can access courses tailored to their particular learning needs and goals.
• Teachers: The single largest commitment made by CIC to the renewal of LINC since 2004-05 has been its increased investment in the salaries, benefits, and training provided to LINC teachers and assessors. The quality of teachers is the primary determinant of program effectiveness (as noted in section 2 of this report).
• Childminding and support services: From 2004-05 to 2008-09, substantial funding ($12M) has been allocated to expand the availability of childminding, facilitating access to training for newcomers (transportation and provision for the disabled) who might otherwise be unable to participate due to barriers related to access. The expansion of childminding services^{Footnote 49} spending increased from $17.3M to $27.8M while transportation services spending increased from $1.4M to $2.9M.
• Infrastructure and resources: A proportion of new spending ($6.5M) has been devoted to facility enhancements (for both training and childminding) and the development of new teaching resources.

5.3.3. Cost-effectiveness and alternatives

Most key informants felt that LINC was adequately funded and there were few calls for more resources. Those who did want more funding tended to point to specific areas in need of enhancement such as expanding childminding services, offering more classes on the weekend and moving into more distant and rural communities. Also, some informants pointed out that not all provinces have the same levels of LINC available and said that could be addressed with additional funds.

Virtually all key informants in CIC and with the provinces believed that LINC was cost-effective. They reasoned: funding is distributed through competitive contacting processes; service providers are required to make a case for funding received; most SPOs are not-for-profit organizations that have reasonable overhead and moderate salaries; and each SPO is subject to rigorous financial reporting requirements.

Many SPOs deliver an array of integrated settlement services, including LINC, which may contribute to cost effectiveness of program delivery.

No informant was convinced there were any more cost effective methods of delivering second language services. All were in agreement that it would not be possible for CIC to deliver the services directly – it has neither the expertise nor the infrastructure required – and that if it did the cost would certainly be much higher. The Ontario LINC Home Study evaluation reported that LINC Home Study costs approximately two-thirds as much as classroom LINC per benchmark completed. Progress for Home Study learners was slower mainly because the number of hours per week in Home Study tends to be much less than the number of hours spent in LINC classes, but in 2005-06 classroom LINC cost over twice as much per seat as LINC Home Study^{Footnote 50}. This suggests that expanding LINC Home Study to complement existing modes of delivery could potentially improve cost-effectiveness in addition to widening accessibility.

Further comparative analysis of other models of delivery would be required in order to better determine the cost-effectiveness of the LINC program.

• Previous
• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11 (current)
• 12
• 13
• 14
• Next