Evaluation of Language Training Services

Research and Evaluation Branch
Immigration, Refugees and Citizenship Canada
Project reference number: E4-2018

December 2020

Executive summary

This report presents the findings of the evaluation of Immigration, Refugee and Citizenship Canada’s (IRCC) Language Training Services. The evaluation was conducted to provide an indepth assessment of this major program and considered issues of program effectiveness, covering the period from 2015 to 2018.

The Evaluation of the Settlement Program (2018) highlighted the need to further assess the different success factors and approaches to language learning. While language training is helping newcomers improve their language ability, progression was shown to vary by skill (i.e., reading, writing, listening and speaking), as well as client characteristics, which pointed to the need for a greater understanding of progression across skills. As such, the evaluation recommended an indepth examination and thorough analysis to provide fulsome outcomes results and specific recommendations for improvements to the Department with the aim of improving language training effectiveness.

The language learning services have been evaluated, focusing on two key areas. The main focus was to better understand language skills improvement – what works for whom and under what conditions, with a view to determining the specific characteristics that influence language skills improvement. The secondary area of focus was to examine whether the language learning framework is adapted to address newcomers’ needs.

Summary of conclusions and recommendations

Based on the evidence analyzed, it was found that language learning services are designed to be flexible and effective in meeting the diverse needs of newcomers and to support their progression. The findings also show that language progression for newcomers is mostly positive, but there are differences between clients and non-clients with respect to likelihood of progression. While clients were seen to progress at the same pace as non-clients when assessed in the short term, using an objective measure, clients appeared to progress more than their nonclient counterparts when assessed on a longer timeframe using a subjective measure. It was also found that some components of language training are associated with a greater likelihood of newcomers improving their language skills, such as full-time language training and multi-level classes, while others lowered chances of progression, such as continuous intake classes.

Furthermore, when assessing other settlement outcomes, the evidence indicated that:

• clients of general formal language training use official languages less frequently than nonclients, while formal language training focused on employment were using it significantly more than non-clients.
• clients of formal language training, and clients who took both formal and informal language training, are more likely to report an increase in the frequency of use of official languages.

Although not a direct objective of language training, employability remains a primary concern for clients. The evaluation carefully analyzed this theme and assessed the impact of language training on various labour market outcomes. Clients of general language training used English or French at work less frequently and were less comfortable using official languages than non-clients, however taking language training focused on employment contributed to making these gaps smaller. Also, clients often had poorer labour market outcomes than non-clients on the short to medium term. The analysis showed that a large part of the difference in employment outcomes between clients and non-clients could be attributed to socio-demographic profiles of individuals 7 (e.g., education, age, gender, year of admission). This suggests that taking language training is not necessarily a cause of poorer labour market outcomes, but rather that clients and non-clients may have different characteristics that explain their outcomes on the labour market. Furthermore, the evaluation found that employment outcomes of clients do not vary greatly based on how language training is delivered, language training focused on employment generally had a positive impact on employment outcomes, and taking language training during core hours was associated with less favourable results.

While the client progression and their labour market outcomes show mixed results, it should be noted that language learning services correspond to the diversity in clients’ need and IRCC funded language learning services are designed in a manner to be conducive to language improvement for newcomers.

In response to the findings from the evaluation, this report has grouped the recommendations into two main themes. First, the evaluation proposes three recommendations around the topic of outcomes measurement. Second, the evaluation recommends improvements to the program to foster success. To this end, the evaluation proposes seven additional recommendations to further support clients, instructors and program stakeholders.

Recommendations

Theme 1: Measuring Outcomes

These recommendations focus on ensuring the objectives and indicators of success for clients in language learning services are clearly outlined. Moreover, these recommendations are made to ensure the department is well-positioned to monitor and report on collected data, and ensure the department has the required tools for measuring desired outcomes.

Objectives of Language Training

For the newcomer, settlement is a multidimensional route encompassing various outcomes. While the Settlement’s Program Performance Information Profile (PIP) outlines both the social and economic outcomes associated to it, it does not identify specifically which outcomes relate to language training, and their associated indicators of success. When it comes to language training, one of the main outcomes that could be derived from this type of settlement service could be seen as language skills progression. However, given the variety of the program offering and goals of learners, multiple outcomes can be associated to language training. In this context, it proved difficult to approach language progression as the outcome on which to base the success of the program and to determine what could be considered as a sufficient level of progression for the program to succeed. As such, the department would benefit from clarifying the primary outcomes associated with language training, as well as defining clear targets for achieving those outcomes.

Recommendation 1: IRCC should clarify and confirm language training program outcomes to ensure all desired results are defined, monitored, and integrated into the program theory and corresponding documentation.

Milestones Test

While the evaluation successfully used the Milestones test as an objective measure of language progression in the short term across the four learning competencies, this tool was expensive to implement, difficult to put in place and significant efforts were required to recruit clients and non-clients for the test. Moreover, the Milestones focuses on assessing language levels of individuals at the CLB 4 to 8, and cannot be used to assess language levels of those at lower or 8 higher proficiency levels. These limitations are important, and limit the department in assessing the vast majority of its client-base, especially given the focus the department decided to place on delivering language training at CLB 1 to 4. Consequently, before IRCC considers future use of the Milestones, it should examine the appropriateness and consider the limitations of using this test with respect to the populations under study, approach to administration, and cost-benefit balance.

Recommendation 2: IRCC should consider the appropriateness of the future use of the Milestones Test, including an examination of its value for money and applicability across client groups.

Employment-related data

The evaluation found a lack of clarity in the administrative data (iCARE) regarding the definition of employment-related language training. This lack of distinction between employment-related language training and general language training with a focus on employment limited the evaluation’s capacity to clearly distinguish the outcomes for those two groups. Collecting precise information on the type of language training delivered would allow the department to better monitor and report on the program. Despite difficulties in defining employment-related language services, the evaluation found that clients in employment-focused language training were more likely to progress and use official languages. In addition, this training component was also associated with more positive employment outcomes. Experts also praise employment-based language training, finding it is highly effective in helping newcomers integrate into the labour market. As a result, IRCC should reconfirm the benefits associated with this type of training by using a standardized definition of employment-related language training.

Recommendation 3: IRCC should develop and implement an approach to better identify employment-related language training in iCARE and monitor its uptake and outcomes.

Theme 2: Program Improvements for Fostering Success

The evaluation found that overall the program has many design features conducive to clients’ language progression and also identified several areas where there is room for improvement. In particular, the evaluation proposes 7 recommendations to further support clients, instructors and program stakeholders.

Instructor supports

It was found that instructors are qualified, well-trained and benefit from many supports. The evaluation however found they would further benefit from:

• more PBLA material which are easily adaptable;
• ensuring instructors’ assessments provided as part of PBLA are aligned with CLBs;
• improving navigability of and better vetting of new and existing Tutela content;
• limiting the amount of unpaid work instructors need to perform; and
• more professional development for CLIC instructors.

The evaluation found that the PBLA is widely-implemented and used. Instructors derive value from PBLA and clients feel empowered by this approach, as it shows them their progression. Given PBLA relies on instructors’ assessments of artifacts against CLBs, experts felt that it introduces an element of subjectivity. As a result, the experts indicated that PBLA was more useful as a learning tool than as an objective tool for assessing benchmarks. Moreover, instructors felt it requires too much preparation time, specifically in creating and adapting materials. While instructors benefit from PBLA supports and other online resources, there is room for 9 improvement. Mixed views were provided on Tutela’s usefulness, specifically with respect to a need for better vetting online content, and making the website easier for users to navigate. NLAB members and interviews felt more PD is needed for instructors who teach literacy clients, and for CLIC instructors.

Recommendation 4: IRCC should develop and implement a plan to ensure that LINC and CLIC instructors are adequately resourced.

Literacy

Literacy needs are present at a broad range of levels (i.e., literacy designations are provided up to CLB 4). Although instructors are well-qualified, and trained, the evaluation found that there is a need to have access to specific materials and professional development to teach to clients with literacy needs. Also, while PBLA is widely-implemented and used, the evaluation found that PBLA may not be helpful for clients at low language levels, especially those with literacy needs.

Recommendation 5: IRCC should implement a plan to enhance language training provision for clients with literacy needs, namely addressing challenges related to use of PBLA with literacy clients.

Continuous Intake

The evaluation found that continuous intake helps SPOs meet occupancy levels and ensures that newcomers can enter a class as soon as possible. However, this flexibility can be disruptive for teachers and students and can also add a challenge for new clients who need to catch up the level of the class. This continuous intake was also seen to hinder progression of clients. On the other hand, multi-level classes were also seen as a way to meet the demand for language training with smaller groups of clients and to have a positive impact on language progression. As such, considerations could be given to leveraging multilevel classes and staggering start dates of smaller groups at each CLB, to meet the demand of clients. Experts also suggested the introduction of temporary holding tanks (i.e., temporary classes that include multiple CLB levels until there are available classes) as another mitigation strategy.

Recommendation 6: IRCC should implement a plan to address the challenges associated with continuous intake for clients and instructors.

Guidelines

The evaluation found that LINC and CLIC instructors are supported by helpful and up-to-date curriculum guidelines. However, there was a lack of formal guidance for employment-related language instructors. There is a need to develop better guidance (e.g., manuals, policy guidance) to meet the needs of instructors who may not feel supported by the lack of guidance, recognizing that content guidelines for employment-related language training requires flexibility to accommodate different program offerings and client needs and desired objectives.

In addition, informal language services were found best designed for newcomers who feel socially isolated, or who are not comfortable in structured learning environments. The evaluation found that the informal language training is largely supported by volunteers. Having guidelines which are clearly linked to the program’s desired outcomes would help support volunteers who might not have the same background as formal training instructors.

Recommendation 7: IRCC should develop guidelines for employment-related language training and informal language training.

Combining formal and informal language training

While informal language services alone were not associated to a higher likelihood of improvement, the evaluation found that clients who took part in both formal and informal language training had higher chances of language skills progression and often had better employment outcomes than clients who only participated in formal language training.

Recommendation 8: IRCC should develop a plan to leverage and optimize the benefits of informal language training for formal language training clients.

Childcare

The evaluation highlights a series of barriers to attending language learning services. In particular wait times to access childcare are long, particularly affecting newcomer women, families with multiple children, and single-parent families.

Recommendation 9: IRCC should examine ways and develop a plan to meet the needs of clients in terms of providing care for children with the view of facilitating access to language training.

Coordination/Communication

While the roles and responsibilities related to the program are generally clear, it was found that NLAB felt that programming changes could be better communicated, and that a feedback loop including Settlement Network would be beneficial. Stakeholders felt that being informed of programming changes and priorities could help them to have more preparation time on calls for proposals.

Recommendation 10: IRCC should implement a strategy to ensure effective and timely dissemination of information on policy changes and priorities between policy, operations and external stakeholders.