ARCHIVED – Enhanced Language Training Initiative: Formative Evaluation

3.0 Evaluation Findings

3.1 Rationale

Key informants (including representatives from CIC, provinces/territories, and stakeholders) and SPO representatives confirmed that labour market integration for newcomers remains a challenge and that there is a continuing role for the federal government. A more in-depth assessment of the rationale for this program will be undertaken in the ELT summative evaluation. (See Section 4.0 for a discussion of some methodological issues for the summative evaluation.)

3.2 Development Projects and ELT Tool Outcomes

3.2.1 ELT Development Component

Findings from the ELT database indicate that development projects produced a total of 103 ELT tools (e.g. Curriculum, Evaluation Tool, Research, Promising Practices, Occupational Terminology and Other tools). As shown in Exhibit 6, the tools covered a range of subject matters – the most common being ELT curricula (56% of all tools developed). Hence, the development projects achieved the expected outputs, as outlined in the ELT logic model (Appendix A1). However the expected output of “new curriculum” appears to have been the primary focus of the development component. The development of “other tools/supports” and “ELT research” appears to have received less focus.

Exhibit 6: Type of Development Tools Produced

Source: CIC ELT database, May 2007

The extent to which the development projects contributed to the expected outcome of “support and improve projects and programs” is reflected in the assessment of the ELT tools in the next section.

3.2.2 Assessment of ELT Tools

Some tools in use were developed by the SPOs, either with delivery project funding or their own funding; while others were based on existing English as a Second Language (ESL) tools. The evaluation assessed the quality of the tools being used by SPOs participating in the case studies (whether developed through development or delivery projects or not), by having the tools reviewed by two external experts.

The tools selected for review were drawn from the twelve case studies carried out through site visits. The evaluation team collected tools – either during or after the site visits – and submitted them to the experts for review. [ note 22 ] The experts reviewed the tools against criteria (see Appendix A for a copy of the review instrument) that were regrouped into the following themes, assessing a range of factors:

• The processes used for developing the tools, including consultations and use of specific expertise;
• The content of the tool, including the clarity of the learning outcomes, appropriateness of the content (for the CLB levels, occupation focus and target audiences) and organization of the material, clarity of the language, vocabulary and style of writing, and the balance among the four skills (speaking, listening, reading and writing);
• The format and design, including the consistency of the format and layout with standards for this type of resource, appropriateness of the format and the use of graphics; and
• The generalizability of the tools to other contexts and other issues.

As reflected in Exhibit 7, the experts rated the content and format and design of tools very positively, and the process and generalizability of tools as good. These findings suggest that they likely contributed to the outcome of supporting and improving projects and programs. The exhibit reflects the overall nature of the comments for each of the criteria groups. [ note 23 ] However, the same information was not available for all tools and, as a result, not all factors can be assessed for each tool.

Exhibit 7: Criteria of Quality Tools

Source: Analysis of expert reviews of ELT tools
* High rating refers to references to the tool being “excellent”, “exemplary” etc. Those that received a mixed rating tended to have both positive and negative comments and those receiving a low rating received generally negative comments.
** The ratings covered 13 tools associated with 12 projects and 11 SPOs.

The experts’ ratings for the process used to craft the tools were generally hampered by the limited related information in the documents (reflected in the number of cases for which there was no information available). However, in the seven for which information was available, the processes were rated positively, with the experts noting specifically the extent of consultations conducted for the development of the tools and the expertise of the developers.

The content of the tools was the most highly rated criterion. Almost all tools were given very positive comments – with some meriting comments such as “exemplary”, “excellent” and “highest quality.” The experts noted such things as:

• The clarity of the objectives, outcomes and, as relevant, occupation-specific focus;
• The appropriateness of the balance among the various skills (reading, writing, listening and speaking);
• Appropriate language that is accessible to the target audiences and the clear “tailoring” of the tool to the client needs;
• Positive use of such things as diagrams, margin notes to support comprehension and use of the tool; and
• The extent to which the materials were well-organized.

The format and design were also rated positively, but with a few more mixed or low ratings than for the content. In the positive ratings, the experts noted that the material was clear and logical, professional looking, appropriate and reader or user-friendly. They noted that there was good use of photos, tables, grids, bolding, headings, pages numbers and headers, as well as formats, such as spiral bindings and DVDs. Where the comments were more mixed or low, the concerns related to unclear or poor organization, material being difficult to follow, or lack of consistency between sections.

The assessments of the generalizability were of two types: 1) assessments that the materials were generalizable as they were – with comments such as “even novice teachers could use this with success” or could easily be used in other provinces; and 2) assessment indicating that the tools were generalizable in part or with some modifications, usually to suit the needs of specific provinces. Two tools were judged to be not generalizable because there were no clear outcomes/goals stated or there was not enough detail to allow their use outside the program. However, one of these tools was judged to be potentially useful for identifying ideas for tools for similar occupation-specific projects.

3.3 Delivery Project Outcomes

This evaluation focused on assessing [ note 24 ] the achievement of the immediate expected outcomes of the delivery projects, as outlined in the ELT logic model (see Appendix A1):

• Preparation for licensure exams;
• Improved language skills;
• Increased knowledge of the Canadian work environment;
• Increased experience in the Canadian work environment;
• Increased job finding skills; and
• Establishment of mentors/contacts and networks.

The final section reflects preliminary information on the longer-term outcomes of increased job opportunities and finding employment commensurate with their skills and experience.

3.3.1 Preparation of Licensure Exams

Three case studies focused on the preparation for licensure exams. One project reported a high level of success, while the remaining two did not have results data available.

• The project for pharmacists reported that all participants passed the licensing exam (thirteen on the first try and two on the second try), got their license and found a job in their profession.
• The project for physicians accommodates about twenty participants each year and reportedly there is an increased likelihood of success in the clinical assessment that provides for entrance into the provincial medical licensure program.
• The results with respect to licensing for the engineering project are not yet available.

In addition, other generic projects assist participants by helping them to identify the paths to licensure in their profession. This has reportedly reduced frustration for participants and facilitated the transition into their profession. One case study noted specifically that it would be helpful if CIC could provide information for each profession and province/territory about the requirements for certification or licensure.

3.3.2 Improved Language Skills

The ELT database provides information on changes in language skills based on the CLB tests done, for most projects, at participant intake and based on SPO assessments of language levels at project completion. [ note 25 ] As shown in Exhibit 8, the database results reflect an increase in language levels with respect to all four dimensions (listening, speaking, reading and writing).

Exhibit 8: Entry and Exit Language Assessments Averages (n = 1610 to 1620)

Source: CIC ELT database, May 2007

The findings from the database were supported by information provided in the case study focus groups. Almost all participants in focus groups indicated that they felt the ELT program had improved their language abilities, noting specifically improvements in language skills, improvements in communication skills and/or improvements in their workplace/ occupation-specific language abilities.

3.3.3 Increased Knowledge/Experience with Canadian Work Environment

Two related ELT outcomes are dealt with together in this section – increased knowledge of the Canadian work environment and increased experience with the Canadian work environment. They are treated together since they both result from the two main components of the ELT projects – the in-class component and the bridge-to-work component. It is difficult to distinguish the impact of the two components in increasing knowledge of the work environment, although increasing experience results more directly from the bridge-to-work component.

In-class component

All ELT projects covered in the case studies included an in-class component. The in-class component involves a variety of teaching approaches, including multi-media approaches to instruction, interactive lessons (e.g., role playing) and employer involvement (e.g., guest speakers). The majority of participants in focus groups indicated that these various approaches were useful.

Although focus group participants in only a few projects indicated that the employer had played a role in the in-class component (typically as a guest speaker, attending/participating in student presentations and hosting job fairs and/or field trips), evidence from the site visit case studies suggests that the focus of the language training, which occurs during the in-class component, is organized around the Canadian work environment. For example:

• The learning of new vocabulary focuses on the vocabulary of the workplace – either specific to an occupation or a generic workplace; and
• Language and communication skills were developed through applied job-related exercises such as preparing résumés, practicing job interview techniques, undertaking informational interviews with employers [ note 26 ] etc.

This contributes to increasing participant knowledge of the Canadian work environment.

Focus group participants were generally satisfied with the in-class component of the project. Employers interviewed during case studies felt that individuals who successfully completed the in-class component were generally well prepared for the Canadian work environment, having learned the basic requirements of the work environment (e.g., arriving on time, appropriate interactions with supervisors and co-workers).

Bridge-to-work Component

ELT delivery projects are required to include at least one bridge-to-work component. Most case studies (70% of the case study SPOs) included work placements (unpaid, paid or partially paid) as the key bridge-to-work component (see Exhibit 9). [ note 27 ] Of these, about one-third were voluntary placements, one-third were at least partially paid and one-third did not specify if they were paid or unpaid). The case studies also suggested that, although a project may offer work placements, not all participants are necessarily placed. Other forms of bridge-to-work activities were observed in the case studies, including job shadowing, mentorships and internships. These may be offered in addition to, or in lieu of, work placements. There was considerable variation in the length of these work placements – ranging from less than five weeks to more than sixteen weeks, but the most common length was between six and ten weeks. [ note 28 ]

Exhibit 9: Profile of Bridge-to-Work Opportunities

Source: Evaluation case studies
* Total number of SPO case studies was 32. Some ELT projects included more than one Bridge-to-Work component. The total adds up to more than 100% because multiple responses were possible.

All but three focus groups indicated that they had increased their experience in the Canadian work environment, either through the classroom component (mentioning specifically field trips, guest speakers and/or job fairs) or through their work placements or subsequent jobs found as a result of the program.

Participants in all focus groups felt that they had better job opportunities as a result of the ELT training. However, information from the case studies suggests that some gains in terms of increased knowledge of, and experience with, the Canadian work environment may not necessarily be related directly to increased commensurate employment:

• SPO staff and participants in interviews and focus groups indicated, for example, that returning for further education – whether in a substantive area or in language skills – was a positive outcome of the program, reflecting that participants were aware of the additional skills needed for entry into the workforce.
• In other cases, they indicated that, while the ELT program had given them a more realistic expectation of the requirement to enter the labour market, this increased knowledge of the Canadian work environment might not necessarily lead them to seek commensurate employment, but rather to seek alternative, and perhaps more attainable, employment.
• Participants in three focus groups indicated that they had increased their self-confidence as a result of the ELT program.

These gains may only contribute to improved employment outcomes in the long-term.

3.3.4 Increased Job Finding Skills

ELT projects covered in the case studies provided a range of approaches to increasing job finding skills, including communication skills in the work place, résumé writing, interviewing strategies, experience in conducting research on employers (including Internet searches) and informational interviews, conducting mock interviews and learning how to do “cold calling”. Participants in a majority of the focus groups felt they had increased their job search skills (cited in seven of eleven groups). Respondents in three focus groups specifically noted the benefits of the focus on developing job finding skills in terms of increasing their confidence and sense of professionalism.

3.3.5 Establishment of Mentors/Contacts and Networks

As noted in Exhibit 9, eleven of the 32 case studies included mentorship activities. The mentorships observed in the case studies involve the SPO (through a staff person) setting up a relationship between a participant and a mentor in the participant’s career field who is willing to volunteer approximately one hour a week to meet with the participant and answer career specific questions and provide advice on networking and licensing issues. The nature of the contact between the participant and the mentor can be flexible – either through face-to-face meetings, telephone calls or emails. Mentorships can be used in conjunction with a work placement (particularly for participants who request or require more guidance and support), as well as for those that have not as yet been able to secure a work placement. However, few of the participants in the focus groups conducted for the case studies had participated in a mentorship.

Focus group participants were very aware of the importance of networking. There was limited information from the case studies about the SPOs playing a formal role in the establishment of networks, focusing instead on teaching techniques for networking during the in-class component. One exception was an SPO that hosted monthly career networking meetings for participants. However, participants in four focus groups indicated that they had already established a network with their peers in the ELT program.

3.3.6 Employment Outcomes

In this formative evaluation, the focus has been on assessing the achievement of the immediate outcomes. Information about the extent to which ELT projects have contributed to the longer-term expected outcome of participants having employment that is commensurate with their employment prior to coming to Canada is limited. The achievement of this longer-term outcome will be the focus of the summative evaluation.

The limited information that is currently available for measuring this longer-term outcome is reported here to give an indication of the trends but, perhaps more importantly, to reflect the methodological and data challenges associated with measuring this outcome.

Information on commensurate employment is collected in the ELT database. Unfortunately, there is only information on this outcome for about one-quarter of the participants identified in the database. Of the 592 participants for whom the information is available, 59% are reported to have secured employment commensurate with the employment they had before arrival in Canada.

The case studies provided in-depth qualitative information about the ELT processes and, to some extent, activities, but less quantitative information on the ELT participants’ outcomes. In spite of the outcome reporting requirements, many of the SPOs do not track participant outcomes systematically (see Section 3.5.4). Some information about participant outcomes was available from 21 of the 32 case studies [ note 29 ] and, of these, sixteen provided information on employment outcomes. [ note 30 ] However, this information ranged from specific information on individual outcomes, based on at least a short-term follow-up with participants, to very qualitative information about staff and participant perceptions of the extent to which participants gained commensurate employment upon completion of the project. Of those case studies that reported, albeit often anecdotal, information on employment outcomes, just over half suggested that there were high levels of employment after their participation to an ELT project.  [ note 31 ]

As it will be critical to assess the achievement of this longer-term outcome in the summative evaluation, the program will need to ensure that adequate information is available for the measurement of this outcome.

3.4 Factors Affecting Achievement of Outcomes

The case studies identified factors that both contribute to, and detract from, the achievement of the ELT outcomes.

3.4.1 Positive Factors

Employer Engagement and Work Placements

High levels of employer engagement appear, from the site visit case studies, to have a positive impact on employment outcomes. High level of engagement includes, for example:

• A project in which a group of employers form the SPO. The employers were involved in participant selection, mentored participants throughout the in-class component, provided job shadowing opportunities, provided opportunities for them to participate in the regular employee training and then offered them a paid (albeit at a lower rate than regular employees) work placement.
• Work placements are integrated into the in-class component. Participants were placed in volunteer work placements one day a week during the in-class training component. This offered the opportunity for participants to apply their knowledge on a regular basis and enabled them to raise specific, practical questions in the classroom. To place participants, a comprehensive interview program was undertaken with employers whereby, following the interview, job offers were made and participants were given 24 hours to either accept or turn down an offer. Subsequent rounds of offers were made until all participants were placed.

In addition to the involvement of employers in the project, the work placements themselves appear to be a strength of the program. Focus group participants and SPO staff indicated that the value of the work placements depended on:

• The quality of the person at the SPO facilitating the placement of participants in jobs (job developer) in terms of their knowledge of the local labour market, the time they have to devote to facilitating work placements, their ability to identify an appropriate match, the time available for monitoring the placement and following-up with the participant and the employer;
• The extent to which the job developer involved the participant in the identification of possible placements/ employers;
• The tasks they were asked to carry out as part of the placement – for example, participants indicated high levels of satisfaction when they were asked to apply their technical knowledge and/or be involved in project work. However, those who were asked to do menial tasks or to observe others were less enthusiastic about their work placement experience; and
• Whether the participants had access to provincial projects providing living allowances and/or child care services.

However, there was no noted difference, from information obtained in the focus groups, in outcomes based on whether the work placement was paid or not.

Employers also noted that access to financial supports for participants was important. Those at the SPOs involved in placing participants indicated that it was easier to recruit employers to provide volunteer placements than paid placements. Although employers indicated that they were still required to spend a lot of time and effort training the ELT participants on their particular approaches to their work.

Qualifications of Instructors

It would be expected that the profile of the ELT instructors would have an impact on the quality of the in-class component and, ultimately, participant outcomes. Information from the case studies suggests that most instructors have relevant qualifications for the classroom instruction. For example, over three-quarters of the case study SPOs had staff with experience and/or certification in teaching English as a second language. While generally unfamiliar with the qualifications of their instructors, clients in focus groups provided positive feedback about their instructors, indicating that they were very satisfied with their instructors and the teaching materials. Focus group participants noted that having experience in the occupation, if the curriculum was focused on a specific occupation, was important.

Integrated Activities

Success in some projects covered by the case studies appeared to be associated with projects in which the various components of the program are integrated. For example, in one case the work placements are integrated into the in-class component and participants return for exam preparation after the work placement. In another case, job shadowing is integrated into the in-class component and participants have volunteer and then paid placements. There may also be a correlation between success and the integration of ELT activities with those of other settlement programs and a “suite” of program options are available to participants in a client-centred approach. This needs to be assessed in the summative evaluation when more participant outcome data information, particularly employment outcome, are available.

Strong Labour Markets

A few case studies reflected very strong participants employment outcomes that were, in the interviews, attributed to the strength of the local labour market and/or the high level of demands for employees in the occupations covered by the project. This, of course, raises the question of whether or not participants would have been able to find employment without their participation to ELT program – particularly in the case of strong local labour markets.

3.4.2 Challenging Factors

Recruiting Appropriate Participants

Although the case studies suggest that SPOs use a variety of, and multiple, mechanisms for promoting the ELT projects in the community, they also report having some difficulty recruiting participants with the appropriate language profile for the program they want to offer (CLB Levels 6 and 7). This is somewhat reflected in information from the ELT database, which shows that between 11% and 22% of participants (depending on the competency) entered the ELT program with CLB levels of 5 or below. The program is available to participants with CLB Levels below 6 and 7 in areas where the LINC program is not available for Levels 4 and 5. As a result, those who are shown as entering the ELT with CLB Levels 4 and 5 may reside in areas where the full range of LINC classes is not available. As a result, those who are enter with CLB Levels 4 and 5 may reside in areas where the full range of levels are not available through the LINC program.

Exhibit 10: CLB Levels at Entry into ELT Program*

Source: CIC ELT database, May 2007
* n = between 2235 and 2250, depending on the competency measured

This raises the issue of the extent to which an ELT-type program (with a focus on bridge-to-work) can be effective with candidates with lower language abilities at entry and whether they will continue to experience labour market challenges after completing the ELT program. There are not sufficient data on participants’ outcomes to test whether the CLB level at entry has an impact on their outcomes. However, one case study SPO interviewed did indicate that, when participants are admitted with a range of language ability (e.g., one class with CLB levels ranging from 6 to 9), participants with higher CLB levels found the in-class portion to be less challenging (e.g., lessons targeted at low levels of CLB, role playing and other interactive exercises limited to low levels of CLB, slow progression through the curriculum).

The most common approach to promotion used by the SPOs was promotion through their own organizations or other settlement organizations (using web sites, bulletin boards, referrals from case managers, etc.). This approach used by three-quarters of SPOs in the sample and, according to interviews, SPO staff felt this to be the most effective promotion activity. However, this approach would appear to target newcomers already involved in settlement programs. As some SPOs had difficulty recruiting sufficient numbers of appropriate candidates, this suggests that this targeted promotion is not the most effective means of reaching newcomers with labour market attachment difficulties.

Barriers to Participation

The most frequently mentioned barrier to participation in ELT was financial. The financial barrier was mentioned in seven of the eleven focus groups. Participants talked about the challenge of not being able to work full-time to support themselves and their families while participating in the ELT program. This was less of an issue in Alberta and Saskatchewan, where clients are able to apply for a provincially-funded living allowance.

Additional barriers mentioned in the focus groups included lack of child care (mentioned in six focus groups), waiting lists (mentioned in three focus groups), location of the SPO offering the classes (mentioned in two groups) and general lack of awareness of the ELT program (mentioned in two groups).

Although ELT offers some supports to participants, including child minding services and transportation supports, information from the case studies suggests that about half the SPOs do not offer these services to participants. Some other supports are provided through the complementary CIC settlement programs (e.g., Host, ISAP), although it did not appear that a lot of clients were accessing these services. Clients in the focus groups did note that they were satisfied with these services when they were available, particularly during the work placements.

3.5 Program Administration

3.5.1 Partnerships with Provinces

Agreements signed with British Columbia, Manitoba and, recently, Ontario include an ELT appendix, which outlines how the additional funds earmarked for ELT-type projects should be spent. The other provinces and territories have not signed special agreements with CIC. [ note 32 ] Among the provinces with agreements, all responsible parties indicated that the roles and responsibilities of the federal and provincial/territorial governments are clear and well understood. However, the nature and frequency of reporting (at the project, participant and outcome levels) from British Columbia and Manitoba was considered by CIC NHQ representatives to be limited, as very little is known about the projects undertaken in these provinces. Representatives from these provinces indicated that they are aware of this concern and are willing to work with their federal counterparts to improve reporting.

Key informants in Ontario indicated that they are hoping to explore opportunities for collaboration and contracting that will serve to minimize duplication and maximize program offerings to newcomers in that province.

Among respondents from provinces/territories without an agreement, the partnership between the province/territory and CIC was generally considered appropriate. CIC regional respondents in all provinces indicated that they have good working relationships with provincial representatives and are looking forward to working on ELT in the future, as a result of the recent decentralization of the program.

3.5.2 Communications/Relationships with CIC

The evaluation found that, generally, relationships between CIC and provinces/ territories are strongest at the regional and local office level (particularly in cases where the CIC and provincial/territorial offices are co-located). Some challenges related to the management of ELT by NHQ cited by provincial/territorial representatives include:

• Lack of face to face meetings;
• High turnover among ELT staff resulted in a lack of corporate memory related to issues pertaining to provinces’ and territories’ particular situation/context; and
• Poor communications regarding matters that would assist provinces/territories in their programming/support of ELT (e.g., the amount of ELT funding available per year, the status of ELT as a “pilot”, the sustainability of funding, the status of an assessment tool, a list of funded projects).

SPO respondents generally felt that communications with CIC were adequate, although many frustrations were voiced by SPOs regarding the overall management of ELT and communications with SPOs:

• High turnover at CIC;
• Delays in negotiating contribution agreements;
• Lack of continuity of funding (often resulting in staffing uncertainties and/or having to lay off staff, inability to promote to/conduct outreach with/attract new participants);
• Lack of coordination of programs for newcomers (including the various ELT projects) and a broader vision of immigrant settlement;
• Inconsistent messages regarding claimable expenses; and
• Changes in the reporting template.

3.5.3 Applications

In the Call for Proposals in 2006/07, 45 proposals were received from SPOs in the Atlantic provinces, Ontario and the Yukon. [ note 33 ] In the Atlantic provinces, a total of 3 proposals were received, one for a development project and two for projects that included both a development and delivery component. In Ontario, 41 proposals were received, including four development projects, 15 delivery projects and 22 projects with both a development and delivery component. One proposal was received from the Yukon for a delivery project.

Generally, respondents from CIC, British Columbia and Manitoba were satisfied with the number of applications received. Some key informants suggested that CIC could attract a larger number of well-established SPOs to submit proposals if their RFPs were modified. Their suggestions included:

• Eliminating the requirement to cost share the project;
• Offering multi-year funding;
• Reflecting any specific needs identified in the RFP e.g., projects targeting remote communities, women, or specific occupations/sectors;
• Allowing for more lead time for project design and set-up;
• Conducting targeted promotion of the RFP for ELT (i.e., approach specific SPOs to submit proposals); and
• Improving the payment of claims process (e.g., clearer procedures, more timely payments).

3.5.4 Current Practices Regarding Reporting

SPOs with a delivery project (except those in British Columbia, Saskatchewan or Manitoba) are required to complete a statistical report annually. Both delivery and development projects must prepare quarterly narrative reports on completed activities, planned activities, achievement of performance objectives, and an assessment of successes, obstacles and opportunities. Finally, a final project report must be submitted that includes: project history, activities undertaken, outputs and outcomes achieved, analysis (including successes, challenges, unexpected outcomes and mitigating circumstances), lessons learned and recommendations for future projects, and a summary statistical report (for delivery projects outlining statistics related to intake and screening, program outcomes, and employer involvement).

Overall, most SPO representatives (outside British Columbia, Saskatchewan and Manitoba) indicated that, while the reporting processes have improved, they continue to feel that there is a lot of reporting required. In provinces where projects are co-funded by the province (e.g., Alberta), SPOs voiced frustration over the need to report to both funders in different formats at different times (since agreements with each funder are usually signed in different months).

In British Columbia, SPOs with development projects are required to submit monthly reports to the Ministry of Economic Development, in addition to a final report and the final deliverable(s). SPOs with delivery projects through Skills Connect report to the province via an online database. This database does include outcome fields. These SPOs must also provide monthly reports to the Ministry. SPO respondents in British Columbia indicated that the online database is a good tool but also has some limitations – for example, related to the inability to change data and generate reports.

The Canada-British Columbia Agreement (2004) indicates that a report format, specific to the ELT initiative, and an evaluation framework will be developed jointly by British Columbia and Canada. There is, however, no evidence that CIC and the Government of British Columbia have jointly produced a report format or evaluation framework. The ELT RBAF indicates that British Columbia has “implemented an outcomes-based management system for its contribution agreements with end recipients. CIC is satisfied that the British Columbia regime for monitoring of agreements is of sufficient rigour to meet Government of Canada requirements.” [ note 34 ] However, interviews with CIC representatives for this evaluation indicate that results are not being shared with CIC. Hence there is a need for improvement in the sharing of lessons learned, best practices and evaluation findings.

In Manitoba, SPOs must provide a quarterly narrative report, an annual financial report and an annual narrative report to the province. SPOs are not required to formally track individual participant outcomes. Like the British Columbia agreement, the Manitoba – Canada ELT Agreement (2004) indicates that an annual report capturing on-going performance indicators specific to the ELT initiative and an evaluation framework will be developed jointly by Manitoba and Canada. There is no evidence, however, that these deliverables have been produced. The ELT RBAF indicates that the Manitoba Immigrant Integration Program (MIIP) conducts activity monitoring that includes a qualitative and quantitative assessment of the project’s success in achieving objectives. It does not appear that data are collected or monitored pertaining to participant outcomes. No assessment of the adequacy of reporting is provided in the RBAF, although CIC interviewees indicated that Manitoba does not share results information with CIC.

Although Saskatchewan does not have a settlement agreement, it does have one contribution agreement with CIC that covers both the development of a curriculum and the delivery through two immigrant-serving organizations in the province. As part of the development of the curriculum, the Government developed an electronic data collection tool that collects information for both ELT and Immigrant Integration Program (IIP) projects. It includes information on clients, program processes, program supports, language training and work experience components, career mentoring and client services and an employment action plan. However, the case study in Saskatchewan indicates that the tool may not yet be used systematically.

CIC ELT Database

Rather than integrating the ELT data collection requirements into the existing CIC database for other settlement programs – iCAMS – the ELT program management decided to create a separate database for the ELT program. This was in part motivated by the costs of integrating an ELT component into iCAMS, but also by the fact that, at the time, iCAMS had not set up systematic reporting on the settlement programs that it covered. ELT program management was concerned that it would not have adequate information for monitoring and evaluating the program and its results and so chose to set up the independent database. However, as will be seen later in this section, there have been a number of challenges with the creation and maintenance of this database.

The database was set up with modules for collecting information on funded projects and individual participants. Program staff at NHQ input information on funded projects. SPOs (except those in Manitoba, Saskatchewan and British Columbia) are required to provide information on participants, by completing a hard copy questionnaire for each participant. These questionnaires are forwarded to CIC NHQ at the end of the project and the data are entered into the database by staff at CIC NHQ. The questionnaire includes information on the ELT program, participant information, participant employment and educational history and current profile and language assessment results, other CIC or federal government services and program fees.

The initial database was modified in the second year of the program. These modifications increased the amount of information that SPOs were required to report. As noted above, the SPOs find completing the questionnaire onerous. Some SPOs also indicated that they had reservations about collecting and sharing some of the required information with CIC. As a result, there have been challenges in completing these questionnaires and maintaining the ELT database, which have resulted in limitations to the data available for evaluation of ELT.

Database Limitations

The expectation in the design of this formative evaluation was that the database would provide considerable information on the program and its participants. However, it became clear that there were considerable limitations to the information available from the database.

The first challenge is that, at the time of the evaluation, the database did not include information on all projects and/or participants. [ note 35 ] As a result of the different delivery structures in different provinces, data on projects and participants is not being collected in the same way in all provinces. The implications are that consistent data is not available for Manitoba, Saskatchewan, British Columbia and, for one year, Ontario:

• In provinces where CIC has a contribution agreement with the province for the delivery of settlement services (Manitoba and British Columbia), there is no obligation on the province to report detailed participant information. In addition, the information in the annual reports from the provinces does not allow CIC to identify with certainty all projects with ELT funding. Where it can, CIC completes the records with information obtained from the province more informally but, as a result, the project data for these provinces may be incomplete.
• CIC has an umbrella agreement with the Government of Saskatchewan, which, in turn, arranges for the delivery of ELT services in the province. As a result, the information on projects or participants from Saskatchewan is not complete.

As a result of these gaps, the number of participants identified in the database is less than the number of participants in ELT-funded activities nationally. However, it is understandable that, given the program delivery structures, information would not be available for all projects or participants. However, in addition to these records missing because of the project delivery structure, there are limitations to, and inconsistencies in, the participant data that is available. This has occurred, in part, because CIC changed the data reporting format during the course of the program. These limitations include:

• There are gaps in the individual participant data. Analysis of the data suggests that the records are more complete for the first two years of the program (2003/04 and 2004/05) than for subsequent years. SPOs are only required to submit the data questionnaires at the end of their contribution agreement with CIC. [ note 36 ] As result, those with multi-year agreements may not yet have submitted their questionnaires. For example, Alberta and Ontario are underrepresented in the actual database and that is likely because some of the SPOs have received multi-year funding in 2005-06. In addition, data from about 1,500 questionnaires received from SPOs were not entered by CIC when the evaluation was conducted.
• Even for those participants for whom some information is available, information is not available for all variables and there is very limited information on participant outcomes.
• Data analysis is complicated by the fact that there is no fixed format for data entry which means that considerable data cleaning is required before meaningful analysis can be undertaken.
• Finally, there are concerns about the quality of the data for some variables.

See Appendix B for a summary of data quality issues, by key variable. These data limitations will have implications for the ELT summative evaluation.

3.6 Alternatives

All provinces and territories operate programs that specifically target newcomers. These programs can include language training, employment assistance, or both. Also, newcomers are eligible to receive services in existing employment services programs operated by the province/territory. Some examples of provincial/territorial programs that may be comparable to ELT include:

• Immigrant Integration Program (IIP) in Saskatchewan is a workforce integration program being piloted in Saskatoon for people with low CLB levels.
• Nova Scotia provides funding to SPOs specifically to assist them to find employment for newcomers. The province also funds an ELT program with the same objectives as the federal government; however, the provincial projects are delivered to a unique clientele (Canadian citizens born abroad and provincial nominees).
• The province of Ontario has many programs for language training and workforce integration. Those that include both components that would be comparable to ELT include:
  • Adult credit programs in high schools for ESL/FSL that include a co-op component.
  • Special Language Training Pilot Program. All classes will be completed by August 31, 2008. Proposals must include targets and how the project will be assessed. The program is directed at school boards and partnerships are encouraged. The two main areas of focus are language for market sectors and language in the workplace.
  • Bridging program. Includes $33M of Ontario Government money and some projects are jointly funded with CIC (currently funding 72 projects with organizations including universities, regulatory bodies and community groups). Projects focus on skilled immigrants and professionally trained individuals and most projects include a language assessment or upgrading component. The project funds three main models of projects: projects that research and aim to better understand requirements for systemic change; projects that work to get internationally trained immigrants (ITIs) registered within regulated professions; projects that aim to help ITIs get a job in their field.
• In Alberta, in addition to co-funding ELT projects, the province also has Integrated Bridging Programs, which are employment focused and run for 3 years at a time.

In British Columbia and Manitoba, ELT funding is directly funnelled into provincial ministries and integrated with provincial funding.

In terms of similar programs funded by other federal departments, the evaluation identified:

• Health Canada funds the International Educated Health Professional Initiative (IEHPI) ($75M over 5 years with $6.5M to provinces and territories and $7.4M for pan-Canadian initiatives for seven priority professions that the regions have identified). The department provides funds to the provinces/territories to invest in preparedness, assessment, faculty development, remediation programs, programs to integrate professionals, including: medicine, nursing, pharmacy, physiotherapy, occupational therapy, lab technologist, and medical radiation technician. There is a broader health human resources strategy that focuses on recruitment and retention of health providers. The objective is to increase the supply of health care providers.
• HRSDC has many programs aimed at helping people find jobs. However, the most commonly cited HRSDC program considered similar to ELT was the job finding clubs. [ note 37 ]

_______________

[22] In some cases, the experts reviewed more than one tool from the same project. The experts were provided with a draft case study report for each case study so that they would have information on the context in which the tools were developed and being used.

[23] For example, in the case of “process”, seven of the thirteen tools reviewed received positive comments (“high ranking”), mentioning one or more of the factors related to the process of the development of the tools (see list above). However, information was not available about the process for the remaining six tools.

[24] Based on two sources of information: findings of the case studies with 32 SPOs (twelve during site visits and twenty by telephone); and the ELT database.

[25] Note that CLB-based tests are placement tests and are not intended to be used as exit tests. The fact that the assessment of language skill improvements is based a) on different tests at entry and exit and b) the exit tests are not standardized across all projects makes this data on language skill improvements somewhat unreliable. In addition, there was more missing data in the fields for the exit levels than for the entry levels.

[26] This is an interview set up and conducted by participants with employers in their occupational area, in order to collect information about the occupation in Canada. Its purpose is not primarily to seek employment.

[27] Not all programs include work placements. Some cannot do so because of limitations on the extent to which the participants are eligible to work, even in unpaid positions – either because of licensing issues or union regulations.

[28] Note that some work placements are limited in length by labour union regulations governing non-unionized workers.

[29] Some projects were not yet completed.

[30] For five projects, participant outcomes are not applicable (because they were development projects or delivery projects that were not yet delivered.

[31] “High levels” mean employment for 80% to 100% of participants. There is little information on whether they gained commensurate employment.

[32] However, the Government of Saskatchewan is the sole ELT CA holder in that province and issues sub-contracts for ELT projects.

[33] Enhanced Language Training CIC Staff Training Kit, Ontario Region, updated June 9, 2006. Page 29. CFPs are issued separately in BC, Manitoba, Alberta and Saskatchewan.

[34] Risk-Based Audit Framework Enhanced Language Training, CIC, September 2005, p. 14

[35] The program has received data for an additional 1,500 participants from projects starting in 2005/06 that will be entered into the database shortly.

[36] CIC can hold back 10% of the funds owed to the SPOs until the questionnaires have been submitted.

[37] It was not possible to set up an interview with the identified HRSDC representative.

• Previous
• 1
• 2
• 3
• 4
• 5
• 6 (current)
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17
• 18
• Next