Evaluation of the Language Instruction for Newcomers to Canada (LINC) Program

2.0  Methodology

Overview and Limitations

The methodology for the LINC evaluation study consisted of the following components:

• a review of program and related documentation;
• interviews with 20 key informants;
• a telephone survey of 150 service provider organizations (SPOs);
• 12 focus groups – eight with current learners/clients and four with graduates; and
• an integrated analysis.

The methodological approach was guided by the evaluation issues, indicators and data sources specified in the Evaluation Framework for Citizenship and Immigration Canada’s Settlement Programs (January 23, 2004). To the extent possible within the available budget and time frame, this methodology incorporated multiple methods and data from different primary and secondary (existing) sources in order to ensure that the findings were valid and captured all key points of view on the LINC program. A limitation, however, was that it was only feasible to consult program clients in a qualitative manner—in the 12 focus groups. Consequently, some of the quantitative indicators noted in the Evaluation Framework were not available (e.g., the percentage of clients who feel language training has helped them meet personal goals, the percentage who report increased confidence in their ability to achieve future goals). These issues could only be examined in a qualitative manner. Moreover, because a client survey was not done, there was no comparison group of newcomers who have not participated in LINC—a feature that would have yielded more conclusive findings regarding incremental program impacts. The decision not to include a survey was due largely to the logistical (and financial) difficulties associated with surveying clients/newcomers who may not have been completely comfortable being interviewed in English or French (i.e., this approach would have required a team of interviewers capable of speaking numerous languages).

Documentation Review

We reviewed program and related documentation to gain an understanding of the LINC program’s environment, mandate, objectives, activities and expected results. This review commenced during the research design stage to guide the development of the evaluation methodology and instruments. In addition, a small-scale review of available program documentation and CIC statistics was useful to help address some evaluation issues related to the continuing need for the program and program delivery.

Key sources of documentation and data reviewed include the following:

• Evaluation Framework for Citizenship and Immigration Canada’s Settlement Programs. Prepared for CIC by Goss Gilroy Inc., January 23, 2004;
• Citizenship and Immigration Canada: Performance Report for the Period Ending March 31, 2003. Minister of Public Works and Government Services Canada, 2003;
• Citizenship and Immigration Canada: 2002–2003 Estimates Part III – Report on Plans and Priorities. Minister of Public Works and Government Services Canada, 2002;
• Information available at the website, www.integration-net.cic.gc.ca;
• Citizenship and Immigration Canada: Settlement Programs, Schedule 4 – Supplementary Terms and Conditions;
• Program administrative data (entered by LINC SPOs) available in the Immigration Contribution Accountability Measurement System (iCAMS), which includes data on services delivered to clients and client characteristics;
• The Longitudinal Survey of Immigrants to Canada (LSIC), conducted by CIC in collaboration with Statistics Canada, which provides information on new immigrants’ adaptation to and integration into Canadian society, as well as the impacts of various services/policies on integration;
• The Landed Immigrant Data System, which provides a profile of immigrants (but not current contact information); and
• The Immigration Database, which links immigrants’ attributes at landing to subsequent personal income tax files and, therefore, provides longitudinal data on immigrants’ economic performance and mobility.

Key Informant Interviews

A total of 20 key informant interviews were conducted with CIC managers and officers involved with the LINC program at National Headquarters (NHQ), regional offices and local offices, as well as with some SPOs (those that were not reached in the survey). A list of names and contact information of interview candidates was developed with assistance from the Evaluation Steering Committee. Most of these interviews were conducted by telephone and were an average of 60 minutes in duration. A few key informants located in the National Capital Region and in cities we visited for focus groups were interviewed in person. All interviewees were assured of confidentiality.

An introductory letter and interview guide were developed and sent to all key informants by e-mail in advance of their appointment, so they could prepare for the interview. Two separate instruments (an interview guide for CIC staff/managers and a questionnaire for SPOs) were used [note 3]. Summary notes were prepared for internal use for the integrated analysis stage of the evaluation.

Survey of Service Provider Organizations

We conducted a telephone survey of 150 service provider organizations (SPOs) that are involved in language assessment and/or training for the LINC program. The interviews were conducted by fully trained interviewers from our Computer-assisted Telephone Interviewing survey centre. Interviews were an average of 30 minutes in duration and were done in the respondent’s preferred official language. The questionnaire included several closed items (with responses indicated on scales/ categories), as well as a few open-ended questions. In addition, the questionnaire included questions targeted at SPOs involved in language assessments, as well as those involved in language training. Each respondent was interviewed for as long as he/she needed to complete all questions.

CIC provided us with an electronic listing of names and contact information for 205 SPOs, including some involved in language assessments, some involved in language training, and some involved in both activities. After updating some information, we were left with a list of 180 SPOs. The regional distribution of these 180 SPOs was as follows:

• Atlantic Canada (n=10);
• Ontario (n=129);
• Prairies (n=40); and
• Yukon (n=1).

In order to help ensure a good response rate, all 180 SPOs were first e-mailed or faxed an introductory letter, describing the purpose and importance of the survey and assuring them of confidentiality, along with a copy of the questionnaire. About one week later, we began with a pretest of the survey questionnaire involving approximately 20 respondents. This was done to check that the wording of questions was clear, the question flow and skip logic were appropriate, and the interview was the intended average length. Following a few minor refinements to clarify the wording of some questions, our interviewers proceeded to complete the remaining interviews.

The survey sample of 150 SPOs included:

• SPOs of different sizes, in terms of the number of classes/groups of LINC clients they have annually – less than five classes (28 percent), five to nine classes (25 percent), 10 to 19 classes (22 percent), and 20 or more classes (25 percent);
• SPOs involved in assessment (13 percent), language instruction (83 percent), or both (five percent); and
• SPOs in all participating regions – Atlantic Canada (6 percent), Ontario (72 percent), Saskatchewan (six percent), Alberta (15 percent) and the Yukon (1 percent).

When the survey was completed, we conducted a basic quantitative analysis of the data, including the frequency distribution of responses and (where appropriate) the average response for each question, as well as cross-tabulations by variables of interest (e.g., size of SPO) [note 4].

Focus Groups with Clients

We conducted a total of 12 focus groups with LINC clients in six cities in participating provinces. Half of the focus groups were held in Ontario, because this province receives the most immigrants/refugees and the most LINC funding. As indicated in Table 2.1, the 12 focus groups included seven groups with LINC clients, one group with CLIC clients (who are learning French), and four groups with LINC graduates.

Table 2.1
Distribution and Composition of Focus Groups

Each focus group included an average of five to nine participants (both males and females) of varying ethnic/linguistic backgrounds (e.g., people from Iran, Iraq, Afghanistan, South Korea, China, Thailand, Russia, the Ukraine, Belaruse, Albania, Guatemala, Columbia, Egypt, Cameroon, Congo, Sudan and Kenya). Some of the graduate groups were smaller, however, because of difficulties in contacting these people (i.e., due to outdated contact information).

In order to recruit suitable participants for each focus group, we first needed to develop a list of current trainees as well as graduates by seeking the assistance of local SPOs (which were recommended by local or regional CIC officials). We asked SPOs to provide the name, phone number and first language of prospective participants in each city. In order to ensure that the current clients had at least basic language skills in English or French, as well as sufficient experience with the program so they could provide informed opinions, the focus group participants were selected to meet the following two criteria: (1) they must have reached Canadian Language Benchmarks (CLB) levels 5, 6 or 7 in language ability; and (2) they must have progressed through at least two CLB levels while in LINC. We asked SPOs to provide a listing of all of their current trainees and graduates who met these criteria or, if this was not possible, a random sampling of these names (e.g., every nth name on their course lists). We also provided the SPOs with an introductory letter, which they could pass on to their clients, advising them of the upcoming group discussions.

After a comprehensive listing had been prepared, we recruited participants for each focus group by telephone. We described the purpose, time, location and other details of the focus group and invited the clients/graduates to attend the discussion. (In some cases, where it was difficult for us to reach clients/graduates, the SPOs assisted with recruitment by contacting students while at their facility.) In addition, we gave each participant a reminder phone call a day before his/her scheduled focus group. We endeavoured to recruit eight participants per group (as long as sufficient names were available).

All focus groups were held at SPO facilities in each city. Each SPO providing a meeting room was given an honorarium of $150 per group (i.e., to compensate them for the use of their meeting room and for their assistance in providing a list of potential participants and making arrangements for snacks/refreshments). Each group discussion was two hours in duration and was moderated by a skilled moderator. Each discussion was audiotape recorded to assist us in the preparation of summary notes used internally for analysis. Snacks and refreshments were served at each group, and all participants were offered an honorarium of $25.

Integrated Analysis

Following the completion of all data collection for the LINC evaluation, the results of each methodological component were analyzed. The findings from the various lines of evidence were then integrated and organized by the evaluation issues. In the integrated analysis, the evidence from different sources was triangulated to identify the issues on which the evaluation findings converged and also to help reconcile any incomplete or contradictory findings. The integrated evaluation findings are presented in Chapter 3.

____________

3. These and all other instruments utilized in this study are provided in a separate document: Evaluation of the Language Instruction for Newcomers to Canada (LINC) Program – Data Collection Instruments, April 30, 2004.

4. The complete survey results are presented in a separate document: Evaluation of the Language Instruction for Newcomers to Canada (LINC) Program – Technical Report on Survey of Service Provider Organizations, April 30, 2004.

5. Initially, efforts were made to convene a focus group discussion with CLIC clients in Ottawa. Due to the transition from one SPO to another at that time, however, it was not possible to obtain a listing of CLIC clients who met our selection criteria. So, the location of the CLIC focus group was changed to Fredericton.

• Previous
• 1
• 2
• 3
• 4 (current)
• 5
• 6
• 7
• Next