Evaluation of the International Student Program

2. Evaluation framework and methodology

2.1 Evaluation framework

This study was guided by an Evaluation Framework developed in consultation with program representatives, and completed in October 2008 (see Appendix A). The evaluation methodologies were later confirmed and the availability of data was assessed in an initial planning phase. The evaluation questions are outlined thematically below.

2.2 Methodology

The evaluation used multiple lines of evidence, as well as both qualitative and quantitative research methods. It examines results of the ISP post-IRPA, providing findings for the 2003 to 2008 reporting period.

2.2.1 Document review

Documents reviewed for this evaluation were of four general types:

• Government of Canada and legislative documents, such as the Speech from the Throne, Budget, Advantage Canada, the Immigration and Refugee Protection Act (IRPA) and IRP Regulations;
• Departmental website and documents, such as Reports on Plans and Priorities (RPP), Departmental Performance Reports (DPR) and Annual Reports;
• Documents and websites of other government departments (CBSA, DFAIT, IC , etc.) and stakeholders, such as CBIE, AUCC, ACCC;^{Footnote 18} and
• Program and policy documents and operational documents.

2.2.2 Interviews

Forty-three interviews were conducted with ISP program stakeholders by telephone and in-person. Interview guides were provided to all interviewees in advance of the interviews. The breakdown of interviews is shown below. See Appendix B for the interview guides.

2.2.3 Surveys

Four surveys were conducted during the evaluation: three conducted internally by the CIC Evaluation Division and one conducted by the Canadian Bureau of International Education (CBIE).

International Students: The Canada First Survey of International Students, conducted by the CBIE, examined the study and work experiences of international students in Canada.^{Footnote 20} The CIC Evaluation Division used data from this survey for the evaluation. Fifty universities and colleges were contacted to participate; and 22 universities and 4 colleges agreed to participate. Institutions were asked to randomly sample 1,000 of their international students,^{Footnote 21} and student respondents completed the survey online.^{Footnote 22} The overall student response rate was 24% (n = 5,925). See Appendix C for the survey questionnaire.

Visa Offices: An online survey was administered to CIC visa offices abroad to gather information on student application processing, its issues and challenges and the activities in place to support and monitor program delivery. Fifty visa offices responded to the survey out of a possible 61 visa offices that process student applications, for a response rate of 82%. See Appendix D for the survey questionnaire.

Educational Institutions: A survey of educational institutions was administered online. Invitation emails were sent by CBIE and Languages Canada to 103 international student advisors from various universities and colleges and 144 representatives from language schools.^{Footnote 23} Representatives were asked to complete only one questionnaire per educational institution – preferably by an individual with a function directly linked to international students or internationalization strategies. Sixty respondents completed the questionnaire, for a response rate of 24%. See Appendix E for the survey questionnaire.

Employers: Employers were surveyed on their awareness and use of the ISP and the program’s strengths and limitations. It was requested that the survey be completed by a person at the company who is primarily responsible for staffing or human resources. CIC worked with the Canadian Association of Career Educators and Employers (CACEE)^{Footnote 24} to distribute the survey to 125 of their members. Twenty-two surveys were completed, for a response rate of 18%. See Appendix F for the survey questionnaire.

2.2.4 Analysis of administrative data

Key program data is gathered on an ongoing basis through CIC’s Field Operations Support System (FOSS) and Computer-Assisted Immigration Processing System (CAIPS). Both research statistics (stocks and flows) and operational statistics, related to the number of study and work permit applications were used in this study. Whenever possible, data already compiled in CIC’s Facts & Figures were also used.

2.2.5 Literature review

A literature review and comparative study of the design and delivery mechanisms for international student programs was commissioned by CIC for this evaluation and conducted by independent researchers. The study analyzed multiple sources to report on Canada’s market share of international students in relation to its chief competitors, and looked at policy, design and implementation differences between Canada’s international student program and similar programs in other countries (the United States, UK, Australia, France and New Zealand).

2.3 Limitations

• Data from CBIE’s Canada First Survey of International Students does not include information on international students in British Columbia. The Province of BC conducted a similar survey of international students in the Fall 2008, and institutions in BC declined to participate in the CBIE survey. Where appropriate, findings from the BC survey were used to represent the views of international students studying in this province.^{Footnote 25} The representation of respondents for both surveys was heavily weighted towards the university level (CBIE: 93%; BC: 76%). However, international students at the university level are Canada’s largest cohort group.
• Data from the survey of employers could not be used, as its low response rate made it unrepresentative. This limitation was addressed by using information related to employment from other lines of evidence, such as the student survey and document review.
• There were some gaps in CIC’s administrative data. Departmental data on field of study were not available, and information on the institutions at which international students are studying could not be used, as the coding information is out of date. Steps were taken to minimize the impact of these gaps by using other administrative data, the document review and extrapolating from the student survey data.

• Previous
• 1
• 2
• 3
• 4
• 5 (current)
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• Next