Evaluation of the International Student Program
2. Methodology
2.1 Evaluation Scope
The scope of the evaluation was determined during the planning phase, in consultation with CIC Branches involved in the delivery of the ISP. The evaluation examined the International Student Program, as well as the use of the Off-Campus Work Permit Program and the Post-Graduation Work Permit Program. The outcomes of these work programs were not included in this evaluationFootnote 1. In addition, the evaluation addressed the five core issues identified in the Treasury Board Secretariat Directive on the Evaluation Function.
2.1.1 Calibration
Given the requirement to evaluate the ISP to meet departmental evaluation coverage requirements, the evaluation was calibrated by:
- Limiting the methodology to three lines of evidence - key informant interviews, document review, and administrative data reviewFootnote 2;
- Focusing solely on assessing the Program's immediate expected outcomes including addressing issues identified in the previous evaluation regarding program integrity as well as application processing; and
- Conducting the evaluation in-house with a team of four evaluators and a total of 180 days allocated to complete the evaluation.
Due to the timing of the regulatory changes which took effect in 2014, the evaluation is expected to serve as a benchmark for the evaluation of the ISP planned for 2018/19. It is anticipated that the timing of the next evaluation of the ISP (scheduled for 2018/19) will allow for a complete assessment of both immediate and intermediate outcomes.
Other ongoing departmental work relating to international students also influenced the decision to scale down the evaluation. CIC is currently conducting an evaluation of the Canadian Experience Class (CEC) which is examining pathways from temporary to permanent residency of clients from 2008 to 2013 (including international students). The CEC evaluation is expected to be completed in fall 2015. As a result, the current ISP evaluation is not examining in detail this aspect of the Program. In addition, the department is conducting a review of the Post-Graduation Work Permit Program, as part of the broader Temporary Foreign Worker Program review announced by the Government of Canada in 2012. The review will present a profile of its clients from 2002 to 2013, including economic outcomes of those who transition to permanent residency, with the overall objective of assessing whether or not the program continues to advance Canada's broad economic and cultural national interest such that it continues to warrant an exemption from the Labour Market Impact Assessment process. This review is expected to be completed in Winter 2014/15.
2.2 Evaluation Issues and Questions
In accordance with the requirements of the Treasury Board Secretariat Directive on the Evaluation FunctionFootnote 3, the evaluation assessed the relevance and performance of the ISP from 2009 to 2013. The performance of the ISP was assessed through the following immediate expected outcomes, which is also outlined in the ISP logic modelFootnote 4:
- Shared understanding among ISP stakeholders of roles, responsibilities and policy and program objectives and effective relationships;
- Responsive programs and policies that facilitate study and work opportunities for international students, and transition to permanent residency;
- Selection processes are timely and consistent while maintaining program integrity;
- International students arrive and study in Canada; and,
- International students gain Canadian work experience.
The evaluation questions, organized by core issues, are presented in Table 2-1Footnote 5.
Core Issues | Evaluation Questions | Section |
---|---|---|
Continued Need for the Program | To what extent is there a continued need for the International Student Program? | 3.1 |
Alignment with Government Priorities | To what extent is the International Student Program aligned with departmental and government-wide priorities? | 3.2 |
Alignment with Federal Roles and Responsibilities | To what extent is the International Student Program aligned with federal roles and responsibilities? | 3.3 |
Core Issues | Evaluation Questions | Section |
---|---|---|
Achievement of Expected Outcomes: Program Management Outcomes | To what extent is stakeholder engagement related to the ISP effective? | 4.1 |
To what extent do ISP policies and programs facilitate study and work opportunities for international students? Do they facilitate transition to permanent residency? | 4.2 | |
To what extent are international students taking advantage of work opportunities and gaining Canadian work experience? | 4.2 | |
To what extent has CIC addressed the recommendations made in the previous evaluation related to program integrity and application processing? | 4.3 | |
Achievement of Expected Outcomes: Processing Outcomes | To what extent do students arrive and study in Canada? | 5.1 |
To what extent have selection decisions been timely and consistent? What impact have CIC's Modernization initiatives had on the ISP? | 5.2-5.4 | |
To what extent do selection processes support program integrity? | 5.5 | |
Demonstration of Efficiency and Economy | To what extent are the program's resources managed effectively to facilitate the achievement of outcomes? Are costs in line with what would be expected in other similar programs? | 6.1-6.2 |
To what extent are there alternatives to the current design and delivery of the International Student Program that would improve efficiency or economy? | Footnote * |
2.3 Data Collection Methods
The evaluation of the ISP included three lines of evidence which are described in greater detail, below. Data collection and analysis for this evaluation took place between February and September 2014.
2.3.1 Document Review
A review of relevant program documentation was conducted to provide both background and context of the ISP, as well as to inform the assessment of the Program's relevance and performance. Government of Canada documents, such as Speeches from the Throne, Budgets, and policy and strategic documents were reviewed for context and for information on CIC and Government of Canada priorities. Departmental documents, including the 2010 ISP evaluation, processing manuals, and monitoring reports were used to address specific evaluation questions. Documents from other government departments were also examined to assess program relevance (e.g. DFATD documents on the International Education Strategy). Appendix B contains a bibliography of documents used in this report.
A small scale international comparison on processing times and work opportunities for international students was conducted. This comparison examined the study programs for the United States, the United Kingdom, and Australia in relation to Canada. These three countries were chosen as they are members of the Five Country Conference, and often viewed as Canada's direct competitors for international studentsFootnote 6.
2.3.2 Administrative Data Analysis
Administrative data from CIC's Global Case Management System (GCMS) and the Field Operations Support System (FOSS) was used to assess the volume of international student applications, entries, and numbers present on December 1 and develop a demographic profile of international students including information on the number, type of study permit and study temporary resident visa (TRV) applications, and the time utilized to process them. The CIC Financial Management Branch also provided four years of data on the costs of the ISP, which was used to assess the Program's resource utilization. Only four years of financial data was available for this evaluation as CIC did not conduct a Cost-Management Model exercise in FY 2008/09.
2.3.3 Interviews
Interviews were a key line of evidence for the majority of evaluation questions. A total of 33 interviews were completed with representatives from five stakeholder groups.
Response Group | Number of Interviewees |
---|---|
CIC Senior Management and Staff (Immigration Branch, Operational Management and Coordination, International Region) | 9 |
Staff at CIC Regional/Local Offices, Central Processing Region, and Canadian Visa Offices Abroad involved with processing study applicationsFootnote 7 | 12 |
Representatives from provincial governmentsFootnote 8 | 5 |
Senior management from the Canadian Consortium for International Education (CCIE) that represent more than 500 education institutions and school boards across CanadaFootnote 9 | 5 |
Representatives from other federal departments involved in the ISP (DFATD, CBSA) | 2 |
Interviews were conducted both in-person and by telephone depending on the preference and availability of interviewees. Where qualitative information is presented in the report, the response scale shown in Table 2-3 was used.
Descriptor | Meaning |
---|---|
All | Findings reflect the views and opinions of 100% of the interviewees |
Majority/Most | Findings reflect the views and opinions of at least 75% but less than 100% of interviewees |
Many | Findings reflect the views and opinions of at least 50% but less than 75% of interviewees |
Some | Findings reflect the views and opinions of at least 25% but less than 50% of interviewees |
A few | Findings reflect the views and opinions of at least two respondents but less than 25% of interviewees |
Limitations and Considerations
The evaluation encountered several data limitations. These, along with mitigation strategies are described in this section. It is important to note that none of the following limitations had a significant impact on the evaluation findings. As a result, data presented in support of evaluation findings should be considered reliable and evaluation findings can be used with confidence.
Limited information and reliability concerns related to data on short-term studentsFootnote 10—data for this population in GCMS and FOSS is only captured for visa-required countries, therefore it does not represent the full population of short-term students. In addition, the department has noted inconsistencies in coding of visas for short-term students who are study permit-exempt (e.g. sometimes coded as visitors other times as short-term students). As a result, the number of applications approved for study TRVs was used as a proxy to estimate the minimum number of this population in Canada (i.e. only visa-required countries). This data issue on short-term students did not impact the results of the evaluation, as this information was presented only to describe the number of entries and assessing the timeliness of study TRV application processing.
Limited information on processing times for certain groups of international students—processing times for change of status applications in Canada from temporary resident to international student were not available because this data is not captured in FOSS (i.e. when a visitor or worker who is already in Canada transitions to student status while still in the country). As a result, for inland processing, the evaluation was only able to obtain processing times for extensions of student status applications.
Reliability concerns for certain administrative data fields—issues surrounding the consistency of coding of Level of Study and Country of Last Permanent Residence variables were noted by the department in 2014. As a result, data under these variables are preliminary estimates and are currently being examined as part of the overall temporary resident data quality assurance exercise. While data on Field of Study is captured on the application form (applicants are asked to pick from a drop down box containing 13 options), data was not available for the evaluation as this field is not mandatory and there are currently no business rules established on grouping the options for reporting purposes.
Unavailability of data on knowledge of official languages—Until June 2014, knowledge of official languages was not captured on the study permit application form.
Page details
- Date modified: