Evaluation of the International Student Program

Appendix D: Evaluation Framework

Program Profile

Evaluation Issue:

The program profile will provide a description of the clients who use the International Student Program and its associated work programs

Indicators:
  • Profile of international students arriving (gender, age, mother tongue, official language ability, level of study, length of study, country of origin, regional distribution by province and relation to Census Metropolitan Area, urban and other)
  • Profile of international students remaining in Canada (stock)
  • Postgrad work program user demographic overview
  • Off campus work program user demographic overview
  • Coop work program user demographic overview
  • Differences in the profile of international students using the work programs and those not using them
Methodology:
  • Analysis of Administrative Data (RDM stats)

Relevance

Evaluation Issue: 1.

To what extent is there a continued need for the International Student Program?

Indicators:
  • Documented rationale for the need for a program for international students
  • Trends in the number and profile of applications
  • Trends in the number and profile of admissions
  • Trends in the number of study permit renewals
  • Number of re-entries
  • Perceptions of stakeholders regarding continued need
Methodology:
  • Document review (TB submissions, legislative and policy documents, operational manuals and bulletins)
  • Analysis of Administrative Data (GCMS)
  • Interviews: CIC (IB, OMC, IR, Regions); P/Ts; OGDs (DFATD); NGOs (CBIE, AUCC, ACCC)

Evaluation Issue: 2.

To what extent is the International Student Program aligned with departmental and government-wide priorities?

Indicators:
  • Alignment between ISP policies/directives and departmental and government wide priorities
  • Perceptions of stakeholders regarding alignment between ISP and departmental and government wide priorities
Methodology:
  • Document review (TB submissions, RPP, Speech from the Throne, Budget, agreements, legislative and policy documents)
  • Interviews: CIC (IB, OMC, IR); OGDs (DFATD)

Evaluation Issue: 3.

To what extent is the International Student Program aligned with federal roles and responsibilities?

Indicators:
  • Alignment with legislative and federal obligations
  • Perceptions of key stakeholders regarding the alignment and appropriateness of the federal role
Methodology:
  • Document review (legislative, program and policy documents, agreements,)
  • Interviews: CIC (IB, OMC, IR); P/Ts

Program Performance

Evaluation Issue: 4.

To what extent has CIC addressed the program issues identified in the previous ISP evaluation related to: Program integrity? Application processing?

Indicators:
Program Integrity:
  • Measures planned/underway to improve program integrity based on findings from the previous evaluation
  • Evidence of issues not yet addressed or resolved
  • Policy/program improvements resulting from program integrity issues being addressed
Application processing:
  • Measures planned/underway to improve timeliness and efficiency of application processing based on findings from the previous evaluation
  • Evidence of issues not yet addressed or resolved
  • Policy/program improvements resulting from application processing issues being addressed
  • Perceptions of ISP representatives regarding extent to which identified issues have been addressed
Methodology:
  • Document review (Management Response Action Plan, audit/policy documents, updates to operational manuals and bulletins, Student Partners Program reports)
  • Interviews: CIC (IB, OMC, IR); OGDs (DFATD)

Evaluation Issue: 5.

To what extent is stakeholder engagement related to the ISP effective?

Indicators:
  • Mechanisms for stakeholder (internal and external) engagement are in place and function well
  • Perceptions of stakeholders regarding the effectiveness of engagement
Methodology:
  • Document review (ISP working group and committee terms of reference and minutes)
  • Interviews: CIC (IB, OMC, IR, Regions); P/Ts, OGDs (DFATD, CBSA, IC, HRSDC); NGOs (CBIE, AUCC, ACCC)

Evaluation Issue: 6.

To what extent do ISP policies and programs facilitate study and work opportunities for international students? Do they facilitate transition to permanent residency?

Indicators:
  • Number and types of study/work streams available for temporary residents
  • Evidence of barriers to access work programs for particular student groups
  • Evidence of unintended impacts of ISP policies and programs
  • Perceptions of adequacy of work/study opportunities available to Foreign Nationals
  • Perceptions of unintended impacts of ISP policies and programs
  • Number of former international students transitioning to permanent residency through CEC, PNP, FSW (for those with and without a work permit)
Methodology:
  • Document review (policy documents, operational manuals and bulletins, eligibility criteria)
  • Analysis of Administrative Data (GCMS)
  • Interviews: CIC (IB, OMC, IR Regions); P/Ts; OGDs (DFATD, CBSA, HRSDC); NGOs (CBIE, AUCC, ACCC)

Evaluation Issue: 7.

To what extent have selection decisions been timely and consistent?

Indicators:
Timely:
  • Average ISP application processing times by application method, and comparison of processing time over period of previous evaluation with current evaluation and compared to other countries (e.g. Australia, United Kingdom, New Zealand, United States, France)
  • Proportion of applications finalized within the established service standards by application method (eapps, VACs, dropbox)
  • Improvements to timeliness resulting from implementation of eapps for ISP applications
Consistent:
  • Mechanisms and tools are in place that promote consistent decision making (consistent training for visa officers, quality assurance exercises, coordination within CIC)
  • Comparison of student application acceptance, withdrawal and refusal rates overall and by CVOA (over time and reasons if available)
  • Perceptions of CIC staff related to timeliness and consistency
Methodology:
  • Document review (training material, training schedule and syllabi, QA templates, operational manuals and bulletins)
  • Analysis of Administrative Data (GCMS, CPC & CVOA stats)
  • Interviews: CIC (IB, Regions, IR including CVOA, CPR); P/Ts; OGDs (DFATD)

Evaluation Issue: 8.

To what extent do selection processes support program integrity?

Indicators:
  • Evidence of mechanisms/procedures for fraud verification (monitoring practices, data capture, strategic analysis, reports, antifraud bulletins/guidance to the field, coordination and information sharing within CIC and with CBSA, etc.)
  • Trends in nature and extent of misuse & fraud over time
  • # of cases referred to CBSA for investigation, # of investigations & # of removal orders & removals
  • Trends in refusals #s and reasons for refusal - over time
  • Number and % of foreign nationals obtaining off campus work permits that do not continue to meet the necessary conditions
  • Perceptions of adequacy of measures/procedures in place to identify fraud/misuse, including identification of gaps
Methodology:
  • Document review (reports on fraud/program integrity)
  • Analysis of Administrative Data (GCMS, CPC & CVOA stats)
  • Interviews: CIC (IB, Regions, IR including CVOAs, CPR); OGDs (CBSA, RCMP)

Page details

Date modified: