Evaluation of Ministerial Instructions (Implementation)

Appendix A: Ministerial instructions logic model and original evaluation matrix

Question Indicator Source
Relevance
1. Is there a continued need to issue Ministerial Instructions (MI)?

1.1 Key stakeholders identify continuing need for the MI

1.2 Stakeholders feedback regarding need to manage application intake

1.3 Consideration and analysis of alternative methods to manage intake (pass mark, etc.)

  • Key-informant interviews (Immigration Branch, OMC, International Region, CPR, Legal Services, CVOA)
  • Document review (2008 Budget, OAG report, Action Plan, Briefing material, “Pressure notes”)
  • CIO site visit
Design and implementation
2. Is the design of the MI flexible and responsive?

2.1 Stakeholders’ perception of the ability to make adjustments to the system (i.e. controlling the FSW intake with the first set of MI)

2.2 Evidence of regular monitoring and related adjustment as required

  • Key-informant interviews (Immigration Branch, OMC, International Region, CPR, Legal Services, CVOA)
  • Document review
  • Data Analysis (Monthly monitoring reports)
  • CIO site visit
3. Did stakeholders and prospective immigrants understand the first set of MI criteria once issued?

3.1 Level of stakeholders and prospective immigrant awareness of the first set of MI

3.2 Stakeholder perception and understanding of MI objectives

  • Key informant interviews (Communications, Call Centres, Immigration Branch, International Region, OMC, CPR, CIO, CVOA)
  • Document review
  • Media analysis/reports
  • CIO site visit
4. Does program delivery, under the first set of MI, facilitate the timely and efficient processing of prospective skilled worker immigrants?

4.1 Completeness and accuracy of information provided by clients (application)

  • % of incomplete applications (disaggregated by cost-recovery, etc.)
  • Number of applications intake (compared to previous years)
  • % of non eligible applications
  • Number of positive referrals to CVOA that resulted in negative decisions

4.2 Application Intake (CIO)

  • Lapsed time for completeness check
  • Time for assessment against MI
  • Lapsed time between application received and referral to CVOA
  • Application volumes/inventory at CIO

4.3 Processing and Final assessment (CVOA)

  • Wait time between each step application assessment
  • Time between application received at the CIO and final decision at CVOA
  • Time between application received at the CVOA and final decision/visa issuance
  • Wastage rates and related trends
  • Application volumes/ inventory in Missions
  • Consistency of CIO and CVOA decisions (applications sent to CVOA where the decisions were reversed)

4.4 Trends in consistency, transparency and timeliness

  • Trends in processing times
  • Number of complaints and legal challenges
  • Perceptions of consistency, transparency, and timeliness of processing
  • Data analysis (CAIPS, FOSSCIO, OMC Stats)
  • CIO site visit
  • Document Review
  • Key-informant interviews (Communications, Call Centres, Immigration Branch, International Region, OMC, CPR, CIO, CVOA)
  • CVOA survey
5. To what extent has the first set of MI reduced the intake and contributed to the reduction of the backlog of FSW applications?

5.1 Number of applications in the MI inventory (CIO and CVOA)

5.2 % reduction of pre-MI backlog

5.3 Evidence of whether the backlog reduction target was met or not

5.4 Stakeholders’ perception of the effectiveness of MI in reducing the backlog

  • Data analysis
  • Document Review
  • Key-informant interviews (CVOA, CIC, CPR, CIO, OMC, Immigration Branch, International Region)
  • CIO site visit
  • CVOA survey
Performance and Unexpected Outcomes
6. Is the processing of FSW applications through CIO more cost-effective than the previous approach?

6.1 Costs to process MI / FSW application (pre/post MI)

6.2 Stakeholders view on possible improvements to the application process

6.3 Evidence of processing efficiency for FSW applications with the CIO compared to previous approach (benefits re: level of effort/cost/fees/transparency)

6.4 Evidence of rationale for CIO (evidence-based decision making)

  • Data analysis (Cost-Management Model)
  • Key-informant interviews (Immigration Branch, OMC, CPR, Finance, CVOA)
  • Document Review (Financial data, CIO documents, budget)
  • CIO site visit
  • CVOA survey
7. Have there been any unanticipated outcomes?

7.1 Evidence of outcomes not accounted for in original plan and design (positive or negative)

7.2 Stakeholders identify potential processing challenges

  • Key-informant interviews (Immigration Branch, International Region, CPR, CIO, OMC, CVOA)
  • Document Review
  • CIO site visit
  • CVOA survey
Report a problem or mistake on this page
Please select all that apply:

Thank you for your help!

You will not receive a reply. For enquiries, contact us.

Date modified: