Evaluation of Ministerial Instructions (Implementation)

Appendix F: Interview guides and survey questionnaires

Interview guide – CIC NHQ

As you may be aware, Power Analysis has been engaged by the Evaluation Division at Citizenship and Immigration Canada to conduct an evaluation of the first set of Ministerial Instructions (MI1).  The purpose of this evaluation is to examine the relevance of the initiative, program design and delivery, program outcomes, and cost-effectiveness.

As part of the evaluation, Power Analysis is conducting interviews with key stakeholders involved in MI. The goal of the interviews is to gain a better understanding of MI, examine how the initiative was designed, coordinated, managed and implemented, collect information to assess performance in achieving results in terms of effectiveness, and identify possible improvements.

The following questions will serve as a guide for the interview. Note that the evaluation focuses on the first set of Ministerial instructions, and therefore we ask you to refer to MI1 when answering the questions. However, if there are differences between MI1 and MI2 that you believe would benefit the discussion, feel free to mention them. In some cases, questions will not be relevant to your particular situation; the interviewers will focus on those that are.

The responses you provide are confidential and will not be attributed to you in the evaluation report (only aggregate information will be released) or in any documentation.

Background

1.   Can you briefly describe your [unit/division/organization]’s role and involvement with the Ministerial Instructions? How long have you been involved with MI?

Program relevance

2.   What was the primary rationale for issuing MI1? Does this rationale continue to be relevant?

3.   Why is it important to manage application intake?

4.   How well did MI1 align with CIC priorities? With Government of Canada priorities?

Program design

5.   What other options were considered for dealing with the large backlogs (e.g., changing pass mark) and why was MI1 determined to be the best option? (Probe: The Auditor General asserted that the inventory reduction strategy was not based on sufficient analysis. How did CIC respond to this?)

6.   What was the primary objective of MI1?  What were the strengths of MI1 in terms of its ability to meet its objectives?  What were the weaknesses? 

Program management and delivery

7.   How does CIC monitor the initiative? Is the monitoring system adequate? Any suggested improvements? How were monitoring data used to make relevant adjustments to MI1? What were the major changes to MI1 brought about via program monitoring?

8.   Could you describe how information regarding the program is communicated between CIC and CIO?  Between CIO and CVOAs? How well does the communication system work?

9.   How were the MI1 criteria communicated to potential immigrants to Canada? To other stakeholders? What evidence is there that stakeholders and prospective immigrants understood MI1 criteria?

Results

10.   On an A to F scale, please rate how well MI1 did in controlling the FSW intake. (A is excellent; B is good; C is average; D is below average; F is poor)

  A
(excellent)
B
(good)
C
(average)
D (below average) F
(Poor)
  rating checkbox 1 rating checkbox 2 rating checkbox 3 rating checkbox 4 rating checkbox 5
Please explain your rating.          

11.   On an A to F scale, please rate how well MI1 did in reducing the backlog of applications.

  A
(excellent)
B
(good)
C
(average)
D (below average) F
(Poor)
  rating checkbox 1 rating checkbox 2 rating checkbox 3 rating checkbox 4 rating checkbox 5
Please explain your rating.          

12.   On an A to F scale, please rate how transparent the decisions were under MI1.

  A
(excellent)
B
(good)
C
(average)
D (below average) F
(Poor)
  rating checkbox 1 rating checkbox 2 rating checkbox 3 rating checkbox 4 rating checkbox 5
Please explain your rating.          

13.   On an A to F scale, please rate the consistency of decisions from one application to the other under MI1.

  A
(excellent)
B
(good)
C
(average)
D (below average) F
(Poor)
  rating checkbox 1 rating checkbox 2 rating checkbox 3 rating checkbox 4 rating checkbox 5
Please explain your rating.          

14.   On an A to F scale, please rate the consistency of decisions between CIO and CVOA under MI1.

  A
(excellent)
B
(good)
C
(average)
D (below average) F
(Poor)
  rating checkbox 1 rating checkbox 2 rating checkbox 3 rating checkbox 4 rating checkbox 5
Please explain your rating.          

15.   On an A to F scale, please rate the flexibility of the design of MI.

  A
(excellent)
B
(good)
C
(average)
D (below average) F
(Poor)
  rating checkbox 1 rating checkbox 2 rating checkbox 3 rating checkbox 4 rating checkbox 5
Please explain your rating.          

16.   What have been the pros and cons of the second set of MI (MI2) as compared to MI1?

  A
(excellent)
B
(good)
C
(average)
D (below average) F
(Poor)
  rating checkbox 1 rating checkbox 2 rating checkbox 3 rating checkbox 4 rating checkbox 5
Please explain your rating.          

17.   Were there any unanticipated outcomes with respect to MI1? MI2? If so, please discuss.

Cost-effectiveness/alternatives

18.   Was the amount of funding allocated to MI1 appropriate to achieve the objectives?  For MI2? In your opinion, are there more cost-effective ways of achieving the program’s objectives?  If so, how?

19.   What improvements do you suggest to the FSW application process under MI?

20.   If you could change anything about MI what would it be?

Thank you for your time and cooperation.


Interview guide – CVOA

As you may be aware, Power Analysis has been engaged by the Evaluation Division at Citizenship and Immigration Canada to conduct an evaluation of the Ministerial Instructions.  The purpose of this evaluation is to examine the relevance of the initiative, program design and delivery, program outcomes, and cost-effectiveness.

As part of the evaluation, Power Analysis is conducting interviews with key stakeholders involved in MI. The goal of the interviews is to gain a better understanding of MI, examine how the initiative was designed, coordinated, managed and implemented, collect information to assess performance in achieving results in terms of effectiveness, and identify possible improvements.

The following questions will serve as a guide for the interview. Note that the evaluation focuses on the first set of Ministerial instructions (MI1), and therefore we ask you to refer to MI1 when answering the questions. However, if there are differences between MI1 and MI2 that you believe would benefit the discussion, feel free to mention them. In some cases, questions will not be relevant to your particular situation; the interviewers will focus on those that are.

The responses you provide are confidential and will not be attributed to you in the evaluation report (only aggregate information will be released) or in any documentation.

Background

1.   Can you briefly describe your office’s role and involvement with the Ministerial Instructions? How long have you been involved with MI?

Program relevance

2.   What was the primary rationale for issuing MI1? Does this rationale continue to be relevant?

3.   Why is it important to manage application intake?

4.   How well did MI1 align with CIC priorities?  With Government of Canada priorities?

Program design

5.   Are there better options for dealing with the large backlogs of FSW applicants (e.g., changing pass mark)? If so, please discuss.

6.   Are the MI sufficiently flexible? What were the main strengths of MI1?  What were the main weaknesses?

Program management and delivery

7.   Could you describe how information regarding the program is communicated between CIO and your office? How well does the communication system work?

8.   How were the MI1 criteria communicated to potential immigrants to Canada? What evidence is there that prospective immigrants understood MI1 criteria?

9.   Was the distribution of roles and responsibilities between CIO and CVOAs appropriate under MI1? Under MI2? What can be improved?

Results

10.   How well did MI1 do in controlling the FSW intake? In reducing the backlog of applications?

11.   In your opinion are the CIO decisions transparent and consistent from one application to the next?

12.   Please comment on the consistency of CIO and CVOA decisions (applications sent to CVOA where the decisions were reversed) under MI1. How has this changed under MI2?

13.   What have been the pros and cons of MI2 as compared to MI1?

14.   Were there any unanticipated outcomes with respect to MI1? MI2? If so, please discuss.

Cost-effectiveness/alternatives

15.   In your opinion, are there more cost-effective ways of achieving the program’s objectives?  If so, how?

16.   What improvements do you suggest to the FSW application process under MI?

17.   If you could change anything about MI what would it be?

Thank you for your time and cooperation.


Interview guide – Provincial government officials

Power Analysis Inc. has been engaged by the Evaluation Division at Citizenship and Immigration Canada to conduct an evaluation of the Ministerial Instructions.  The purpose of this evaluation is to examine the relevance of the initiative, program design and delivery, program outcomes, and cost-effectiveness.

As part of the evaluation, Power Analysis is conducting interviews with key stakeholders involved in MI. The goal of the interviews is to gain a better understanding of MI, examine how the initiative was designed, coordinated, managed and implemented, collect information to assess performance in achieving results in terms of effectiveness, and identify possible improvements.

The following questions will serve as a guide for the interview. Note that the evaluation focuses on the first set of Ministerial instructions (MI1), and therefore we ask you to refer to MI1 when answering the questions. However, if there are differences between MI1 and MI2 that you believe would benefit the discussion, feel free to mention them. In some cases, questions will not be relevant to your particular situation; the interviewers will focus on those that are.

The responses you provide are confidential and will not be attributed to you in the evaluation report (only aggregate information will be released) or in any documentation.

Background

1.   Can you briefly describe your [department/ministry]’s involvement with the Ministerial Instructions? How long have you been involved with MI? How familiar would you say you are with MI1? (Note to interviewer: if the informant is not particularly conversant with MI1 mention the criteria for MI1: New FSW applications were eligible for processing if they: included an offer of arranged employment; were from a foreign national living legally in Canada for one year as a temporary foreign worker or international student; or were from a skilled worker with at least one year of experience under one or more of the 38 priority occupations identified at that time.)

2.   What was the primary rationale for issuing MI1? Does this rationale continue to be relevant?

3.   Why is it important to manage application intake?

4.   How well did MI1 align with your province’s priorities?

5.   What evidence is there that stakeholders and prospective immigrants understood MI1 criteria?

6.   On an A to F scale, please rate how well MI1 did in controlling the FSW intake.
(A is excellent; B is good; C is average; D is below average; F is poor)

  A
(excellent)
B
(good)
C
(average)
D (below average) F
(Poor)
  rating checkbox 1 rating checkbox 2 rating checkbox 3 rating checkbox 4 rating checkbox 5
Please explain your rating.          

7.   On an A to F scale, please rate how well MI1 did in reducing the backlog of applications.

  A
(excellent)
B
(good)
C
(average)
D (below average) F
(Poor)
  rating checkbox 1 rating checkbox 2 rating checkbox 3 rating checkbox 4 rating checkbox 5
Please explain your rating.          

8.   On an A to F scale, please rate how transparent the decisions were under MI1.

  A
(excellent)
B
(good)
C
(average)
D (below average) F
(Poor)
  rating checkbox 1 rating checkbox 2 rating checkbox 3 rating checkbox 4 rating checkbox 5
Please explain your rating.          

9.   On an A to F scale, please rate the consistency of decisions from one application to the other under MI1.

  A
(excellent)
B
(good)
C
(average)
D (below average) F
(Poor)
  rating checkbox 1 rating checkbox 2 rating checkbox 3 rating checkbox 4 rating checkbox 5
Please explain your rating.          

10.   On an A to F scale, please rate the flexibility of the design of MI.

  A
(excellent)
B
(good)
C
(average)
D (below average) F
(Poor)
  rating checkbox 1 rating checkbox 2 rating checkbox 3 rating checkbox 4 rating checkbox 5
Please explain your rating.          

11.   Are there better options for dealing with the large backlogs of FSW applicants (e.g., changing pass mark)? If so, please discuss.

12.   What were the main strengths of MI1?  What were the main weaknesses? 

13.   Were there any unanticipated outcomes with respect to MI1? MI2? If so, please discuss.

14.   What improvements do you suggest to the FSW application process under MI?

15.   If you could change anything about MI what would it be?

Thank you for your time and cooperation.


CVOA Questionnaire

About this survey

Citizenship and Immigration Canada (CIC) is conducting an evaluation of the Ministerial Instructions.  This research will provide essential information related to relevance, program design and delivery, program outcomes, and cost-effectiveness.

Central to the study is a survey of Canadian Visa Offices Abroad.  We ask for one consolidated response per CVOA.  Please feel free to consult with other visa officers in the office who deal with MI-FSW applications, but throughout the survey, we ask you to answer for your CVOA as a whole. The terminology will be using ‘you’ to refer to your CVOA.

The survey asks for your perspective on various aspects of Ministerial Instructions. Most questions refer to the first set of (MI1). MI1, published November 28, 2008, introduced eligibility criteria for all new Federal Skilled Worker (FSW) applications submitted on or after February 27, 2008.Footnote 37 According to this set of Instructions, new FSW applications would be eligible for processing if they:

  • included an offer of arranged employment,
  • were from a foreign national living legally in Canada for one year as a temporary foreign worker or international student; or
  • were from a skilled worker with at least one year of experience under one or more of the 38 priority occupations identified at that time.

Please complete the survey by June 22, 2011. Please keep a completed copy on your computer in case the survey does not get to us and we need to ask you to resend it.

The questions are easy to answer. We ask you to:

Click on the box next to your answer                    rating checkbox YES     rating checkbox NO

or Enter a number in the appropriate space           ____ Years

or Provide a brief written answer                         _________

Note: to change your selection, simply click on the box again to delete the x.

Thank you for your help.

______________________________

1.   In what CVOA do you work?

Ministerial Instructions in General

2.   The original rationale for Ministerial Instructions was to address a large backlog in applications for the Federal Skilled Worker program. To what extent does this rationale continue to be relevant? (Check one box on the 5-point scale

To a great extent To some extent Not at all Unsure
rating checkbox 1 rating checkbox 2 rating checkbox 3 rating checkbox 4 rating checkbox 5 rating checkbox 9

3.   How would you rate the flexibility the Ministerial Instructions give to the Minister in terms of supporting the attainment of the immigration goals established by the government?

rating checkbox 1     A (EXCELLENT)
rating checkbox 2     B (GOOD)
rating checkbox 3     C (AVERAGE)
rating checkbox 4     D (BELOW AVERAGE)
rating checkbox 5     F (POOR)
rating checkbox 9     UNSURE

First set of Ministerial Instructions

4.   To what extent did prospective immigrants to Canada understand the criteria of MI1? (Check one box on the 5-point scale)

To a great extent To some extent Not at all Unsure
rating checkbox 1 rating checkbox 2 rating checkbox 3 rating checkbox 4 rating checkbox 5 rating checkbox 9

5.   Overall, how would you rate MI1?

rating checkbox 1     A (EXCELLENT)
rating checkbox 2     B (GOOD)
rating checkbox 3     C (AVERAGE)
rating checkbox 4     D (BELOW AVERAGE)
rating checkbox 5     F (POOR)
rating checkbox 9     UNSURE
 
Comment?

6.   How would you rate the quality of each of the following elements of MI1? (Check one box in each row) For any element you rated D or F, please explain why in the box below the rating.

Element of MI1 A
(excellent)
B
(good)
C
(average)
D (below average) F
(Poor)
Communications between CIO and your office rating checkbox 1 rating checkbox 2 rating checkbox 3 rating checkbox 4 rating checkbox 5
Reason for D or F.          
Clarity of the requirements for application rating checkbox 1 rating checkbox 2 rating checkbox 3 rating checkbox 4 rating checkbox 5
Reason for D or F.          
The timeliness of the response to applications rating checkbox 1 rating checkbox 2 rating checkbox 3 rating checkbox 4 rating checkbox 5
Reason for D or F.          
Transparency of CIO eligibility decisions rating checkbox 1 rating checkbox 2 rating checkbox 3 rating checkbox 4 rating checkbox 5
Reason for D or F.          
Transparency of CVOA eligibility decisions with respect to MI1 rating checkbox 1 rating checkbox 2 rating checkbox 3 rating checkbox 4 rating checkbox 5
Reason for D or F.          
Consistency of CIO eligibility decisions from one application to the next rating checkbox 1 rating checkbox 2 rating checkbox 3 rating checkbox 4 rating checkbox 5
Reason for D or F.          
Consistency of CVOA eligibility decisions from one application to the next with respect to MI1 rating checkbox 1 rating checkbox 2 rating checkbox 3 rating checkbox 4 rating checkbox 5
Reason for D or F.          
Consistency between CIO and CVOA eligibility decisions rating checkbox 1 rating checkbox 2 rating checkbox 3 rating checkbox 4 rating checkbox 5
Reason for D or F.          
Ability to control FSW intake rating checkbox 1 rating checkbox 2 rating checkbox 3 rating checkbox 4 rating checkbox 5
Reason for D or F.          
Ability to reduce FSW backlog rating checkbox 1 rating checkbox 2 rating checkbox 3 rating checkbox 4 rating checkbox 5
Reason for D or F.          

7.   In processing FSW applications, how did the MI1 affect your workload as compared to your workload prior to the introduction of MI1? (Check one box on the 5-point scale)

Big increase in my workload No effect Big decrease in my workload
rating checkbox 1 rating checkbox 2 rating checkbox 3 rating checkbox 4 rating checkbox 5

Comment?

8.   Under MI1, approximately what proportion of CIO positive initial eligibility decisions did your office reverse, excluding applicants who failed to return the full application with the 120 day time limit?

_____%

9.   As compared to the system prior to MI1, did the number of complaints from FSW applicants under MI1

rating checkbox 1    DECLINE
rating checkbox 2    STAY ABOVE THE SAME
rating checkbox 3    RISE

10.   Did MI1 have any consequences that were unexpected?

rating checkbox 1    YES    Please specify
rating checkbox 2    NO

Second set of Ministerial Instructions

11.   Overall, how would you rate MI2?

rating checkbox 1     A (EXCELLENT)
rating checkbox 2     B (GOOD)
rating checkbox 3     C (AVERAGE)
rating checkbox 4     D (BELOW AVERAGE)       Why?
rating checkbox 5     F (POOR)       Why?
rating checkbox 9     UNSURE

12.   In processing FSW applications, how does MI2 affect your workload as compared to your workload under MI1? (Check one box on the 5-point scale)

Big increase in my workload No effect Big decrease in my workload
rating checkbox 1 rating checkbox 2 rating checkbox 3 rating checkbox 4 rating checkbox 5

Comment?

The Centralized Intake Office

13.   Overall, how would you rate the CIO decision process as it exists today?

rating checkbox 1     A (EXCELLENT)
rating checkbox 2     B (GOOD)
rating checkbox 3     C (AVERAGE)
rating checkbox 4     D (BELOW AVERAGE)       Why?
rating checkbox 5     F (POOR)       Why?
rating checkbox 9     UNSURE
 
Comment?

14.   Are there any current roles and responsibilities of the Centralized Intake Office (CIO) that should not be their responsibility? 

rating checkbox 1    YES    Please specify
rating checkbox 2    NO
rating checkbox 9    UNSURE

15.   Are there any services/activities that CIO does not currently carry out that it should be responsible for? 

rating checkbox 1    YES    Please specify
rating checkbox 2    NO
rating checkbox 9    UNSURE

Alternatives to Ministerial Instructions

16.   Are there better options than Ministerial Instructions for controlling FSW intake?

rating checkbox 1    YES    Please specify
rating checkbox 2    NO
rating checkbox 9    UNSURE

17.   Are there better options than Ministerial Instructions for dealing with the large backlogs of FSW applicants?

rating checkbox 1    YES    Please specify
rating checkbox 2    NO
rating checkbox 9    UNSURE

Suggested Improvements

18.   What improvements do you suggest to the FSW application process under MI?

19.   What improvements do you suggest for the MI in general?

Report a problem or mistake on this page
Please select all that apply:

Thank you for your help!

You will not receive a reply. For enquiries, contact us.

Date modified: