2. Methodology

2.1. Evaluation issues and questions

The evaluation was conducted in accordance with the requirements of the Directive of the Evaluation Function (Treasury Board of Canada Secretariat, 2009) and examined relevance and performance. The evaluation questions, organized by core issue, are presented in Table 2.1 (see Appendix A for the full set of evaluation questions, indicators, and methodologies). The logic model for the regulation of immigration consultants can be found in the Appendix B.

Table 2.1: Evaluation Questions: Regulation of Immigration Consultants

Item Core issues Evaluation questions Section of the report
Relevance Continued Need for the Program (assessment of the extent to which the program continues to address a demonstrable need and is responsive to the needs of Canadians) 1.1 What need is the regulation of immigration consultants aiming to address? 3.1.1
Alignment with Government Priorities (assessment of the linkages between program objectives and (i) federal government priorities and (ii) departmental strategic outcomes) 1.2 Is the regulation of immigration consultants aligned with federal roles and responsibilities and GoC and CIC priorities? 3.1.2
Alignment with Federal Roles and Responsibilities (assessment of the role and responsibilities of the federal government in delivering the program)
Performance Achievement of Expected Outcomes (assessment of progress toward expected outcomes (including immediate, intermediate and ultimate outcomes) with reference to performance targets, program reach, program design, including the linkage and contribution of outputs to outcomes) 2.1 Are the appropriate governance and management processes in place to achieve program outcomes? 3.2.1
2.2 Is ICCRC a viable, transparent, accountable, and well-managed organization? (Capacity) 3.2.2
2.3 Has the ICCRC adequately informed stakeholder groups on the immigration consulting sector? (Communication) 3.2.3
2.4 Are members receiving accreditation from the ICCRC? Are they receiving professional development opportunities to develop competencies and improve qualifications? (Competencies) 3.2.4
2.5 To what extent is there a fair, transparent and accessible complaint and discipline mechanism in place to regulate member conduct? (Compliance) 3.2.5
Demonstration of Efficiency and Economy (assessment of resource utilization in relation to the production of outputs and progress toward expected outcomes) 3.1 Have program resources been used appropriately to achieve program outcomes? 3.2.6

2.2. Evaluation scope

The evaluation approach and methodology were set out in an evaluation plan developed prior to the commencement of the evaluation. This planning phase was undertaken from October to January, 2012-13, and was conducted in consultation with all CIC Branches involved in the regulation of immigration consultants, as well with the ICCRC. The evaluation is aligned with the risk-based Departmental Evaluation Plan, which was developed in part based on assurance mapping that provides senior management with information on the ways in which the department ensures appropriate oversight of programs, based on level of risk. The level of effort associated with the evaluation was calibrated in order to ensure that the department meets its coverage requirements. Footnote 11

Recognizing that the ICCRC was recently established, the evaluation was designed as an implementation evaluation and examined the extent to which the ICCRC was able to accomplish what was outlined in the contribution agreement in the areas of capacity building, communications, competencies, and compliance. Given the timing of the evaluation, only the expected immediate outcomes were examined:

  • ICCRC is a viable, transparent, accountable, and well-managed organization;
  • Stakeholder groups have information on the immigration consulting sector, including accreditation, potential of fraud, and recourse mechanisms;
  • Members receive accreditation and professional development opportunities to continually develop their competencies and qualifications; and
  • Fair, transparent and accessible complaint and discipline mechanisms are established.

The evaluation examined activities of the ICCRC since it was established as the official regulatory body for the immigration consultant industry in June 2011 up to December 2012. However, as the contribution agreement was signed before the ICCRC became operational (in March 2011) the evaluation also included activities that took place between when the Agreement was signed and June 2011. The activities of the past regulator, CISC, were not included in the scope of this evaluation. In addition, as the evaluation focused on the activities of the ICCRC, the evaluation did not explore in depth the issues around unauthorized representatives. Footnote 12

2.3. Data collection methods

The evaluation of the ICCRC included the use of multiple lines of evidence and complementary research methods to help ensure the strength of information and data collected. Data collection for this evaluation took place between January and April, 2013, with an updated assessment on the financial data completed in February 2014. Each of the lines of evidence drawn upon is described in greater detail below.

2.3.1. Document review

A review of relevant program documents was conducted to provide background and context, informing the assessment of the relevance and performance of the ICCRC. Official government documents, such as Speeches from the Throne, Budget Speeches, and policy and strategic documents were reviewed for contextual background information and for information on CIC and Government of Canada priorities. Previous government reports on immigration consulting were reviewed to provide information on the industry and its need for regulation. Departmental reference documents, including contribution agreement documents and processing manuals, were used to address specific evaluation questions. See the Technical Appendices for a list of documents reviewed.

2.3.2. Administrative data analysis

Administrative data from the ICCRC were used to develop a profile of ICCRC members and also to assess the degree of progress made by the ICCRC towards the activities and outputs established in the contribution agreement. The Grants and Contributions Financial Management (GCFM) division of CIC’s Financial Management Branch also provided an analysis of the financial position of the ICCRC, which was used to inform the assessment of the financial viability of the organization, its resource utilization and the overall ability, based on the information presented at the time, of the ICCRC to adhere to the negotiated repayment schedule.

CIC administrative data from the Global Case Management System (GCMS) were used to determine the extent to which immigrants to Canada employed the services of immigration consultants. Other CIC administrative data from OMC provided information on the number and type of complaints regarding immigration consultants received by the Department. CBSA administrative data were also analyzed to examine the volume of cases referred to it by the ICCRC or CIC, as well as the outcomes of these referrals.

2.3.3. Interviews

A total of 38 interviews were completed for the evaluation (Table 2.2). Interviews were undertaken with four key stakeholder groups (see the Technical Appendices for the interview guides). This included CIC senior management and program officers involved with the ICCRC; representatives of the ICCRC (including the past and present CEOs, ICCRC staff, and members of the Board of Directors); representatives from other government departments (OGDs) involved with the regulation of immigration consultants (i.e., the Canada Border Services Agency, Employment and Social Development Canada and the Immigration and Refugee Board of Canada); and other stakeholders in the immigration representation business (e.g., immigration lobby groups).

Table 2.2. Summary of interviews completed

Interview group Number of Interviews
CIC representatives 15
ICCRC staff 17
Representatives from other government departments 4
Other stakeholders 2
Total 38

Interviews were conducted both in-person and by telephone. Where qualitative information is presented in the report, the scale shown in Table 2.3 was used. Footnote 13

Table 2.3. Interview data analysis scale

Descriptor Meaning
All Findings reflect the views and opinions of 100% of the interviewees
Majority / most Findings reflect the views and opinions of at least 75% but less than 100% of interviewees
Many Findings reflect the views and opinions of at least 50% but less than 75% of interviewees
Some Findings reflect the views and opinions of at least 25% but less than 50% of interviewees
A few Findings reflect the views and opinions of at least two respondents but less than 25% of interviewees

A site visit was conducted in March 2013 to the ICCRC head office in Burlington, Ontario. The site visit provided an opportunity for evaluation team members to interview ICCRC staff in-person, observe the facilities obtained by the ICCRC through the contribution agreement, and better understand the ICCRC’s operating environment.

In addition to the 15 interviews that were conducted with CIC representatives, the evaluation also asked questions regarding the validation of authorized representatives and mechanisms to file complaints in five CIC offices abroad. These questions were asked of CIC officers in New Delhi, Beijing, Hong Kong, London, and Paris during site visits conducted in support of other evaluation studies. This information did not constitute a distinct line of evidence, but provided further context for the evaluation findings.

2.3.4. ICCRC member survey

To obtain the views of the ICCRC members, an online survey was developed by CIC (in consultation with the ICCRC). The survey included questions about members’ experiences related to becoming ICCRC members, the quality of information received from the ICCRC, and the ICCRC complaints and discipline process, as well as their views on the regulation of the immigration consulting profession, and questions on demographic characteristics (see the Technical Appendices for the survey instrument).

Invitations to complete this survey were distributed to members through the ICCRC’s membership list and it was pretested before being administered to the full membership. A total of 2,436 members Footnote 14 were asked to respond to the survey, 1,263 of whom completed the survey between March and April 2013 (Table 2.4).

Table 2.4. ICCRC member survey response rate

Response type Number %
Completed survey 1,263 51.8%
Started survey but did not complete 179 7.3%
Did not respond 994 40.8%
Total 2,436 100.0%

The survey achieved a margin of error of ±1.91% using a 95% confidence level. Footnote 15

2.4. Limitations and considerations

The evaluation contained a balance of qualitative and quantitative lines of evidence, allowing for the triangulation of research findings. While mitigated by the use of multiple lines of evidence, some limitations to the evaluation’s methodologies should be noted:

  • There was limited administrative data available to analyse and determine trends related to the use of immigration consultants. Prior to the introduction of GCMS, CIC systems did not keep track of which regulatory body the representative was a member of. The first full year for which this data was available is 2012.
  • As the ICCRC was set up as an arms-length body, interviewees outside of the organization were limited in their knowledge of ICCRC operations. The lack of external perspectives from interviews was augmented by the use of information from the administrative data review, as well as the survey of ICCRC members.
  • Interviews with CIC staff in missions was not built into the evaluation as a distinct line of evidence. However, it was later determined that there were opportunities to enhance understanding of the processes in place to validate the use of representatives and to file complaints by gathering information from certain missions as part of case studies being conducted for other evaluations. To strengthen the findings with respect to these processes, representatives from CIC’s Centralized Processing Region (CPR) were also interviewed.
  • There were some differences between survey respondents and the full population of ICCRC members, including a slight under-representation of ICCRC members who were previous CSIC members and a slight over-representation of members with less experience in immigration consulting. The differences in survey results between the two groups were found to be minor, therefore, there is a high degree of confidence in the results obtained.
Report a problem or mistake on this page
Please select all that apply:

Thank you for your help!

You will not receive a reply. For enquiries, contact us.

Date modified: