Evaluation of the Federal Business Immigration Program — 2. Methodology

2.1. Evaluation Issues and Questions

This evaluation assesses the extent to which immediate and intermediate outcomes of the program have been achieved. Evaluation questions addressed during the evaluation were developed to determine the extent to which the program is meeting its expected outcomes.

The first immediate outcome of the BIP is to have a selection process that is consistent, objective and efficient. Second, the admission of entrepreneur immigrants is expected to contribute to new business start-ups, job creation and job retention in accordance with the conditions for this economic class. Thirdly, investors' funds are expected to generate additional provincial investment in economic development.

The following intermediate outcomes are anticipated to occur in the longer-term as a result of the three above-mentioned immediate outcomes:

• Selected and admitted BI class immigrants are economically established in Canada
• Additional job opportunities for all Canadians
• Benefits of expanded economic development activities are shared across Canada

Ultimately, the BIP is intended to contribute to the migration of permanent residents that strengthens Canada's economy.

The evaluation issues and questions addressed were:

Relevance

• 1) Is there an ongoing need for the Business Immigration Program?
• 2) Is the Business Immigration Program consistent with departmental and government-wide priorities?
• 3) Is the Business Immigration Program consistent with federal roles and responsibilities?

Program performance – management outcomes

• 4) Have policy advice and directives supported effective program delivery?
• 5) Have selection decisions for the Business Immigration Program been timely, consistent and objective?
• 6) How effective was the ongoing monitoring of the Entrepreneur Class terms and conditions?

Program performance – program outcomes

• 7) Has the Entrepreneur Class contributed to the creation of new business start-ups or acquisitions and jobs creation?
• 8) To what extent are business immigrants becoming established economically?
• 9) To what extent are the economic benefits of business immigration being shared across Canada (BIP overall) and participating provinces (IN only)?
• 10) Have there been any unintended impacts associated with BIP?

Program performance – efficiency and economy

• 11) Are the selection criteria of the BIP effective in identifying applicants who will establish themselves economically?^{Footnote 10}Are there alternative selection criteria that could meet policy objectives more effectively?
• 12) How cost-effective is the current approach to selecting business immigrants?

For each of the evaluation questions, performance indicators were identified. They are presented in Appendix B, Evaluation Issues, Questions and Indicators.

2.2. Evaluation Scope

The evaluation examined the BIP program delivery from 2007 to 2011, as well as the performance of the program during this time period.

Although administrative data were available beyond 2011 after data collection for the evaluation had been completed, the last year of administrative data to be considered for the evaluation was 2011. Although the addition of data beyond 2011 was considered, the timelines for the evaluation did not allow it. Using data up to 2011 also ensures alignment with the time frame covered by other methods (such as key informant interviews, case studies and survey).

It should be noted that, under the terms of the Canada-Quebec Accord, the province of Quebec has the sole responsibility for its selection programs. Therefore, business immigrants selected by Quebec would be processed under the Quebec Business Immigration Program (QBIP). This evaluation excludes business immigrants with a Certificat de Sélection du Québec (CSQ), and only includes business immigrants admitted under the federal BIP and destined for provinces and territories outside of Quebec.

2.3. Data Collection Methods

The evaluation questions and performance indicators were addressed using multiple lines of evidence to gather qualitative and quantitative data from a wide range of perspectives including program managers, stakeholders and clients.

The multiple lines of evidence approach included six methods. The data and information gathered were incorporated into an evidence matrix that supported triangulation of information across lines of evidence, which was then used to develop the key findings, conclusions and recommendations.

A hybrid approach was employed for this evaluation, where both Citizenship and Immigration Canada's (CIC) Research and Evaluation Branch and contracted consultants jointly conducted all phases of the evaluation.

The evaluation employed the following six lines of evidence: a literature and document review, key informant interviews, case study site visits, a telephone survey, administrative data analysis, and cost-effectiveness analysis.

2.3.1. Literature and document review

A combined literature and document review was conducted to provide background and context, informing the assessment of the relevance and performance of the BIP. The review examined relevant background policy and program documents related to the BIP, as well as reports and research articles published in Canada and abroad on business immigration programs. International programs were included in the review. The bibliography of referenced documents is presented in the Technical Appendices.

2.3.2. Key informant interviews

Thirty-nine key informant interviews (KIIs) were conducted, both in-person and by telephone, to obtain the perceptions of key stakeholder groups on the relevance, design, implementation, and performance of the BIP. The table below shows the number of interviews by type; the Key Informant Interview Guide is presented in the Technical Appendices.

2.3.3. Case study site visits

Case study site visits were conducted at Canadian Visa Offices Abroad (CVOA) to interview staff with knowledge of operational and international context issues. Three case study site visits were conducted in CVOA: London, Paris, and Hong Kong. These CVOA were selected to ensure the evaluation had a good representation of the different CVOA delivering the BIP across the CIC network (i.e., offices for which BIP is representing a small and high share of their case load), while still representing a high share of the applications processed. Two of the CVOA visited were receiving significant number of BIP applications and the other one was receiving low volume of BIP applications, thus providing a balanced perspective in regards to the operational context. The Case Study Interview Guide is presented in the Technical Appendices.

Where qualitative information was presented in the report, the scale shown in the table below was used.

2.3.4. Telephone survey

A telephone survey was conducted with BIP immigrants admitted to Canada between 2007 and 2011, in order to obtain information on their application experience and their economic establishment in Canada. Two mail-outs were sent to the 12,364 of 12,402 BIP immigrants for whom complete addresses were available, in order to obtain their informed consent to participate in the telephone survey. As a result, 1042 positive consents were obtained, representing 8.4% of the population contacted. 877 immigrants were interviewed, providing a margin of error of ± 3.2% using a 95% confidence level for the survey data collected. The number surveyed corresponds to 84.2% of those who provided their written consent to the survey, and 7.1% of the total population contacted. The survey was conducted in seven languages (English, French, Mandarin, Cantonese, Farsi, Korean and Japanese). The BIP Survey Questionnaire is presented in the Technical Appendices. As further discussed in the Limitations section of this report (see Section 2.4), survey results were weighted to ensure appropriate representation of the BI population, so that survey results can be used with confidence.

2.3.5. Administrative data analysis

An administrative data analysis of existing Government of Canada (GoC) databases was completed to provide information on the application processing, characteristics of immigrants admitted, and the economic outcomes for immigrants. These databases included the Global Case Management System (GCMS), the Computer-Assisted Immigration Processing System (CAIPS), and the Field Operations Support System (FOSS), which provided information about BIP applications and admissions.

This analysis was also supported by information from the Longitudinal Immigration Database (IMDB)^{Footnote 11} and the Canadian Employer-Employee Dynamics Database (CEEDD) on economic outcomes for BIs. The CEEDD provided information on the population of businesses and business owners reporting corporate income tax, and the IMDB provided information on the population of individual tax filers. In addition to addressing the economic outcomes of BIs, the IMDB also provided information on out-migration and interprovincial mobility.

Although representing the full population of individual tax filers, not all immigrants admitted are found in the IMDB. Various reasons can account for this, such as never having filed an individual income tax report, or not being linked to the IMDB despite having filed an income tax report. Overall, most of the BIs admitted between 1995 and 2010 were captured in the IMDB. The capture rate of all immigrants admitted who file an income tax report each year since admission is similar among BIP classes. The lowest capture rates are found either for the admission year (year 0) or the last year of observation considered for a given cohort in the study, and reach a low of 58% of ENs, 48% for SEs and 52% for INs for specific cohorts. Highest capture rates are generally found between the first and third years since admission, and reach up to 85% of ENs, 82% of SEs and 86% of INs for specific cohorts (for more information on capture rates, see Appendix C).

While the primary focus of the evaluation was on the BIP from 2007 to 2011, the evaluation examined longer-term trends in the applications received through the BIP (from 1995 to 2011), as well as the mobility and economic outcomes of BIs (from 1995 to 2010).

Consequently, there are some considerations when triangulating the information provided by the IMDB, CEEDD, and the survey of business immigrants. IMDB results include multiple cohorts and taxation years (1995-2010); CEEDD results are provided for only one taxation year (2009) and for multiple cohorts (1995-2009); survey results were captured for 2007-2011 cohorts in 2013. In addition, IMDB results do not capture the activities of incorporated businesses, whereas CEEDD and survey results reflect both un-incorporated and incorporated business activities. As such, it is expected that results from the different lines of evidence will differ to some extent, as they draw from different cohorts, taxation years and capture different concepts. Results drawn from these different data sources will need to be interpreted differently to account for methodological specificities.

2.3.6. Analysis of program costs

In order to examine the resource utilization by the BIP, program delivery costs were examined in relation to the volume of applications processed. CIC Cost Management data, including federal processing costs for BIP immigrants under both the federal and Quebec BIP, were used to that purpose. The data did not allow distinguishing between costs incurred for the federal BIP program and those incurred for the delivery of the Quebec BIP program.

2.4. Limitations and Considerations

The evaluation employed a balance of qualitative and quantitative lines of evidence, enabling the triangulation of research findings. Mitigation strategies were used to address the limitations, and along with the triangulation of multiple lines of evidence, were considered sufficient to ensure evaluation findings can be used with confidence. Yet, the following limitations and mitigating strategies of the evaluation methodologies should be considered:

• Gaps in literature and document review: Although CIC has a good collection of documentation on the BIP, there is a lack of academic literature focused on the economic activities of business immigrants. Rather, the available literature examines more generally the self-employment activities of immigrants, with no distinction made about the class under which they were admitted to Canada.

  Mitigation strategy: This deficit was addressed by gathering direct information on the economic outcomes of BIs through the telephone survey and administrative data analysis of incomes they report to the Canadian Revenue Agency on their income tax report. However, information obtained though these other lines of evidence only apply to the specific cohorts studied, and cannot be generalized to all cohorts admitted since the program was introduced. As such, mitigation strategies could not completely counter the lack of relevant academic literature on economic activities of BIs.

• Representativeness of survey respondents: The survey sample obtained through the informed consent process differed from the total BI population admitted between 2007 and 2011, with ENs being over-represented and INs being under-represented. Additionally, there was an under-representation in the survey sample of respondents from Asia.

  It is also possible that the results of telephone survey of BIs could have been positively skewed by those respondents more compliant with the terms and conditions of the EN program. This self-selection bias is however the same that would be expected in any survey undertaking.

  Mitigation strategy: The survey results were weighted to address these imbalances identified in order to ensure that they were representative of the total BIP population (for more information on weighting, see Appendix D). The sample size in the BIP survey was sufficient to report averages for BIP and its three classes for all indicators.

• Gaps in administrative data: Gaps in several administrative data sources were noted. First, because of inconsistencies in selection criteria point information in CAIPS and GCMS, the evaluation was unable to use this information to get a description of the points received by BIs on the selection grid and to explore which selection criteria would explain better economic performances.

  Mitigation strategy: The application and admission records capture similar types of information (i.e., information related to the socio-demographic characteristics of individuals) although in some instances to a lesser level of detail, which was at least sufficient to support socio-demographic analysis of the BIP cohort. In addition, the evaluation gathered qualitative data on this topic.^{Footnote 12}

  Second, the evaluation had planned to use information from the Entrepreneurship Monitoring Information System (EMIS) and the Investment Monitoring Information System (IMIS) to assess the economic activities of ENs and INs, as well as the degree of monitoring of and compliance with EN terms and conditions. Unfortunately, these CIC databases had a high rate of missing values, especially for the years under study. As a result, the evaluation was unable to draw upon these sources of information.

  Mitigation strategy: This limitation was anticipated when planning the evaluation. As such, the evaluation was designed in a way that allowed to gather the information needed through other lines of evidence, namely from the KIIs with CIC representatives and the telephone survey with BIs.

  Third, although administrative data on application, admission and tax records became available to cover additional years after data collection for the evaluation had been completed, analysis focused on the original scope period for the evaluation (2007 to 2011). This ensured alignment with the time frame covered by other methodologies (such as key informant interviews, case studies and the survey), so that the respondents across lines of evidence were reflecting on the same period of the BIP.

• Definition and interpretation on economic performance: Economic establishment and performance are concepts difficult to define. Once defined their operationalization and interpretation also poses challenges. Given the nature of the business activities that BIs are likely to undertake in Canada and of the Canadian tax system (such as deductions that can be obtained), the extent of BIs economic performance is difficult to fully assess.

  Mitigation strategy: The evaluation adopted a conservative approach in defining and measuring the economic performance of BIs. As such, results provide a minimum estimate of their economic performance in Canada. For more details on the assessment of the economic performance, refer to section 5.1.

• Gaps in financial data: In CIC's Cost Management Model (CMM), Federal and Quebec business immigrant financial data are captured in the aggregate form. This limited the Evaluation's ability to focus only on costs incurred by the federal part of the program. As such the examination of costs presented in this report includes both Quebec and Federal costs associated with the BIP. Additionally, the scope of the financial information available was limited due to the absence of CMM data for the 2008/09 fiscal year.

Despite all the limitations mentioned above, the design of the evaluation and data collection approaches were able to mitigate the impacts. Strategies were taken to ensure the evaluation presented valuable, reliable information to support strong findings. Where possible, multiple lines of evidence were used to inform the evaluation questions. Despite the different methodological approaches, the results from the different lines of evidence generally converge towards common and integrated findings. The use of different lines of evidence was also designed in a way that they would complement each other in order to fill information gaps.

The evaluation methodologies and limitations are discussed in greater detail in the Technical Appendices.

• Previous
• 1
• 2
• 3
• 4
• 5
• 6 (current)
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17
• Next