Evaluation of Pre-Arrival Settlement Services

Research and Evaluation Branch
Immigration, Refugees and Citizenship Canada
April 2018

Ci4-179/2018E-PDF
978-0-660-27129-3

Reference Number: E3-2017

On this page

• List of acronyms
• Executive summary
• Evaluation of Pre-Arrival Settlement Services—Management Response Action Plan
• 1. Introduction
  • 1.1. Purpose of the Evaluation
  • 1.2. Pre-Arrival Services Background and Context
    • 1.2.1. History and Evolution
    • 1.2.2. Types of Services
    • 1.2.3. Delivery Partners
    • 1.2.4. Location of Pre-Arrival In-Person Services
    • 1.2.5. Cost of Pre-Arrival Settlement Services
• 2. Pre-Arrival Services Profile
  • 2.1. Clients and Non-Clients
    • Non-Refugees
    • Refugees
  • 2.2. Typology of Pre-Arrival Service Provider Organizations
• 3. Methodology
  • 3.1. Evaluation Approach
  • 3.2. Evaluation Scope
  • 3.3. Data Collection Methods
    • Lines of Evidence
  • 3.4. Considerations and Limitations
• 4. Relevance
  • 4.1. Continued Need and Relevance for Pre-Arrival Services
    • Variations in the need for pre-departure orientation
• 5. Impact of Pre-Arrival Services
  • 5.1. Usefulness of Pre-Arrival Services
  • 5.2. Alignment of Pre-Arrival Services Information with Specific Client Needs
  • 5.3. Clients gain knowledge of life in Canada and the Canadian work environment
  • 5.4. Clients make informed decisions about life in Canada
  • 5.5. Creating Pathways to Settlement Services and Usage in Canada
  • 5.6. Clients Participate in the Canadian Labour Market
    • Non-Refugees
    • Refugees
• 6. Program Management
  • 6.1. Uptake of Pre-Arrival Services
  • 6.2. Promotion of Pre-Arrival Services
  • 6.3. Achievement of Targets
  • 6.4. Governance and Management
  • 6.5. Location of In-Person Pre-Arrival Settlement Services
    • Non-Refugees
    • Refugees
  • 6.6. Cost per Client for Pre-Arrival Services
  • 6.7. Efficacy of Delivery Models
    • Language of service
    • Online vs. in-person
• 7. Conclusions and Recommendations
  • Considerations for the Next Call for Proposals
  • Recommendations
• Appendix A: Detailed List of in-Person Pre-Arrival Service Locations (FY2015/16 – FY 2017/18)
• Appendix B: Detailed Profiles
• Appendix C: Detailed List of Pre-Arrival SPOs (FY2015/16 – FY 2017/18)
• Appendix D: IRCC Settlement Program Logic Model
• Appendix E: List of Evaluation Questions
• Appendix F: Key Survey Frequency Tables

List of tables and figures

• Figure 1: Evolution of Pre-Arrival Settlement Services
• Table 1: Profile of Pre-Arrival Services
• Figure 2: Location of In-Person Pre-Arrival Services
• Figure 3: SPO Expenditures (April 1, 2015 to August 31, 2017)
• Figure 4: Non-Refugee Clients – Time from using Pre-Arrival Service to Admission
• Table 2: Share of non-refugees indicating the information was helpful
• Table 3: Share of non-refugees indicating the information was helpful to prepare to look for job
• Table 4: Share of refugees indicating the information session was helpful
• Table 5: Non-refugees level of information prior to coming to Canada
• Table 6: Non-refugees’ knowledge of life in Canada upon arrival
• Table 7: Non-refugees’ difficulties experienced during the first three months in Canada
• Table 8: Official Uptake of Pre-Arrival Services by Fiscal Year of Admission (April 1, 2015 to August 31, 2017)
• Table 9: iCARE-based and Adjusted Uptake of Pre-Arrival Services by Immigration Category (April 1, 2015 to August 31, 2017)
• Table 10: Expected Annual Client Target, Pre-Arrival SPOs
• Table 11: Top 10 Refugee Source Countries, Locations of In-Person Pre-Arrival Services and Estimated Uptake, April 1, 2015 – August 31, 2017
• Table 12: Overall Projected vs. Actual Cost per Client – Pre-Arrival SPOs (FY 2015/16 – FY 2017/18)
• Table 13: Projected vs. Actual Cost per Client – Pre-Arrival SPOs (FY 2015/16 – FY 2017/18)
• Table 14: Pre-Arrival SPO Performance by SPO Types
• Table B1: Profile of Pre-Arrival Clients and Non-Clients Admitted to Canada between April 1, 2015 – August 31, 2017
• Table B2: Profile of IRCC-funded Domestic Settlement Service Clients Admitted to Canada between April 1, 2015 – August 31, 2017
• Table F1: Non-Refugees - Please indicate your level of agreement with the following statements
• Table F2: Non-Refugees - Please indicate whether or not you had enough information on any of the following topics prior to coming to Canada
• Table F3: Refugee clients - Please rate your level of agreement with the following statements
• Table F4: Non-Refugees - During your first three months in Canada…

List of acronyms

ACCC

Association of Canadian Community Colleges

AEIP

Active Engagement and Integration Project

CA

Contribution Agreement

CEC

Canadian Experience Class

CC

Community Connections

CICan

Colleges and Institutes Canada

CIIP

Canadian Immigration Integration Project

COA

Canadian Orientation Abroad

ERS

Employment-Related Services

FY

Fiscal Year

FSW

Federal Skilled Worker

GAR

Government-Assisted Refugee

GCMS

Global Case Management System

GCS

Grants and Contributions System

I&O

Information and Orientation Services

iCARE

Immigration Contribution Agreement Reporting Environment

IOM

International Organization for Migration

IS

Indirect services

IRCC

Immigration, Refugees and Citizenship Canada

NARS

Needs Assessments and Referrals

PSR

Privately Sponsored Refugee

SPO

Service Provider Organization

Executive summary

Purpose of the Evaluation

This report presents the findings of the evaluation of Immigration, Refugee and Citizenship Canada’s (IRCC) Pre-Arrival Settlement Services. The evaluation was conducted to inform future program improvements and in fulfillment of requirements under the Treasury Board Policy on Results and section 42.1 of the Financial Administration Act and covered the period of fiscal (FY) years 2015/16 - 2017/18.

Overview of Pre-Arrival Settlement Services

The objective of IRCC’s pre-arrival services is to provide selected Permanent Residents,^{Footnote 1} including refugees, with accurate, relevant information and supports so that they can make informed decisions about their new life in Canada and begin the settlement process (including preparation for employment) while overseas. It is expected that these services will enable clients to be better prepared upon arrival to Canada to settle and integrate into Canadian society.

Pre-arrival services provide the same types of services as IRCC-funded in-Canada Settlement services, with the exceptions of Language Assessments and Language Training. Through contribution agreements, IRCC funds service provider organizations (SPOs) such as immigrant-serving agencies, industry/employment-specific organizations or educational institutions to provide the following types of pre-arrival services: Needs Assessments and Referrals, Information and Orientation, Employment-Related Services and Community Connections.

Pre-arrival services vary considerably in scope, delivery models and size (e.g., project funding and number of clients targeted). Some focus on providing information and supports to specific immigration categories or sub-sets of newcomers, some target specific sectors or professions while others only support newcomers destined to certain provinces. Most service providers offer services via web-based platforms while a few provide in-person services.

Summary of Conclusions and Recommendations

Pre-arrival services are needed and useful, as they help newcomers prepare for their settlement before they depart for Canada.

One of the primary expected outcomes from pre-arrival services is to provide relevant information and supports so that newcomers can make informed decisions about their new life in Canada and begin the settlement process before arriving. In this regard, all client groups found pre-arrival services to be useful and participants are gaining knowledge of life in Canada and the Canadian labour market at a higher rate than non-clients. For example, a greater share of pre-arrival clients (compared to non-clients) indicated having enough information on how to contact organizations that provide help in settling in Canada, how to have professional credentials and/or qualifications recognized, and to understand Canadian workplace culture and norms. Many pre-arrival clients are also taking actions as a result of pre-arrival services, such as looking for work, changing what they bring to Canada and deciding to further training/education to upgrade their skills. Finally, pre-arrival services have also been effective at providing referrals and linking clients with in-Canada settlement services, and these clients have been accessing IRCC-funded settlement services in Canada at a higher rate than non-users of pre-arrival services.

While clients find pre-arrival services useful, the majority of newcomers are not aware of their existence and uptake remains low for non-refugee immigrants. Ineffective promotion of these services, coupled with the absence of a comprehensive strategy to guide pre-arrival service expansion and a lack of clarity within IRCC regarding roles and responsibilities for the program delivery has resulted in a missed opportunity for the Department to positively impact more newcomers, and also in higher than expected per client costs.

Some areas for program improvements have been identified, and as such, this evaluation report proposes the following recommendations.

Framework and guidance

While there is a need to provide a mix of different services, the approach to service expansion has lacked a clearly defined strategy or framework, including a definition of how the various services were expected to be delivered, to complement one another and align with in-Canada settlement services.

Recommendation 1: IRCC should develop a comprehensive program framework and guidance for pre-arrival services that provides a clear strategic direction for program delivery.

This framework should:

• Articulate the vision for IRCC pre-arrival services, including objectives and expected results
• Consider the appropriate mix of the various delivery models and approaches
• Consider the alignment of service offerings and delivery approaches with the differing profiles and needs of various client types
• Include a strategy to identify and prioritize the optimal locations for the delivery of in-person services
• Consider the cost of services and value for money

Promotion

Despite efforts to expand the availability of pre-arrival services by increasing the number of SPOs in 2015, the absence of an effective promotion strategy from IRCC affected the reach and impact of pre-arrival services. In addition, there are opportunities to actively inform or enroll prospective clients into pre-arrival services at an earlier stage.

Recommendation 2: IRCC should develop and implement a pre-arrival services promotion strategy to significantly increase awareness and uptake.

This strategy should:

• Outline the key activities and guidance needed to improve awareness and increase program participation
• Clarify the roles and responsibilities for IRCC (including Missions abroad) and SPOs with respect to promotion
• Consider earlier opportunities for informing potential clients to help ensure they have sufficient time to access services

Governance

The rapid expansion of partners and stakeholders, coupled with a lack of clarity related to roles and responsibilities made it difficult to ensure a coherent approach to management of pre-arrival services.

Recommendation 3: IRCC should clarify and strengthen its governance to lead and coordinate pre-arrival services by:

• Establishing clear roles and responsibilities among internal IRCC stakeholders, including across NHQ, regions and International Network
• Clarifying the role of Missions in the delivery and monitoring of in-person pre-arrival services and SPOs

Strengthening the Continuum of Settlement Services

While many pre-arrival SPOs have undertaken partnerships and established networks of cross-referrals on their own, there are opportunities for IRCC to take a stronger leadership role to ensure pre-arrival SPOs are connected to one another and to IRCC’s domestic Settlement network, which would ensure more seamless pathways and efficient delivery for pre-arrival clients.

Recommendation 4: IRCC should establish a mechanism to promote collaboration, cross-referrals and sharing of best practices among pre-arrival SPOs and pre-arrival and domestic settlement SPOs.

Performance Measurement

A lack of standardized performance measurement tools and challenges in data collection have affected the ability of the Department to effectively report on pre-arrival services results in an in-depth manner and identify trends and address issues quickly, as they arise.

Recommendation 5: IRCC should strengthen performance measurement and reporting for pre-arrival services, by:

• Developing key indicators and data strategies to support collection of performance information
• Considering developing a targeted Performance Information Profile for pre-arrival services which aligns with IRCC’s Settlement Program and Resettlement Program Performance Information Profiles

Evaluation of Pre-Arrival Settlement Services—Management Response Action Plan

Recommendation #1

IRCC should develop a comprehensive program framework and guidance for pre-arrival services that provides a clear strategic direction for program delivery.

This framework should:

• Articulate the vision for IRCC pre-arrival services, including objectives and expected results
• Consider the appropriate mix of the various delivery models and approaches
• Consider the alignment of service offerings and delivery approaches with the differing profiles and needs of various client types
• Include a strategy to identify and prioritize the optimal locations for the delivery of in-person services
• Consider the cost of services and value for money

Response #1

IRCC agrees with this recommendation.

IRCC acknowledges the need for a clearly defined and coordinated strategy for pre-arrival programming, to ensure the delivery of high quality services that are client-centred and aligned with our key partners, including provinces and territories.

The Department is developing a program framework that articulates an overall vision and guidance for the delivery of pre-arrival programming, including objectives and expected outcomes. Through a more structured and streamlined service delivery approach, IRCC will ensure that clients are informed of available services and can easily access them; clients will be systematically directed to a small number of service providers for orientation, assessment and referral to tailored pre-arrival and in-Canada services, based on specific needs. Programming will focus on the coordination of services through distinct pathways for refugees, economic and family class, and Francophone immigrants. Consideration will be given to different service offerings to meet the needs of various client types.

Policy and functional guidance that is aligned with the overall vision for the pre-arrival program will be a key part of supporting the framework.

IRCC acknowledges that the rapid expansion of service providers delivering pre-arrival services incurred considerable infrastructure costs, and as such, costs to deliver services and value for money, in addition to client experience, are key considerations in the development of the program framework.

Action #1a

Develop a pre-arrival program framework that includes a vision, objectives and expected outcomes that are aligned with the Settlement and Resettlement Programs (with distinct pathways and services for refugees, economic and family class and Francophone immigrants).

Accountability: Settlement and Integration Policy Branch

Support: Settlement Network, Refugee Affairs Branch, International Network, Provinces/Territories

Completion Date: Q1 2018/19

Action #1b

Disseminate program framework to the Settlement Sector via the funding guidelines for the Spring 2018 pre-arrival program intake process.

Accountability: Settlement Network

Support: Settlement and Integration Policy Branch

Completion Date: Q1 2018/19

Action #1c

Integrate the program framework, including value for money considerations, into negotiation of contribution agreements (CAs) with service providers.

Accountability: Settlement Network

Support: Settlement and Integration Policy Branch

Completion Date: Q2 2018/19

Action #1d

Develop and issue policy guidance for IRCC staff to ensure consistent interpretation of policy requirements.

Accountability: Settlement Network

Support: Settlement and Integration Policy Branch

Completion Date: Q2 2018/19

Action #1e

Develop and issue functional guidance for service provider organizations (including guidance related to cost of services) to ensure consistent implementation of services.

Accountability: Settlement Network

Support: Settlement and Integration Policy Branch

Completion Date: Q2 2018/19

Action #1f

Complete a comparative analysis of in-person and online services to ensure optimization of the two delivery models in terms of location, value for money and client experience.

Accountability: Settlement and Integration Policy Branch

Support: Research & Evaluation, Settlement Network

Completion Date: Q4 2019/20

Recommendation #2

IRCC should develop and implement a pre-arrival services promotion strategy to significantly increase awareness and uptake.

This strategy should:

• Outline the key activities and guidance needed to improve awareness and increase program participation
• Clarify the roles and responsibilities for IRCC (including Missions abroad) and SPOs with respect to promotion
• Consider earlier opportunities for informing potential clients to help ensure they have sufficient time to access services

Response #2

IRCC agrees with this recommendation.

IRCC acknowledges the need for IRCC leadership to better connect our clients to pre-arrival settlement services prior to their arrival in Canada so that they can begin their settlement journey with realistic expectations and be better prepared.

This whole-of-Department promotion strategy will be client-focused, integrated within the immigration process, and include targeted orientation to refugees overseas. The strategy will be developed in consultation with Provinces and Territories as well as Employment and Social Development Canada (ESDC).

Action #2a

Develop a whole-of-Department promotion strategy outlining key activities and timelines. The strategy will identify roles within IRCC, in particular National Headquarters and Missions, and of SPOs in promoting services.

Accountability: Communications Branch

Support: Settlement and Integration Policy Branch, Settlement Network, International Network, Client Experience Branch, Refugee Affairs Branch, Provinces/Territories

Completion Date: Q2 2018/19

Action #2b

Implement Promotion Strategy.

Accountability: Communications Branch

Support: Settlement and Integration Policy Branch, Settlement Network, International Network, Client Experience Branch

Completion Date: Q2 2018/19

Action #2c

Analyze the timing of pre-arrival client eligibility within the immigration process, and develop options to optimize the length of time available for clients to access services. The options will be presented to Senior Management for consideration.

Accountability: Settlement and Integration Policy Branch

Support: Client Experience Branch, Settlement Network

Completion Date: Q4 2018/19

Recommendation #3

IRCC should clarify and strengthen its governance to lead and coordinate pre-arrival services by:

• Establishing clear roles and responsibilities among internal IRCC stakeholders, including across NHQ, regions and International Network
• Clarifying the role of Missions in the delivery and monitoring of in-person pre-arrival services and SPOs

Response #3

IRCC agrees with this recommendation.

The Department acknowledges the challenges associated with the rapid expansion of pre-arrival services in 2014, including low client awareness and uptake; a lack of coordination; and inconsistent service delivery.

IRCC is committed to improving its internal governance to increase coordination and support improved pre-arrival services.

Action #3a

Establish (via a Director General level table) a structured Accountability Framework for pre-arrival services that clearly identifies roles and responsibilities across IRCC.

Accountability: Settlement and Integration Policy Branch

Support: Settlement Network, International Network, Communications Branch, Refugee Affairs Branch

Completion Date: Q2 2018/19

Action #3b

Develop guidance and tools for Missions to support project monitoring.

Accountability: Settlement Network

Support: International Network

Completion Date: Q3 2018/19

Action #3c

Finalize and implement Accountability Framework.

Accountability: Settlement and Integration Policy Branch

Support: Settlement Network, International Network, Communications Branch, Refugee Affairs Branch

Completion Date: Q4 2018/19

Recommendation #4

IRCC should establish a mechanism to promote collaboration, cross-referrals and sharing of best practices among pre-arrival SPOs and between pre-arrival and domestic settlement SPOs.

Response #4

IRCC agrees with this recommendation.

In January 2018, IRCC hosted an in-person pre-arrival settlement services meeting, bringing together representatives from pre-arrival settlement service providers, provincial and federal governments, to help strengthen communication and partnerships and consider how to enhance program outcomes.

Building on the successful meeting, IRCC recognizes the need for more formalized coordination among pre-arrival services providers and between service providers and key partners, including provinces and territories and domestic settlement service providers.

Engagement with pre-arrival services providers and across key partners, including domestic settlement service providers, provinces and territories and ESDC, will be key to establishing formalized coordination mechanisms.

Action #4a

Fund a National pre-arrival coordinating body through the Spring 2018 intake process, with a secretariat function to facilitate consultations with the settlement sector, foster learning and exchange and help ensure alignment of programming.

Accountability: Settlement Network

Support: Settlement and Integration Policy Branch, Settlement Sector

Completion Date: Q2 2018/19

Action #4b

Establish a pre-arrival working group with pre-arrival service providers and key partners, including domestic service providers, Provinces and Territories and ESDC.

Accountability: Settlement and Integration Policy Branch

Support: Settlement Network, Settlement Sector, Provinces/Territories, Employment and Social Development Canada

Completion Date: Q3 2018/19

Recommendation #5

IRCC should strengthen performance measurement and reporting for pre-arrival services, by:

• Developing key indicators and data strategies to support collection of performance information
• Considering developing a targeted Performance Information Profile for pre-arrival services which aligns with IRCC's Settlement Program and Resettlement Program Performance Information Profiles

Response #5

IRCC agrees with this recommendation.

IRCC recognizes that in order to ensure high-quality, client focused and effective and efficient services, pre-arrival programming needs to be measured and analysed on an ongoing basis and will seek to strengthen its approaches to performance measurement and reporting.

Action #5a

Develop and implement a performance measurement and data strategy for pre-arrival that is aligned with IRCC’s Settlement and Resettlement performance information profiles.

Accountability: Settlement and Integration Policy Branch

Support: Research & Evaluation, Settlement Network, Provinces/Territories, Refugee Affairs Branch

Completion Date: Q2 2018/19

Action #5b

Strengthen reporting of pre-arrival services delivered to refugees through a targeted service arrangement with the International Organization for Migration (IOM).

Accountability: Settlement Network

Support: Research & Evaluation, International Network, Settlement Network, Settlement and Integration Policy Branch

Completion Date: Q2 2018/19

Action #5c

Create management tool (e.g. dashboard) for regular and ongoing tracking of client uptake and profiles, to support responsive programming.

Accountability: Settlement and Integration Policy Branch

Support: Research & Evaluation, Settlement Network

Completion Date: Q3 2018/19

1. Introduction

1.1. Purpose of the Evaluation

This report presents the findings of the evaluation of Immigration, Refugee and Citizenship Canada’s (IRCC) Pre-Arrival Settlement Services, hereafter referred to as pre-arrival services, for fiscal years (FY) 2015/16 to 2017/18. The evaluation was conducted to inform IRCC’s Intake Process for pre-arrival services in Spring 2018. It is also being conducted in fulfillment of requirements under the Treasury Board Policy on Results and section 42.1 of the Financial Administration Act.^{Footnote 2}

The scope of this evaluation included in-depth analyses using multiple lines of evidence, focusing primarily on the outcomes for both immigrants and refugees that had accessed pre-arrival services. The evaluation also assessed other elements such as the effectiveness of IRCC and Service Provider Organization (SPO) promotion and its impact on uptake, and delivery models (e.g., web-based vs. in-person), and overall program management.

1.2. Pre-Arrival Services Background and Context

The objective of IRCC’s pre-arrival services is to provide selected Permanent Residents,^{Footnote 3} including refugees, with accurate, relevant information and supports so that they can make informed decisions about their new life in Canada and begin the settlement process (including preparation for employment) while overseas. It is expected that these services will enable clients to be better prepared upon arrival to Canada to settle and integrate into Canadian society.

1.2.1. History and Evolution

The Government of Canada has funded the delivery of pre-arrival services since 1998. While initially only provided to refugees, services were expanded to non-refugee immigrants in 2001 (see Figure 1).

An evaluation of pre-arrival services was completed in July 2012.^{Footnote 4} This evaluation concluded that while overall clients found pre-arrival services useful, there was no formal articulated common approach or framework in place for the provision of pre-arrival services.

In 2014, IRCC launched a Call for Proposals to expand pre-arrival services and increased funding from $9M in 2014/15 to $24M in 2015/16 and $32M in 2016/17. The intent was to provide more comprehensive tailored in-person, and web-based pre-arrival orientation and supports. This call resulted in funding for 27 SPOs through 28 contribution agreements (CA). Prior to this expansion, only three SPOs delivered these services: Colleges and Institutes Canada (CICan),^{Footnote 5} SUCCESS, and the International Organization for Migration (IOM). In FY2014/15, the fiscal year prior to the expansion, these three SPOs provided pre-arrival services to 19,455 clients.^{Footnote 6}

In June 2016, the Department conducted an internal Management Review of the expanded pre-arrival services. Although minimal data was available, the Review highlighted a number of implementation challenges requiring action by the Department (see Section 6.4). To provide the Department and service providers with more time to put in place new operational measures, a decision was made to extend 23 out of 28 CAs for one year to March 31, 2018, for a total value of approximately $27.2M for FY2017/18.^{Footnote 7}

Figure 1: Evolution of Pre-Arrival Settlement Services

Source: Internal program documents.

1.2.2. Types of Services

Pre-arrival services provide the same types of services as IRCC-funded in-Canada Settlement services, with the exceptions of Language Assessments and Language Training. Support Services such as interpretation and translation, childminding, as well as transportation are also provided by just over one third of pre-arrival SPOs to enable newcomers to access pre-arrival services.^{Footnote 8} Through contribution agreements, IRCC funds SPOs such as immigrant-serving agencies, industry/employment-specific organizations or educational institutions to provide the following types of pre-arrival services:

• Needs Assessments and Referrals (NARS): NARS are conducted to assess newcomers’ needs and link them to appropriate settlement and community-based services.
• Information and Orientation services (I&O): I&O services are offered to newcomers to provide relevant, accurate, consistent and timely settlement-related information and orientation that is needed to make informed settlement decisions, as well as promoting an understanding of life in Canada. Examples of I&O include settlement and labour market orientation sessions, general life skills development activities, etc.
• Employment-Related Services (ERS): ERS aim to equip newcomers with the skills, connections and support needed to enter into the labour market and contribute to the economy. Examples of ERS include resume screening, interview skills and practice, self-marketing and job-seeking skills, employment networking, pathways to foreign credential recognition, etc.
• Community Connections (CC): CC includes activities to support the two-way process of integration and facilitate adaptation on the part of newcomers and their host communities. Examples of CC include mentoring and matching newcomers with Canadians.
• Indirect Services (IS): IS includes activities that are not provided to newcomers directly but are designed to support the development of partnerships, capacity building and the sharing of best practices among SPOs. For example, indirect projects may focus on: developing new and innovative interventions, updating training content, conducting research, creating new tools as well as curricula, etc.

Pre-arrival services vary considerably in scope, delivery models and size (i.e., project funding and number of clients targeted).

For non-refugees, pre-arrival services are provided by both targeted and generalist SPOs. Targeted SPOs focus on providing information and support to specific sub-sets of newcomers. For example, some targeted SPOs are occupation-specific (i.e., only providing information for specific sectors or professions), regional-specific (i.e., only providing information for newcomers destined to certain provinces or regions), or Francophone SPOs (i.e., serving only Francophone newcomers).

As displayed in Table 1, a few targeted SPOs are both occupation-specific and regional-specific at the same time and a few are Francophone and regional-specific at the same time. Conversely, “generalist” SPOs provide information and services that do not focus on one particular sub-group of newcomers but instead provide pre-arrival services to all eligible newcomers.

Most pre-arrival services for non-refugees offer services via web-based platforms (e.g., information, needs assessment tools, webinars, live one-on-one needs assessments and counselling, virtual job fairs, etc.), while a few projects offer in-person services, primarily in top source countries.^{Footnote 9} Services to French speaking immigrants are offered by four Francophone SPOs as well as other SPOs who have French language capacity.

In FY2015/16, IRCC also funded 2 projects to provide only indirect services (i.e., projects not directly serving newcomers) including capacity-building and pre-arrival platform development for specific employment sectors.^{Footnote 10} Table 1 provides further details on the various types of pre-arrival services funded. Additional details regarding the various SPO types are provided in Section 2.2.

Pre-arrival services for refugees are provided via the Canadian Orientation Abroad (COA) Program which is delivered by the International Organization for Migration (IOM) and includes a 3 or 5 day in-person orientation session.^{Footnote 11},^{Footnote 12}

Table 1: Profile of Pre-Arrival Services

Source: GCS, iCARE

1.2.3. Delivery Partners

Several distinct groups within IRCC play a role in the administration and management of pre-arrival services:

• Settlement and Integration Policy Branch is the lead on developing policy regarding pre-arrival services
• Settlement Network is responsible for managing contribution agreements with pre-arrival SPOs and providing functional guidance – some contribution agreements are managed within National Headquarters and some are managed within IRCC’s Regional Offices across Canada
• International Network is responsible for processing PR applications and in some instances has supported promotional efforts to liaise with pre-arrival SPOs and promote pre-arrival services within IRCC Missions/visa offices abroad
• Centralized Network has been involved in developing a standardized invitation letter to inform prospective clients of the availability of pre-arrival services
• Communications Branch has been involved in developing IRCC’s pre-arrival services website^{Footnote 13}

In addition to IRCC and pre-arrival SPOs, a variety of stakeholders and partners play key roles in supporting the overall integration process for newcomers.

• Provinces and Territories provide funding (along with IRCC) for settlement services in Canada. Provinces/Territories also provide settlement information support and services (including websites which can be accessed by newcomers prior to their departure for Canada) in areas such as language training, labour market integration, recognition of foreign credentials, business development and youth integration. They also work with the Government of Canada on foreign qualification recognition issues.
• In-Canada Settlement Service Provider Organizations provide information and support to newcomers upon arrival to support newcomers’ settlement and integration. Many pre-arrival SPOs partner with in-Canada SPOs to provide referrals and ensure newcomers have a seamless continuum of settlement services upon arrival in Canada.
• Municipalities have developed websites and online resources which can be accessed by newcomers prior to arrival. Some also provide additional support to newcomers once in Canada (e.g., housing and public transportation).
• Other Federal Departments (e.g., Employment and Social Development Canada, Health Canada, Service Canada and Heritage Canada) fund various initiatives affecting newcomers, such as multiculturalism or foreign credential recognition initiatives and have also developed online resources which have been adopted by pre-arrival SPOs (e.g., Working in Canada Job Bank, National Occupation Classification system).
• Educational institutions offer bridging and training programs to help newcomers upgrade their skills once in Canada.
• Employers and Employer Associations, including Chambers of Commerce, Immigrant Employment Councils and Sector Councils play a role in supporting newcomer employment and the foreign credential recognition process.
• Regulators and Apprenticeship Authorities are responsible for licensure/trade certification and in some cases work with pre-arrival SPOs to provide industry-specific information to newcomers.

1.2.4. Location of Pre-Arrival In-Person Services

Since FY2015/16, eight SPOs have offered in-person pre-arrival services to newcomers either via permanent sites or temporary sites^{Footnote 14} overseas. Of these SPOs, 3 provide in-person services in multiple countries, while 5 focus on providing services in only one country.

In-person pre-arrival services are provided in 35 countries

• Permanent sites are located in 23 countries
• Temporary sites are located in 12 countries
• Services are provided to refugees in 25 countries
• Services are provided to non-refugees in 17 countries^{Footnote 15}

Figure 2 provides a map of countries where in-person pre-arrival services are offered. A full listing of countries is provided in Appendix A.

Figure 2: Location of In-Person Pre-Arrival Services

For map details on locations of pre-arrival services, please see Appendix A

1.2.5. Cost of Pre-Arrival Settlement Services

According to IRCC financial data, $61.7^{Footnote 16} million was expended on pre-arrival services between April 1, 2015 and August 31, 2017.

• $38.7 million (63%) was spent on funding SPOs which provide services in-person^{Footnote 17}
• $23.0 million (37%) was spent on funding SPOs which provide services online only

In terms of IRCC’s Settlement Program components, just over half (55%) of pre-arrival expenditures fell under Information and Orientation, followed by Employment-Related Services (18%), Indirect Services (6%), Needs Assessments and Referral Services (5%) and Community Connections (3%).^{Footnote 18}

As displayed in Figure 3, the largest share of pre-arrival SPO expenditures was spent on salaries and benefits (45%) followed by professional fees, consulting and administration (15%).

Figure 3: SPO Expenditures (April 1, 2015 to August 31, 2017)

Source: GCS

2. Pre-Arrival Services Profile

2.1. Clients and Non-Clients

iCARE administrative data was used to examine the profile of newcomers who were admitted to Canada between April 1, 2015 and August 31, 2017, including those who had used at least one pre-arrival service (i.e., pre-arrival clients) and those that had not accessed a pre-arrival service (i.e., pre-arrival non-clients). Overall, during this period, 30,163 newcomers (defined as non-refugees and refugees) used at least one-pre-arrival service and 382,733 had not used any pre-arrival services.^{Footnote 19}

In addition, a separate analysis compared the socio-demographic profile of pre-arrival clients with clients of domestic IRCC-funded settlement services.

Overall, non-refugees (both pre-arrival clients and non-clients) have a higher level of education, greater knowledge of English/French than refugees (both pre-arrival clients and non-clients) and access pre-arrival services earlier but at a lower rate (see Section 6.1 for more details on uptake).

Non-Refugees

Key characteristics of the 19,726 non-refugees^{Footnote 20} who have received at least one pre-arrival service and were admitted to Canada between April 1, 2015 and August 31, 2017 include the following:

• Immigration category: Most (87%) are economic immigrants and 13% are sponsored family
• Family status: Just over half (55%) are principal applicants and 45% are spouses or dependants
• Gender: Just over half (54%) are male
• Age: Just over three quarters (76%) are between the ages of 25 and 44
• Knowledge of Official Languages: Most (93%) have some knowledge of an official language
• Level of Education: Just over half (55%) have a Bachelor or Master’s degree
• Country of Citizenship: Just under three quarters come from the Philippines (42%), India (15%) or China (15%)
• Intended Province of Destination: Most are destined to Ontario (32%), Alberta (14%) or Quebec (13%)
• Type of Pre-Arrival Services Accessed: Nearly all (97%) access an Information and Orientation while most (83%) access a Needs Assessment and Referral Service^{Footnote 21}
• Number of pre-arrival SPOs accessed: Most (78%) access services from 1 pre-arrival SPO
• Time at which services accessed: Around half (53%) access pre-arrival services 5 weeks or more before being admitted to Canada while 39% accessed services either in the same week as their admission to Canada (8%) or 1-4 weeks before being admitted to Canada (31%)

Figure 4: Non-Refugee Clients – Time from using Pre-Arrival Service to Admission

Source: GCMS & iCARE

In addition, 75% of non-refugee pre-arrival clients aged 18 and above at time of admission and who have an intended occupation specified^{Footnote 22} intend to work in:

• Managerial occupations (19%)
• Professional occupations (NOC skill level A) (56%)
• Skilled and technical occupations (NOC skill level B) (21%)
• Intermediate and Clerical (3.7%) or Elemental and Labourers (0.3%) (NOC skill level C and D)

When comparing non-refugee pre-arrival clients with non-clients and clients of IRCC-funded domestic Settlement services, a higher proportion of pre-arrival clients:

• were male (54% of clients vs. 47% of non-clients and 45% of domestic clients)
• were between the ages of 25 to 44 (76% of clients vs. 56% of non-clients and 59% of domestic clients)
• were economic immigrants (87% of clients vs. 56% of non-clients and 64% of domestic clients)
• had ability in an official language (93% of clients vs. 80% of non-clients and 81% of domestic clients)
• had a Bachelor Master’s degree (53% of clients vs. 41% of non-clients and 44% of domestic clients)

Refugees

Key characteristics of the 10,437 refugees^{Footnote 23} over the age of 10 who have received at least one pre-arrival service and were admitted to Canada between April 1, 2015 and August 31, 2017 include the following:

• Immigration category: PSRs were the largest group (48%) followed by GARs (44%) and BVOR (8%)
• Family status: Just over half (54%) are principal applicants and 46% are spouses or dependants
• Gender: Just over half (52%) are male
• Age: Just under half (48%) are between the ages of 25 and 44 while just over one third (34%) were between the ages of 10 to 24
• Knowledge of Official Languages: Most (63%) have no knowledge of an official language
• Level of Education: A small proportion (9%) have a Bachelor or Master’s degree
• Country of Citizenship: Syria (40%), Iraq (14%) or Eritrea (14%)
• Intended Province of Destination: Most are destined to Ontario (44%), Quebec (15%) or Alberta (12%)
• Type of Pre-Arrival Services Accessed: All (100%) access an Information and Orientation while only a small proportion (2%) access a Needs Assessment and Referral
• Number of pre-arrival SPOs accessed: Nearly all (~100%) access services from only 1 pre-arrival SPO (i.e., International Organization for Migration)^{Footnote 24}
• Time at which services accessed: Most (80%) access pre-arrival services either in the same week as their admission to Canada (43%) or 1-4 weeks before being admitted to Canada

Few differences were observed when comparing refugee pre-arrival clients to non-clients and clients of IRCC-funded domestic Settlement services.

A full profile of pre-arrival service clients, non-clients and domestic IRCC-funded Settlement clients is presented in Appendix B.

2.2. Typology of Pre-Arrival Service Provider Organizations

Pre-arrival services funded by IRCC vary considerably in scope, delivery models and size (i.e., number of targeted clients and project funding).

Of the 27 SPOs that were funded since the beginning of FY 2015/16, 2 were providing indirect services, and 25 were targeted to provide direct services to clients. In terms of the 25 providing direct services to clients, the most common type of pre-arrival service provided by pre-arrival SPOs was Information and Orientation (84%), followed by Employment-related Services (76%), Needs Assessments and Referrals (56%) and Community Connections (32%).^{Footnote 25}

In addition, other characteristics of the 25 pre-arrival SPOs providing direct services between FY 2015/16 and FY 2017/18 were as follows:

• Francophone organizations: 4 (16%) specifically targeted Francophone clients, while 13 (52%) were able to offer services in French
• Service delivery model: 8 (32%) offered services in-person (with some services online), 17 (68%) offered services only online
• Providing services in non-official languages: 9 (36%) offered services in local languages
• Pre and post-expansion organizations: 3 (12%) had been offering services prior to the 2015 expansion while 22 (88%) were added as part of the 2015 expansion
• Temporary sites: 3 (12%) delivered services in temporary delivery sites (in addition to permanent sites)
• Occupation-specific: 7 (28%) targeted specific professions
• Destination-specific: 7 (28%) targeted newcomers destined to specific locations in Canada
• Immigration classes: 10 (40%) targeted only economic immigrants, 1 (4%) targeted only refugees
• Project Funding (3 fiscal years): 5 (20%) under $1,000,000, 10 (40%) between $1,000,001 and $2,000,000, 3 (12%) between $2,000,000 and $3,000,000, 5 (20%) over $3,000,000
• Annual client targets: 7 (28%) target 0-250, 5 (20%) target 251-500, 5 (20%) 501-750, 2 (8%) 751-1,000 and 6 (24%) over 1,000

A full listing of pre-arrival SPOs and the services they provide is included in Appendix C.

3. Methodology

3.1. Evaluation Approach

The evaluation scope and approach were determined during a planning phase, in consultation with the IRCC branches involved in the design, management and delivery of pre-arrival services. The terms of reference for the evaluation were approved by IRCC’s Performance Measurement and Evaluation Committee in September 2017, and the evaluation was conducted by the IRCC Evaluation team from September 2017 to January 2018.

3.2. Evaluation Scope

The evaluation focused on the impact of IRCC-funded pre-arrival services since FY 2015/16, and was guided by IRCC’s Settlement Program logic model, which outlines the expected immediate and intermediate outcomes for pre-arrival and domestic IRCC-funded Settlement services (see Appendix D for the logic model).

The primary areas of focus for the evaluation were:

1. the extent to which pre-arrival services are providing refugees and other immigrants with accurate, relevant information and supports so that they can better prepare for life in Canada and begin the settlement process (including preparation for employment and foreign credential recognition) while overseas
2. the effectiveness of promotion of pre-arrival services by IRCC and pre-arrival SPOs
3. the extent to which pre-arrival services are providing refugees and other immigrants with linkages and pathways to accessing settlement services in Canada
4. the efficacy of the various pre-arrival delivery models (e.g., web-based vs. in-person)

As a secondary focus, the evaluation also conducted a review of contextual and performance issues of pre-arrival services, and to the extent possible, identified gaps, management and program challenges to inform the continued value of delivering pre-arrival services.

Specific evaluation questions were developed to address these core issues.^{Footnote 26}

3.3. Data Collection Methods

Multiple lines of evidence were used to gather qualitative and quantitative data from a wide range of perspectives, including pre-arrival service clients, stakeholders, and program officials.

Lines of Evidence

Newcomer Client and Non-Client Surveys

Two surveys were developed and administered to newcomers that were admitted to Canada between April 2015 and August 2017 with valid contact information, over the age of 18.

1. Survey of non-refugee immigrants – the survey contained 48 questions and covered a variety of topics including awareness and usefulness of pre-arrival services and the extent of difficulties faced by respondents post-arrival in Canada. The survey was administered to newcomers who had accessed at least one IRCC-funded pre-arrival service and included a comparison group of newcomers who had not accessed a pre-arrival service. The survey was administered in English and French. A total of 4,303 non-refugee immigrants completed the survey (2,015^{Footnote 27} had accessed at least one pre-arrival service and 2,288 had not accessed a service, representing a 12.0% response rate).
2. Survey of refugees – the survey contained 21 questions and focused on the awareness and usefulness of pre-arrival services. The survey was administered to refugees that had accessed at least one IRCC-funded pre-arrival service and included a comparison group of refugees that had not accessed a pre-arrival service. The survey was administered in English, French and Arabic. A total of 2,443 refugees completed the survey (1,448^{Footnote 27} had accessed at least one pre-arrival service and 995 had not accessed a service, representing an 8.7% response rate).

Survey results were weighted to ensure representativeness. With the weights applied, the profile of survey respondents reflects the overall population. The overall margins of error for the non-refugee immigrant and refugee surveys were +/-1.48% and +/-1.89%, respectively, with confidence intervals of 95% for both.

Interviews/questionnaires

Interviews/questionnaires were conducted with key stakeholders, including IRCC National Headquarters staff (15), IRCC Regional staff (20), IRCC International staff (9), SPOs (24) and Provincial/Territorial representatives (6). These key informants provided insight into the relevance and performance of pre-arrival services.

Administrative Data Analysis

Administrative data analysis included examining the socio-demographic characteristics of pre-arrival services clients as well as the specific services they received.

Information on planned SPO activities and targets was derived from IRCC’s Grants and Contributions System (GCS).

Data on client uptake and services received was obtained from IRCC’s Immigration Contribution Agreement Reporting Environment (iCARE) which is a system used by SPOs to report on clients and services as per their Contribution Agreements. Information from iCARE is subsequently linked by IRCC with the socio-demographic client information obtained through immigration files contained in IRCC’s Global Case Management System (GCMS).

Financial Data Analysis

Financial data for pre-arrival services costs were analyzed using IRCC’s SAP financial system.

Document and Literature Review

A targeted review of key documentation was conducted to support an assessment of program relevance and provide context for various delivery models. Documentation sources included IRCC, SPO material, as well as academic literature.

Direct observation of service delivery and focus groups with newcomers post-services

Pre-arrival information sessions/workshops (15) to newcomers delivered by 7 different SPOs (both online and in-person) were observed to support an analysis of various delivery models.

Post-session focus groups were conducted (6) with 72 newcomers (both immigrants and refugees) to gain newcomer perspectives on the awareness and perceived usefulness of pre-arrival services immediately following the receipt of in-person services.

3.4. Considerations and Limitations

Overall, the evaluation design employed numerous qualitative and quantitative methodologies. The different lines of evidence were complementary and reduced information gaps and the results converged towards common and integrated findings.

A few limitations to this evaluation include:

• Only a small number of focus groups were conducted with pre-arrival clients – While insights were gained from conducting focus groups with clients directly following the receipt of pre-arrival services, due to limited number of sessions, focus group results are likely not representative of all immigrants or refugees and should be considered exploratory.
• Known under/over-reporting of pre-arrival clients in iCARE for some SPOs affected the confidence in client counts – iCARE only began to capture information on pre-arrival services in October 2015. As such, services provided during two quarters of the first fiscal year considered for this evaluation were missing. In addition, some SPOs have faced challenges in reporting clients, particularly refugees, in iCARE; this was also confirmed during the evaluation analysis. Despite these challenges, the evaluation was able to effectively estimate client uptake by comparing iCARE records with SPO client records as well by verifying with survey respondents whether they had received pre-arrival services (see Section 6.1).
• Other factors to which non-refugee client outcomes may be attributed – While the evaluation used multivariate regression models to isolate the unique impact of IRCC-funded pre-arrival services and certain socio-demographic variables (e.g., age, gender, country of citizenship, year of admission in Canada, etc.) on influencing client outcomes, it was not possible to control for all contributing factors such as learner motivation or all the different delivery models and curricula used by SPOs. Despite this, statistically significant results were obtained which allowed for robust inferences to be made about the contribution of pre-arrival services.
• Representativeness of the refugee survey – While survey results were weighted to ensure the profile of respondents is aligned to the profile of the targeted refugee population, a potential non-respondent bias remains, as the survey was only conducted online, in English, French and Arabic. As such, survey responses are more likely to represent the views of refugees who are literate in one of these languages and who were able to access the survey online.

Despite these limitations, the triangulation of the multiple lines of evidence, along with the mitigation strategies used in this evaluation were considered sufficient to ensure that the findings are reliable and can be used with confidence.

4. Relevance

4.1. Continued Need and Relevance for Pre-Arrival Services

Both the literature reviewed and an analysis of data from IRCC’s Needs Assessment iCARE module confirmed the existence of gaps in knowledge, as many newcomers face a variety of challenges and have a variety of needs upon arrival (i.e., limited contacts or networks and unfamiliarity with Canadian institutions, how to find work or educational opportunities in Canada and dealing with cultural barriers).^{Footnote 28} For example, a review of pre-arrival needs assessments revealed that the three needs most frequently identified by non-refugees were to increase knowledge of life in Canada (95%), community and government services (94%) and working in Canada (86%). Some (30%) of non-refugee NARS clients also had non-IRCC needs that were identified, the most frequent being in the areas of education/skills development (26%), and employment (22%).

While literature and several IRCC surveys of newcomers^{Footnote 29} confirmed that many newcomers utilize multiple sources of information to fill these gaps, including post-arrival settlement services, websites, informal networks of friends or family in Canada, some key informants noted that many newcomers may either not be knowledgeable about or not have access to these sources until they come to Canada.

The main objective of providing settlement services prior to arrival is to further accelerate settlement and integration, as it is expected that by accessing these services newcomers become more informed about the settlement and integration process at an earlier stage and will take steps to better prepare for their eventual settlement and integration in Canada.

In particular, interviewees noted that pre-arrival services are designed to reduce information gaps and allow newcomers to be better prepared upon arrival in Canada by:

• Reducing stress both prior to coming to Canada and post-arrival
• Contributing to more realistic expectations of life in Canada and steps needed to successfully settle and integrate
• Allowing economic immigrants to begin searching for a job and the foreign credential recognition process earlier

While a few interviewees suggested that taking pre-arrival services may reduce the need for in-Canada settlement services, most noted that newcomers will still need support in Canada and explained pre-arrival services should be seen as complementary to post-arrival settlement services as they serve as the first step on the settlement service continuum by providing newcomers with tailored needs assessments and referral information on where to obtain more information and support once in Canada.

A few interviewees also suggested that requiring all newcomers to access at least one pre-arrival service could be beneficial. These interviewees cited the example of IRCC’s Atlantic Immigration Pilot, which currently requires all newcomers to receive a needs assessment from a service provider and develop a personalized settlement plan in order to be processed under the Pilot. These interviewees noted that there is likely an even greater need for non-Atlantic Pilot immigrants to receive a needs assessment and develop a settlement plan as, unlike Atlantic Pilot participants, most are not admitted to Canada with a prior job offer.

Variations in the need for pre-departure orientation

Key informants agreed the need for pre-arrival services differs by socio-demographic characteristics (e.g., immigration category, education, age, sex, country of origin, family status and knowledge of official languages) and stressed the need for flexible and tailored pre-arrival services to serve the unique needs of the newcomers.^{Footnote 30}

Needs of Refugees

The need for pre-arrival services for refugees was noted in IRCC’s 2016 Rapid Impact Evaluation of the Syrian Refugee Initiative, which found that the lack of pre-arrival services as part of the Syrian Refugee Initiative was a major gap which resulted in many of them “arriving in Canada lacking basic information, having gaps in knowledge, and arriving with assumptions.”^{Footnote 31}

This finding was supported by interviewees, as nearly all indicated that refugees had a need for pre-arrival services to develop a very clear picture of conditions in Canada and of the expectations placed on them, and to help reduce the anxiety felt by refugees in the first weeks in a new country.

However among refugees, key informants suggested that camp-based refugees are more in need of specific information focusing on an overview of Canada, travelling and immigration procedures at the airport, while urban refugees have a greater need for employment-related information. This was supported during focus groups with urban privately-sponsored refugees who all stated they would like more pre-arrival information on the Canadian labour market and how to find a job in Canada.

Needs of Non-Refugees

The majority of interviewees felt that economic immigrants also have a need for pre-arrival services. However, many interviewees highlighted that the type of information needed varies based on a broad variety of factors such as level of education, occupation, country of origin.

Both interviewees and administrative data on services accessed indicate that, when compared with other immigration categories, economic immigrants generally seek more specific employment-related information and services (e.g., resume building, credential recognition, sector-specific labour market information). For instance, analysis of iCARE data indicates that 31% of economic immigrants who accessed pre-arrival services obtained employment-related services compared to 17% for family class immigrants and 0.3% for resettled refugees. In addition, a higher proportion of pre-arrival clients who were principal applicants accessed pre-arrival services as compared to spouses and dependants.^{Footnote 32} Many economic immigrants are also interested in receiving and benefit from non-employment focused information and orientation services about life in Canada (e.g., weather, culture, healthcare, laws, cost-of-living, etc.) that contribute to more realistic expectations about living in Canada and explaining the steps needed to successfully settle and integrate.

Interviewees were split on the need to provide pre-arrival services to Family Class immigrants. While some interviewees suggested that family class immigrants have a lower level of need for pre-arrival services, because they already have family members in Canada who can provide information and support, others noted that the information provided by family members in Canada is not always accurate or detailed enough to support the specific needs of Family Class immigrants.

Other socio-demographic characteristics

Aside from immigration category, some interviewees mentioned that women may require more specific information around Canadian cultures and gender equality. Similarly, individuals with higher levels of education and/or principal applicants may be more likely to require specific labour market information as opposed to those with lower levels of education or spouses/dependants. Youth was also mentioned as a particular group requiring specific information tailored to them. In terms of countries, some interviewees noted that type and level of needs differ according to the distance between Canadian culture and the culture of origin. For example, while individuals from Western countries also have a need for pre-arrival services related to foreign credential recognition and employment, they may have less of a need for information and orientation on Canadian culture and values.

5. Impact of Pre-Arrival Services

5.1. Usefulness of Pre-Arrival Services

Both non-refugee and refugee pre-arrival service clients reported a high level of satisfaction with the services they received.

Overall, 79% of non-refugees surveyed indicated that pre-arrival services were useful or very useful.

As indicated in Table 2, when asked about specific topics, the majority of non-refugee clients indicating that pre-arrival services helped them in a number of areas including preparing for the trip to Canada, knowing how to contact organizations that provide help in settling in Canada and understanding their rights, freedoms and responsibilities.

Table 2: Share of non-refugees indicating the information was helpful

Source: Non-refugee survey

In addition, as indicated in Table 3, non-refugee clients also indicated that pre-arrival services helped them prepare to look for a job in Canada.

Table 3: Share of non-refugees indicating the information was helpful to prepare to look for job

Source: Non-refugee survey

Nearly all refugees surveyed agreed that the information session they attended was helpful: to prepare for the trip, to adjust to life in Canada and Canadian culture and to understand their rights, freedoms and responsibilities as well as other aspects of their resettlement in Canada (see table 4).

Table 4: Share of refugees indicating the information session was helpful

Source: Refugee survey

As indicated in Table 5, a higher proportion of pre-arrival clients, compared to non-clients, consistently reported having enough information prior to coming to Canada. As indicated in Table 5, the largest differences between pre-arrival clients were regarding: contacting organizations that provide help settling in Canada (27 percentage point difference between clients and non-clients), knowing how to have professional credentials or qualifications recognized (20 percentage point difference between clients and non-clients), and understanding Canadian workplace culture and norms (15 percentage point difference between clients and non-clients).

Table 5: Non-refugees level of information prior to coming to Canada

Source: Non-refugee survey

Of the above topics, when controlling for socio-demographic characteristics (e.g., age, gender, immigration category, level of education, country of citizenship), there were no statistically significant differences between clients and non-clients in terms of having enough information prior to coming to Canada to: prepare for the trip to Canada, meet their initial settlement needs or understand their rights, freedoms and responsibilities. This suggests that the differences between clients and non-clients for these aspects are more related to individual characteristics, than to the pre-arrival services received.

Non-refugee clients surveyed were also asked if there were areas where more information or services would have been helpful.

Of the 13% who indicated that additional information about life in Canada would have been helpful, the following areas were most often identified:

• 24% cited employment-related information
• 13% cited information about daily life in Canada
• 9% cited information about regional and local specificities

While many refugees (63%) felt that the amount of information provided to them through the orientation sessions was appropriate,^{Footnote 33} about one third (31%) also felt they could have benefited from more information, especially in the areas of employment, education and health care. This was confirmed during focus groups with urban refugees as the majority indicated they would have liked to have more pre-arrival employment-related information and support.

Just over half of refugee clients surveyed felt the information provided to them about what to expect in Canada (56%)^{Footnote 34} and where to find help in Canada (53%) was accurate or very accurate.

Areas where refugee clients felt the information provided to them was less accurate were:

• Jobs in Canada (58% of those indicating the information on what to expect in Canada was not at all accurate or somewhat accurate)
• Cost of living (48% of those indicating the information on what to expect in Canada was not at all accurate or somewhat accurate)
• Financial supports (33% of those indicating the information on what to expect in Canada was not at all accurate or somewhat accurate)

Through observation of an orientation session for refugees, it was noted that the service provider did acknowledge that there will be an adaptation period to adjust to life in Canada, however they did not go into detail in terms of specific challenges that refugees may encounter (e.g., high cost of living, difficulty of finding employment and living on limited income/social assistance, etc.). Not providing more in depth information on these topics may have contributed to refugee perceptions of information not being accurate.

5.2. Alignment of Pre-Arrival Services Information with Specific Client Needs

In order to measure whether pre-arrival service information was aligned with specific needs, the evaluation examined the extent to which material provided through each pre-arrival service is tailored for specific groups,^{Footnote 35} as well as whether any particular groups found pre-arrival service information not useful, thus indicating a potential misalignment with specific client needs.

As noted in Section 2.2, more than half (60%) of SPOs are targeted in that they only provide pre-arrival services to certain groups of newcomers (i.e., specific professions, specific destinations) suggesting a high degree of alignment with the needs of one particular newcomer group. Moreover, during interviews, all of the “generalist” SPOs noted that they attempt to provide tailored material or support for particular groups, with some developing specific material for specific source countries, immigration categories, regulated vs. non-regulated professions, etc.

IRCC interviewees noted that in general, pre-arrival services seem to be aligned with specific clients’ needs although there are opportunities to include more tailored services aligned with labour market needs and immigration flows, especially related to employment and labour market information or adapted to local destination. For example, while there are currently pre-arrival services specifically for IT professionals and nurses, there are no specific pre-arrival services for doctors or engineers. Despite this desire for more tailored services, it was also mentioned by several interviewees and observed that SPOs funded with a focus on very specific clientele may face difficulties in attracting clients because the pool of eligible newcomers is reduced.

Overall, few significant differences were observed between various non-refugee subgroups (e.g., gender, immigration category, marital status) in terms of perceived usefulness of pre-arrival services and nearly all subgroups examined found pre-arrival services useful.^{Footnote 36}

In terms of refugees, as indicated in section 5.1, both Government-Assisted Refugees and Privately Sponsored Refugees generally found pre-arrival services helpful with no statistically significant differences observed between the two groups.

5.3. Clients gain knowledge of life in Canada and the Canadian work environment

Survey results indicate that non-refugees (both clients and non-clients) report a high level of knowledge of life in Canada upon arrival. However, as indicated in Table 6, a higher proportion of non-refugee pre-arrival clients consistently reported having a higher knowledge compared to non-clients. The largest differences between pre-arrival clients were in terms of knowing where to go for assistance to settle in Canada (22.3 percentage point difference between client and non-clients), being prepared to look for a job (10.0 percentage point difference between client and non-clients) and knowing what to do to settle (7.2 percentage point difference between clients and non-clients).^{Footnote 37}

Table 6: Non-refugees’ knowledge of life in Canada upon arrival

Source: Non-refugee survey

Of the above topics, when controlling for socio-demographic characteristics (e.g., age, gender, immigration category, level of education, country of citizenship), there were statistically significant differences between clients and non-clients who had taken any pre-arrival services in terms of knowing what they needed to do to settle in Canada and knowing where to go for assistance to help settling in Canada, suggesting that pre-arrival services have a positive impact on these aspects of the knowledge of life in Canada.^{Footnote 38}

In addition, when controlling for these same socio-demographic characteristics, there were a statistically significant differences in terms of being prepared to look for a job in Canada upon arrival between clients that had accessed an employment-related pre-arrival service compared to those who had not accessed an employment-related pre-arrival service. This suggests that employment-related pre-arrival services are positively impacting clients’ knowledge of how to look for a job in Canada.

As a few pre-arrival projects include mentoring/matching activities, the survey also asked information related to having a mentor/matching arrangement. A higher proportion of non-refugee pre-arrival clients reported having access to a mentor or matching arrangement when they first arrived in Canada (58% of clients vs 36% of non-clients).^{Footnote 39}

As indicated in Table 7, a higher proportion of non-refugee pre-arrival clients (compared to non-clients) reported that the following were not at all difficult or somewhat difficult after their first three months in Canada:

• accessing child care (12.6 percentage point difference between clients and non-clients)
• having professional credentials and credentials recognized (10.2 percentage point difference between clients and non-clients)
• knowing where to get help on settlement questions (9.2 percentage point difference between clients and non-clients)
• finding/accessing job-related training opportunities (9.1 percentage point difference between clients and non-clients)^{Footnote 40}

Table 7: Non-refugees’ difficulties experienced during the first three months in Canada

Source: Non-refugee survey

Of the above topics, when controlling for socio-demographic characteristics (e.g., age, gender, immigration category, level of education, country of citizenship) there were no statistically significant difference between clients and non-clients who had taken any pre-arrival services in terms of level of difficulty in terms of meeting their initial settlement needs and accessing child care in the first three months following admission.

However, when controlling for these same socio-demographic characteristics, there was a statistically significant difference in terms of having their professional credentials and qualifications recognized and finding/accessing job-related training opportunities between clients that had accessed an employment-related pre-arrival service compared to those who had not accessed an employment-related pre-arrival service. On the other hand, there were no significant differences between clients that had accessed an employment-related pre-arrival service and those who had not accessed an employment-related pre-arrival service in terms of difficulties looking for a job, getting a job, and getting a job that matches skills and qualifications in the first three months in Canada. This suggests that employment-related pre-arrival services have a positive impact in preparing individuals to enter the labour market, but have less of a direct impact on the longer-term outcome of finding employment.

Given that the focus of pre-arrival services for refugees (i.e., COA) is on preparing for the trip to Canada and providing general orientation and that employment is typically considered a longer-term outcome for this group, the refugee survey did not include questions pertaining to refugees’ experiences upon arrival or difficulties after three months in Canada.

5.4. Clients make informed decisions about life in Canada

Pre-arrival services are expected to provide newcomers with information and skills to make informed decisions about life in Canada. A large majority (80%) of non-refugees reported making a change as a result of taking pre-arrival services.

Specifically, non-refugee clients who indicated they made changes as a result of the pre-arrival services they received reported taking the following actions:

• look for work in Canada (53%)
• change what they were going to bring to Canada (42%)
• decide to further training/education to upgrade their skills instead of starting a job right away (37%)
• try to save more money before departing for Canada (34%)
• search for housing (33%)
• begin the process to have educational credentials recognized in Canada (25%)
• change the type of job they would try to find in Canada (22%)

As immediate settlement needs upon arrival to Canada for refugees are supported by IRCC-funded Resettlement Assistance Program SPOs and private sponsors, the refugee survey did not ask clients questions about making informed decisions. However, during observation of refugee focus groups, clients noted that most of the information they received through COA was new and felt it would eventually help them make informed decisions upon arrival in Canada.

5.5. Creating Pathways to Settlement Services and Usage in Canada

According to iCARE, the majority (83%) of non-refugee clients received at least one pre-arrival NARS^{Footnote 41} and of those, 95% obtained at least one referral for an IRCC Program need,^{Footnote 42} with a referral rate of about 90% or more for each type of need identified.

Most non-refugees surveyed (88%) indicated that pre-arrival services helped them know how to contact organizations that provide help settling in Canada. In addition, more non-refugee clients reported knowing where to go for settlement assistance (87%) than non-clients (65%) and fewer non-refugee clients (11%) also reported difficulties in their first three months in Canada in knowing where to get help on settlement questions compared to non-clients (20%).

Of the non-refugee clients surveyed who indicated receiving a referral, most were satisfied with their referrals as 54% indicated that referrals/linkages to settlement services in Canada completely or mostly met their needs; 40% that their needs were met somewhat, and 6% that they did not meet their needs at all.

Regarding refugees, both a review of COA material and observation of a COA session confirmed that GARs are provided with information about IRCC’s Resettlement Assistance Program SPOs and PSRs with information on the rights and obligations of their sponsors and that they are also eligible to receive IRCC-funded settlement services.

Over half (60%) of the pre-arrival clients admitted to Canada between April 2015 and August 2017 had accessed at least one IRCC-funded domestic settlement service, compared to 40% of the non-clients admitted to Canada over the same time period.

Refugees (pre-arrival clients and non-clients) are higher users of domestic settlement services than non-refugees (pre-arrival clients and non-clients) as between April 2015 and August 2017:

• Just over half (51%) of the non-refugee clients admitted to Canada had accessed IRCC-funded settlement services, compared to 36% of the non-clients admitted to Canada over the same time period
• Similarly, more refugee clients (79%) accessed IRCC-funded settlement services, compared to non-clients (75%)

Of the pre-arrival services clients who accessed settlement services in Canada, most obtained NARS (75%), I&O (88%) and Language Assessment (54%) services and about one quarter also accessed Language Training, Community Connections or Employment-related services.

Noting the high rate of referrals for pre-arrival clients, both interviews and a review of documentation revealed that most pre-arrival SPOs have strong partnerships and collaboration with in-Canada SPOs. Around one quarter (24%) of pre-arrival SPOs also offer settlement services in Canada making it easier for referral and service continuity. In addition, many providers, especially generalist SPOs, have referrals built into their delivery model thereby referring 100% of their clients to in-Canada SPOs.

Despite the high rates of referrals, a few challenges were noted. As pre-arrival SPOs form partnerships on their own, they do not partner with a wide range of IRCC-funded settlement service providers and instead rely on their established partners in Canada to make onward referrals once the client is in Canada. In some cases it was observed that the initial newcomer referrals were to organizations outside the city of intended destination which may not be the most efficient means of connecting newcomers with IRCC’s wide network of domestic settlement service providers.

Pre-arrival services were also identified by some interviewees as an opportunity for French-speaking newcomers to integrate into a “Parcours francophone”, to initiate and establish connections with Francophone communities, while, for some francophone organizations, enabling a “pour et par” approach. However, a limited number of pre-arrival SPOs are connected with domestic Francophone SPOs.

5.6. Clients Participate in the Canadian Labour Market

While pre-arrival services are not expected to provide newcomers with employment, they are expected to facilitate pathways to employment and expedite the job search and foreign credential recognition process, which will eventually lead to employment. As a result, the survey was used to compare the incidence of employment among non-refugees and refugees that had accessed pre-arrival services and those that had not accessed pre-arrival services.^{Footnote 43}

Non-Refugees

Overall, non-refugee clients reported working at a higher rate than non-clients.

• Clients: 65% were working, 18% were out of work and looking for work and 17% were unable to work or not looking for work^{Footnote 44}
• Non-clients: 50% were working, 7% were out of work and looking for work, and 36% were unable to work or not looking for work

However, among non-refugees, of those participating in the Canadian labour market (working or looking for employment), an equal share of clients and non-clients were working at the time of the survey (78%).^{Footnote 45}

Of those who indicated they were working:

• 75% of clients were working full time compared to 66% of non-clients
• 20% of both clients and non-clients were employed part-time
• A lower proportion of clients were self-employed (6%) compared to non-clients (14%)

Slightly more non-refugee pre-arrival clients who indicated working at the time of the survey felt that their current job was related at least to some extent to their education (65%) or experience (75%), compared to non-clients, for whom 60% indicated their job was related to their education and 71% to their experience.

While there are many factors which determine type of employment one can have, results indicate that, even after controlling for socio-demographic characteristics, non-refugees who have obtained pre-arrival services, and employment-related pre-arrival services in particular, are more likely to be employed in a job that was related to their education or experience.

Refugees

Although employment is not a major focus of COA, for comparison purposes, refugees were asked about their employment in the survey.

Overall, refugee clients reported working at a higher rate than non-clients.

• Clients: 53% were working, 19% were out of work and looking for work and 28% were unable to work or not looking for work
• Non-clients: 48% were working, 20% were out of work and looking for work, and 32% were unable to work or not looking for work

Among refugees, of those participating in the labour market (working or looking for employment), a slightly higher proportion of clients were working at the time of the survey (73% of clients vs 71% of non-clients).

PSR clients and non-clients participating in the labour market reported working at a higher rate than GARs:

• PSRs: 78% of clients reported working vs. 73% of non-clients
• GARs: 61% of clients reporting working vs. 62% of non-clients

Of all those refugees who were working:

• Clients: 63% were employed full time, 31% were employed part-time and 6% were self-employed
• Non-clients: 64% were employed full time, 26% were employed part-time and 10% were self-employed

6. Program Management

The evaluation assessed the extent to which program management outcomes have been achieved. As per the Settlement Program logic model, program development, management and governance are expected to support consistent and responsive program delivery and partners and stakeholders are expected to be engaged in settlement and integration strategies to address newcomers’ needs. These issues were explored via key informant interviews with program and external stakeholders. The evaluation also examined administrative data to assess program uptake, achievement of targets and cost per client.

6.1. Uptake of Pre-Arrival Services

As per the Settlement Program Terms and Conditions, essentially all permanent residents (PR) who have not yet become Canadian citizens^{Footnote 46} are eligible to receive pre-arrival services funded by IRCC. As services are not mandatory, not all newcomers access IRCC-funded pre-arrival services.

According to IRCC data, between April 1, 2015 and August 31, 2017, 30,163 admitted newcomers^{Footnote 47} received a pre-arrival service (at least one), which represents about 7.3% of all eligible newcomers admitted during this period (see Table 8).^{Footnote 48} However, as data on pre-arrival clients were not fully captured in FY15/16, the overall uptake rate is slightly under-estimated. Beyond the first fiscal year uptake remained stable at around 12% of all eligible newcomers.

Table 8: Official Uptake of Pre-Arrival Services by Fiscal Year of Admission (April 1, 2015 to August 31, 2017)

Source: GCMS and iCARE

As displayed in Table 9, uptake varies by immigration category, as a greater percentage of refugees accessed pre-arrival services compared to other immigration categories.

Since FY 2015/16, IRCC and some pre-arrival SPOs have faced challenges with regard to reporting of pre-arrival services clients in IRCC’s iCARE reporting system. In many cases, pre-arrival SPOs noted that they have faced challenges in obtaining the necessary client identification numbers with which to validate their clients in IRCC’s iCARE data reporting system. As such, the number of clients is underreported for some groups of newcomers, especially refugees.

As a result of known underreporting for refugees, an estimate of uptake was calculated by analyzing COA client records. When adjusting for these underreported clients, as displayed in Table 9, the estimated uptake for pre-arrival services increased from 7.3% to 11.6% of newcomers between April 1, 2015 and August 31, 2017 (from 17.3% to 46.5% for refugees).

Table 9: iCARE-based and Adjusted Uptake of Pre-Arrival Services by Immigration Category (April 1, 2015 to August 31, 2017)

Source: iCARE

Given that a large cohort of Syrian refugees did not receive COA as part of the Syrian Refugee Initiative (November 4, 2015 and March 1, 2016), refugee uptake was also calculated to exclude Syrian refugees. When excluding Syrian refugees, the official (using iCARE data) and estimated (using COA data) uptake for refugees was 23.4% and 64.9%, respectively.

Based on iCARE data, little variation in official uptake was found between genders. While official uptake for refugees was at 17% for males and females, it was slightly higher for non-refugee males (6%) than females (5%).

Official uptake for French-speaking^{Footnote 49} newcomers was lower than for non-French-speaking newcomers. Over the time period considered for the evaluation (i.e., PRs admitted to Canada between April 2015 and August 2017), 4% of the French-speaking immigrants admitted accessed at least one pre-arrival service. Uptake was even lower for non-refugees (2.9%), but higher for refugees (20%).

6.2. Promotion of Pre-Arrival Services

The previous evaluation of pre-arrival services noted that promotion of pre-arrival services to non-refugees had been a challenge prior to the expansion of pre-arrival services in 2015.^{Footnote 50}

Both interviewees and documents revealed that promotion of pre-arrival services to non-refugees continued to be a significant challenge for IRCC and SPOs beyond the expansion as there has not been a consistent process to inform clients of availability of services. For example, some Missions sent letters/brochures at Acknowledgement of Application Receipt stage, others at medical instructions stage or Confirmation of Permanent Residence stage; while some of these letters name pre-arrival SPOs directly while others do not. Interviewees that could speak to changes before and after the expansion of services noted that prior to the expansion, some IRCC Missions played an active role in helping pre-arrival SPOs recruit clients (e.g., sending monthly messages and reminders to prospective clients, participating in SPO promotional efforts), however since the expansion, given the number of additional pre-arrival SPOs, Missions have been less able to undertake these activities.

During interviews with SPOs, many expressed surprise and confusion following the expansion of service providers in 2015 upon learning that IRCC had not developed a standard process to inform prospective clients of pre-arrival services at the outset of the expansion. As a result, many SPOs developed their own strategies for promotion and outreach after projects launched (e.g., promoting services via social media, print media, or radio, forming partnerships with immigration consultants). In addition, many SPOs encountered differing levels of support and engagement from IRCC Missions. Despite the efforts of SPOs to promote pre-arrival services, most IRCC and SPO interviewees suggested that having SPOs take the lead on promotion is not optimal as they have less credibility than official messaging from IRCC, and, unlike IRCC, they do not have access to the contact information of newcomers which could be used to recruit participants.

As refugees are informed of Canadian Orientation Abroad as part of the travel booking process managed by the IOM, promotion was less of a challenge as compared to non-refugees. However both refugee survey respondents and focus group participants noted that apart from COA, they were unaware of other pre-arrival services that they could access, suggesting that promotion to refugees can be improved.

The newcomer surveys revealed that 71% of eligible newcomers surveyed were not aware of the existence of pre-arrival services.

• Of non-refugees that did not access pre-arrival settlement services, 79% stated they were unaware of their existence and 83% indicated they would have tried to access these services had they known about them.
• Of refugees that did not access pre-arrival settlement services, 64% stated they were unaware of their existence and 90% indicated they would have tried to access these services had they known about them.

IRCC took steps to improve promotion, however this was not until 2016 and evidence suggests that promotion continues to be an ongoing challenge.

In March 2016, a webpage listing pre-arrival service providers and the services available was launched. In June 2016, the Department also initiated work to inform clients of available services before they arrive in Canada in an effort to increase the number of newcomers accessing pre-arrival services. More specifically, the Department began informing some permanent resident applicants at the beginning of their application process of their eventual eligibility for pre-arrival services.^{Footnote 51}

In March 2017, IRCC updated its webpage listing all pre-arrival SPOs, and throughout 2017, IRCC also conducted several social media campaigns to promote the pre-arrival services website. An analysis of web analytics revealed that these social media campaigns were an effective means of increasing promotion as days in which social media posts were made generally resulted in an increase in web traffic to the pre-arrival services website. Despite the apparent success of promotional campaigns on increasing web traffic, it was not possible to determine their precise impact on increasing uptake. In addition, clients in focus groups confirmed that they were still confused about which services they should access and the availability of pre-arrival services in general.

In October 2017, IRCC implemented an automated process to send an invitation letter to all eligible newcomers when they receive a positive eligibility decision in their immigration application. While interviewees were hopeful that this process may help increase awareness of pre-arrival services, many suggested that this process may occur too late as by the time newcomers find out about pre-arrival services many do not have enough time to access them. Information from interviews, documents reviewed and newcomer surveys revealed that newcomers find out about pre-arrival services through various means at various stages of their application process.

For non-refugees, information from the survey indicated that:

• About half (53%) became aware of pre-arrival services through communication issued by IRCC (either by an email, letter or brochure provided by the Canadian Embassy, Consulate or High Commission (31%), or via IRCC’s website or social media (22%) while 22% became aware from a friend/acquaintance and 7% via an immigration consultant.
• Over half (56%) first heard about pre-arrival services after they were informed they had been accepted as a permanent residence, while 13% found out about the services after they applied, but before they were accepted as a permanent resident, and 15% knew about the services before they applied for permanent residency.

Close to half of non-refugee non-clients (47%) and 36% of non-refugee clients stated they would have liked to find out about pre-arrival services sooner in their immigration process. Interviewees suggested that an ideal time for non-refugees to access pre-arrival services would be two to three months before their departure.

Finally, some interviewees noted that immigration consultants may act as a barrier to newcomers finding out about pre-arrival services as they may be reluctant to inform their clients about the availability of free pre-arrival services provided by the Government of Canada as they may be charging newcomers for similar information and services.

6.3. Achievement of Targets

Each contribution agreement between IRCC and pre-arrival SPOs providing direct services to clients outlines targets for the number of clients served. As indicated in Table 10, the number of expected clients served per direct service SPO was variable, with just under half (48%) of SPOs expected to serve less than 500 clients per year and the remainder expected to serve 500 or more clients per year.

Table 10: Expected Annual Client Target, Pre-Arrival SPOs

Source: GCS

In FY2015/16 and FY2016/17, direct service pre-arrival SPOs served 33,324/80,975^{Footnote 52} (41%) of newcomers targeted to receive pre-arrival services. Only 2 SPOs (8%) met or exceeded targets within the first 2 fiscal years since the expansion (FY2015/16 and FY2016/17) while

• 12 SPOs (48%) served less than 25% of their initial target
• 8 SPOs (32%) served between 26-75% of their initial target
• 3 SPOs (12%) served over 75% but less than 100% of their initial target

No trends were observed between target size and achievement rate.

In addition to the challenges associated with promotion/client awareness of pre-arrival services (see Section 6.2), pre-arrival SPOs interviewed described other challenges in meeting targets:

• Except for pre-existing SPOs, initial targets were established with few or no benchmarks and limited information
• Delayed signing of contribution agreements for the majority of pre-arrival SPOs
• New pre-arrival SPOs offering online services needed to develop internal capacity (IT infrastructure and information material) prior to serving clients which impacted the number of months in operation and the number of clients served

In addition to failing to meet targets, these challenges have meant that the annual number of clients served by 27 SPOs has remained about the same as pre-expansion levels when there were only 3 IRCC-funded SPOs serving clients (from 19,455 clients in 2014/15 to 19,358 clients in 2016/17).

Given challenges with existing targets, in 2017, the global annual target for pre-arrival services (both in-person and online) was reduced by 7%, with individual SPOs annual targets being increased, decreased or held constant.^{Footnote 53} An analysis of administrative data for the partial fiscal year 2017/18 suggests that 19 SPOs (82%) are on pace to meet their revised targets for 2017/18. All four Francophone SPOs, in particular, have faced ongoing challenges in meeting their targets.

Of the few IRCC and SPOs interviewees that could comment on the 2 indirect SPOs funded by IRCC, most achieved their targets (i.e., expected tools and deliverables were completed), there was no strategy to ensure the continued use of the products. For example, while one indirect SPO completed its scheduled activities to develop a guide for the internationally-trained individuals seeking to work in the health sector, there was no formal plan put in place to ensure that this tool would be used beyond the life of the contribution agreement with IRCC.

6.4. Governance and Management

The 2012 IRCC evaluation of pre-arrival services^{Footnote 54} noted a lack of coordination and strategic direction to manage pre-arrival services and made several recommendations, including that IRCC clarify roles and responsibilities, establish a governance structure and establish a departmental-wide approach in place for the promotion of eligible participants. While IRCC agreed with the recommendations and took some actions to address them,^{Footnote 55} many interviewees noted that roles and responsibilities among IRCC and SPOs remain unclear and highlighted a lack of a coordinated approach to manage all pre-arrival services in unison. In particular, interviewees mentioned:

• A lack of clarity among IRCC and SPOs around who is responsible for promotion and how it should be carried out
• IRCC Missions were not aware of what role to play in supporting pre-arrival services (with some staff not aware of the existence of pre-arrival services) which has resulted in pre-arrival SPOs having to continuously invest time to build relationships with Mission staff
• A lack of a central function to promote coordination and partnerships across pre-arrival SPOs, domestic stakeholders and provinces and territories

Regarding the call for proposals for pre-arrival service providers, IRCC interviewees noted several challenges:

• Lack of thorough review process to ensure sufficient need for niche^{Footnote 56} pre-arrival services (e.g., construction workers in British Columbia, internationally-trained professionals moving to Calgary, etc.)
• Lack of a global strategy and review process to minimize overlap across the various pre-arrival projects (e.g., little consideration of how pre-arrival projects will complement each other)
• NHQ review of proposals but negotiations and management of many agreements were held in IRCC regions – this created challenges as most expertise around pre-arrival services held at NHQ

Beyond the call for proposals stage, during interviews, staff within IRCC noted challenges during the first few years following the expansion regarding SPOs that were not delivering and reporting pre-arrival activities as per their original contribution agreement. These interviewees attributed these difficulties to a lack of clear direction within IRCC which created misunderstandings around what pre-arrival services should include. As a result, IRCC had to work closely with several post-expansion SPOs to revise contribution agreements, expected activities and client targets.

In 2016, an internal Management Review was conducted of pre-arrival services and identified several implementation challenges:

• Delays in the implementation of contribution agreements
• Lack of a standardized referral process
• Data gaps, particularly regarding entering refugee client data into iCARE

As a response to these challenges, IRCC undertook a number of actions. The pre-arrival website was revised and a standardized invitation letter was issued to newcomers.^{Footnote 57} IRCC also renewed the majority of contribution agreements^{Footnote 58} until June 2018 to allow pre-arrival SPOs to have enough time to deliver services and address implementation challenges. IRCC also issued a number of operational bulletins and functional guidance, and established an internal Pre-Arrival Working Group to help respond to emerging management issues. While interviewees considered these actions useful, they were perceived to be reactive instead of proactive and many remained confused by roles and responsibilities. Nearly all interviewees suggested that IRCC take steps to strengthen governance of pre-arrival services by clarifying governance of pre-arrival services and roles and responsibilities of the various parties involved in the administration and management of pre-arrival services.

While recognizing the existence of the iCARE reporting system for pre-arrival services, key informants from IRCC noted a lack of ongoing performance measurement tools and metrics to effectively measure and report on the uptake of pre-arrival services, identify trends and address issues quickly, as they arise. These interviewees also mentioned concerns with underreporting, most acute for refugees as well as well as potential over reporting among a few particular SPOs. Limited monitoring coupled with the absence of standardized performance measurement tools specific to pre-arrival services make it difficult for IRCC to know the precise number of clients served by pre-arrival SPOs.

All IRCC missions and many staff at NHQ felt that IRCC missions should take a greater role in supporting NHQ by conducting limited monitoring and oversight of pre-arrival SPOs, citing the Visa Application Centre model as an ideal structure (i.e., clear monitoring guidelines and roles and responsibilities defined as well as supervised and unsupervised visits of providers).

While most SPO interviewees were satisfied with working relationships and mechanisms for communication to support management of the contribution agreements, some pre-arrival SPOs, especially those providing in-person services, explained that they would benefit from having more information on trends/analytics on immigration processing to support better resource planning.

6.5. Location of In-Person Pre-Arrival Settlement Services

Non-Refugees

As per initial funding guidelines for the pre-arrival services Call for Proposals in 2014, SPOs were advised to submit funding proposals to provide in-person pre-arrival services in locations to serve as many immigrants as possible as well as a few select countries for Francophone immigrants.^{Footnote 59}

In-person pre-arrival services for non-refugees were provided in 17 countries which collectively accounted for 64% of non-refugees admitted between April 1, 2015 and August 31, 2017.^{Footnote 60} In-person pre-arrival services are provided in 11 (55%) of the top 20 source countries but also in lower volume locations such as Lebanon, Colombia, Sri Lanka, Romania, Moldova and Taiwan.^{Footnote 61} Within the top 20 source countries, pre-arrival services are not offered in four English/French-speaking countries (i.e., United States, Jamaica, United Kingdom, France) and five non-English/French-speaking countries (i.e., Iran, Bangladesh, Algeria, Vietnam and Morocco).

Between April 1, 2015 and August 31, 2017, more than one pre-arrival SPO has offered in-person services in three high volume countries: China, India and the Philippines. Interviewees noted that while a few pre-arrival SPOs providing in-person services have formed partnerships with each other, many compete for clients in key markets. For example, in some cases, pre-arrival SPOs have travelled to conduct ad-hoc sessions in different countries where other IRCC-funded pre-arrival service providers already exist. Most interviewees viewed competition between in-person (and online) SPOs unfavourably and attributed it to a lack of a strategy from IRCC for deciding where to offer in-person services.

Refugees

As per initial funding guidelines for the pre-arrival services Call for Proposals in 2014, expected service delivery locations for refugees included 16 countries.^{Footnote 62}

As indicated in Table 11, between April 1, 2015 and August 31, 2017, the top source countries for refugees (i.e., country of citizenship for refugees admitted to Canada) are mostly served by the International Organization for Migration with the exception of the first wave of Syrian refugees (November 1, 2015 – March 31, 2016) who were not provided pre-arrival services due to logistical challenges.

Despite covering most locations where refugees are located, uptake was lower for certain nationalities. Among the top ten refugee nationalities, the lowest refugee uptake was observed for stateless citizens (3%) followed by citizens of Syria (32%), Somalia (50%), Afghanis (61%) and Ethiopians (61%).

Table 11: Top 10 Refugee Source Countries, Locations of In-Person Pre-Arrival Services and Estimated Uptake, April 1, 2015 – August 31, 2017

Source: GCMS and iCARE

6.6. Cost per Client for Pre-Arrival Services

In order to assess the cost per client to provide pre-arrival services, the evaluation examined the projected cost per client (total planned expenditures^{Footnote 63} / total projected clients) overall and by SPO as well as the actual cost per clients served (actual expenditures^{Footnote 64} / actual clients served). In addition, as a result of known underreporting for refugees, an estimate of adjusted cost per client was calculated by analyzing COA client records to account for the refugee clients not captured in iCARE.

Table 12: Overall Projected vs. Actual Cost per Client – Pre-Arrival SPOs (FY 2015/16 – FY 2017/18^{Footnote *†††})

Source: iCARE, GCS and SAP

As indicated in Table 12, the overall average projected cost per client for providing pre-arrival services was $686. Each contribution agreement was negotiated separately, which resulted in an array of projected costs per client, ranging from $219 to $7,512 per SPO.

Between 2015/16 and 2017/18,^{Footnote 65} the overall actual cost per clients admitted was $1,910,^{Footnote 66} with actual costs per client varying significantly by SPO, as the lowest and highest SPO cost per client was $48 and $27,779, respectively.

The cost per client variance was anticipated by the Department as the service offerings and methods of delivery varied across SPOs, and certain providers had the pre-existing infrastructures to ramp up service delivery more readily than others. In addition, significant factors (described in sections 6.2 and 6.3) affected the number of clients served, which as a result had an impact on the actual cost per client. Among these factors were challenges related to the lack of awareness among prospective clients, delays in signing contribution agreements for most SPOs, and the time needed by some SPOs to develop delivery capacity.

As displayed in Table 13, 22 of the 25 SPOs (88%) exceeded their projected cost per client, with 13 (52%) exceeding by over 100%.^{Footnote 67}

Table 13: Projected vs. Actual Cost per Client – Pre-Arrival SPOs (FY 2015/16 – FY 2017/18^{Footnote **})

Source: GCS, iCARE

When adjusting for the underreported clients not captured in iCARE between 2015/16 and 2017/18, the estimated adjusted cost per client for providing pre-arrival services was $1,281.

No clear trends were associated with cost per client and different SPO types (i.e., online vs. in-person SPOs, pre-expansion SPOs vs. post-expansion SPOs, SPOs providing settlement services in Canada vs. SPOs not providing settlement services in Canada, occupation-specific SPOs vs. non-occupation SPOs) as across and within each of these SPO types there was a wide range of cost per clients. For example, the cost per client for occupation-specific SPOs ranged from $592 per client to $9,276 and for non-occupation specific SPOs ranged from $48 to $15,831.

In February 2017, IRCC issued functional guidance to establish a standard benchmark cost per client for pre-arrival contribution agreement extensions. The guidance indicated that a typical cost per client for pre-arrival services should not exceed $2,000. Between April 1, 2015 and August 31, 2017, just over half of pre-arrival SPOs (13 SPOs or 52%) had exceeded this benchmark.

6.7. Efficacy of Delivery Models

Different SPO types (i.e., generalist SPOs, occupation-specific SPO, regional-specific SPOs, Francophone SPOs) as well as delivery aspects (e.g., language of service, online delivery vs in-person delivery) were assessed to determine whether there were specific differences attributable to each model/aspect in terms of client outcomes, costs and uptake. In addition, the evaluation was able to identify key factors for success and challenges associated with the different models.

In general, key factors for success identified from interviews and observations of SPOs with the lowest cost per clients for all delivery models included:

• having a longstanding history of providing settlement services or services to newcomers or well-established in their area of expertise
• having strong partnerships with other pre-arrival SPOs and domestic organizations or employers;
• strong engagement with partners overseas to recruit clients
• not having too specific of a clientele (e.g., while some regional SPOs and occupation-specific SPOs were successful in serving a large number of clients, SPOs that aimed to be both regional and occupation-specific experienced more difficulties)

In terms of specific delivery models, an analysis of key metrics (i.e., cost per client, # of clients served, client satisfaction) indicated that a higher proportion of generalist SPOs met these success criteria, when compared to other SPO types (i.e., occupation, regional or Francophone-specific).

Table 14: Pre-Arrival SPO Performance by SPO Types

Source: GCS, iCARE

In addition, the analysis identified the following success factors and challenges for various SPO types:

• Generalist SPOs: Of the SPOs that were successful in this category, all provided some form of in-person services.
• Occupation-specific SPOs: Of the SPOs that were successful in this category, all provided pre-arrival services for a broad range of occupations within specific industry sectors (e.g., information technology) and provided services tailored to clients destined nationally as opposed to being regional-specific.
• Regional-Specific/Francophone SPOs: No SPOs met all three criteria for success in either of these categories, which suggests that providing services only to specific regions or small populations of newcomers is more difficult to do effectively as the number of potential clients that are targeted be reached is lower than generalist or occupation-specific SPOs.

Few differences were found related to delivery models regarding clients’ perceived helpfulness of pre-arrival services and outcomes from the newcomer surveys.

Language of service

Nearly all (94%) of pre-arrival services clients were happy with the language in which services were delivered. When observing sessions, clients seemed more engaged when sessions were delivered in a local language – both refugee and non-refugee focus group participants confirmed they preferred receiving pre-arrival services in their native language as it allows for better understanding of the content communicated.

In addition, a service level analysis indicated that over 90% of services received by clients who indicated an official language preference of French for each of the types of services delivered (NARS, I&O, ER and CC) were not delivered in English (i.e., services were delivered in French or a non-official language).

Online vs. in-person

The evaluation found there is need for both online^{Footnote 68} and in-person services as many interviewees noted that in many countries, online delivery is not possible or very challenging due to technological issues (e.g., bandwidth and connectivity).

• Nearly all focus group participants and over half (54%) non-refugee clients surveyed indicated that they would prefer to access services in-person, while 42% indicated they would prefer receiving the services online and 4% over the telephone.
• The majority (69%) of non-refugees surveyed who had not taken any pre-arrival services indicated that if they had, they would have preferred to access the services online.

Among non-refugees surveyed, a greater proportion of clients who were served by SPOs delivering in-person services found pre-arrival services useful or very useful compared to those who did not receive services from SPOs providing in-person services (88% vs 80%).

Although pre-arrival services to refugees are typically provided in-person, many refugee focus group participants as well as those surveyed also indicated they would have like to receive some pre-arrival services online. Most (87%) of refugees who had not received pre-arrival services indicated that they would have tried to access pre-arrival services online, while 55% said they would have tried to access them in-person.

7. Conclusions and Recommendations

Pre-arrival services are needed and useful, as they help newcomers prepare for their settlement before they depart for Canada.

One of the primary expected outcomes from pre-arrival services is to provide relevant information and supports so that newcomers can make informed decisions about their new life in Canada and begin the settlement process before arriving. In this regard, all client groups found pre-arrival services to be useful and participants are gaining knowledge of life in Canada and the Canadian labour market at a higher rate than non-clients. For example, a greater share of pre-arrival clients (compared to non-clients) indicated having enough information on how to contact organizations that provide help in settling in Canada, how to have professional credentials and/or qualifications recognized, and to understand Canadian workplace culture and norms. Many pre-arrival clients are also taking actions as a result of pre-arrival services, such as looking for work, changing what they bring to Canada and deciding to further training/education to upgrade their skills. Finally, pre-arrival services have also been effective at providing referrals and linking clients with in-Canada settlement services, and these clients have been accessing IRCC-funded settlement services in Canada at a higher rate than non-users of pre-arrival services.

While clients find pre-arrival services useful, the majority of newcomers are not aware of their existence and uptake remains low for non-refugee immigrants. Ineffective promotion of these services, coupled with the absence of a comprehensive strategy to guide pre-arrival service expansion and a lack of clarity within IRCC regarding roles and responsibilities for the program delivery has resulted in a missed opportunity for the Department to positively impact more newcomers, and also in higher than expected per client costs.

Considerations for the Next Call for Proposals

Selecting the appropriate mix (i.e., number and type) of service providers, as part of upcoming calls for proposals is integral in ensuring pre-arrival services as a whole are efficient and effective in meeting expected outcomes. While the Department outlined criteria for eligible pre-arrival services and client groups in 2015, it did not apply a formal strategy to manage pre-arrival services in alignment with market demand and complementarity among different service providers. In addition, following the expansion of services providers in 2015, many service providers faced challenges in rolling out services and meeting targets.

There are opportunities for the Department to capitalize on best practices and lessons learned since the expansion.

Specifically, the evaluation identified the following considerations for the next Call for Proposals for pre-arrival services:

• Minimizing overlap across the various pre-arrival projects (i.e., the number of SPOs seeking to provide the same type of service to the same clients)
• Ensuring an appropriate balance between online and in-person services
• Providing funding for “niche” SPOs only when there is sufficient demand
• Defining assessment criteria to reflect key success factors (i.e., SPOs with experience/expertise, strong partnerships and engagement with stakeholders)
• Providing pre-arrival services in local languages, whenever possible
• Increasing awareness and provision of online service offerings for refugees, including employment-focused information and providing a more realistic picture of common challenges faced in Canada
• Allowing sufficient time for SPOs to implement new agreements and for developing capacity, tools and material to serve clients
• Providing a centralized intake process to assess client needs and then direct clients to tailored pre-arrival services based on their specific needs
• Promoting an expanded referral process to reach out to a wide range of domestic service providers
• Ensuring indirect services projects have a plan to be sustainable/relevant beyond funding of a contribution agreement

In addition to these considerations, the evaluation proposes five recommendations to improve the management and delivery of pre-arrival services.

Recommendations

IRCC funds a wide variety of pre-arrival SPOs that differ in size, targeted client groups, and delivery approaches. While there is a need to provide a mix of different services (i.e., generalist information, employment/occupation-specific information), the approach to service expansion has lacked a clearly defined strategy or framework, including a definition of how the various services were expected to be delivered, to complement one another and align with in-Canada settlement services. Developing a framework with a clear vision and definitions for pre-arrival services will help ensure a structured and streamlined approach to pre-arrival service delivery to meet client needs.

Recommendation 1: IRCC should develop a comprehensive program framework and guidance for pre-arrival services that provides a clear strategic direction for program delivery.

This framework should:

• Articulate the vision for IRCC pre-arrival services, including objectives and expected results
• Consider the appropriate mix of the various delivery models and approaches
• Consider the alignment of service offerings and delivery approaches with the differing profiles and needs of various client types
• Include a strategy to identify and prioritize the optimal locations for the delivery of in-person services
• Consider the cost of services and value for money

Despite efforts to expand the availability of pre-arrival services by increasing the number of SPOs in 2015, the absence of an effective promotion strategy from IRCC affected the reach and impact of pre-arrival services. Not only were a majority of newcomers not aware of the existence of pre-arrival services, SPOs were required to recruit participants and liaise with IRCC Missions. This impacted resources available to provide services, and overall uptake remained low and costs per client remained higher than initially projected. Although enhancements in 2017 were made to improve IRCC’s pre-arrival website and launched in fall, a standardized letter inviting participants to access pre-arrival services at the time at which they receive a positive eligibility determination on their immigration application, many newcomers remain unaware or confused by the various services available to them.

In addition, even when non-refugees had knowledge of pre-arrival services, the point at which they become aware often does not provide enough time for many to access services. As such, there are opportunities to actively inform or enroll prospective clients into pre-arrival services at an earlier stage.

Recommendation 2: IRCC should develop and implement a pre-arrival services promotion strategy to significantly increase awareness and uptake.

This strategy should:

• Outline the key activities and guidance needed to improve awareness and increase program participation
• Clarify the roles and responsibilities for IRCC (including Missions abroad) and SPOs with respect to promotion
• Consider earlier opportunities for informing potential clients to help ensure they have sufficient time to access services

The rapid expansion of partners and stakeholders, coupled with a lack of clarity related to roles and responsibilities made it difficult to ensure a coherent approach to management of pre-arrival services. For example, the responsibility for promotion was unclear and lacked coordination: some pre-arrival SPOs actively promoted their services while others expected the Department to lead this. Further, as IRCC Missions have had no formal official role regarding pre-arrival services since the expansion of services, there has not been a coherent and consistent approach to promoting pre-arrival services and dealing with in-person SPOs. There are opportunities to better leverage IRCC Missions as they are well- positioned due to their proximity to newcomers and in-person SPOs.

Clarifying roles and responsibilities within the Department through more formalized governance would increase the coordination around pre-arrival services.

Recommendation 3: IRCC should clarify and strengthen its governance to lead and coordinate pre-arrival services by:

• Establishing clear roles and responsibilities among internal IRCC stakeholders, including across NHQ, regions and International Network
• Clarifying the role of Missions in the delivery and monitoring of in-person pre-arrival services and SPOs

Pre-arrival SPOs have largely been left to determine the appropriate mix of services and to develop partnerships with other pre-arrival SPOs and domestic settlement service providers. While many have undertaken partnerships and established networks of cross-referrals on their own, there are opportunities for IRCC to take a stronger leadership role to ensure pre-arrival SPOs are connected to one another and to IRCC’s domestic Settlement network, which would ensure more seamless pathways and efficient delivery for pre-arrival clients.

Recommendation 4: IRCC should establish a mechanism to promote collaboration, cross-referrals and sharing of best practices among pre-arrival SPOs and pre-arrival and domestic settlement SPOs.

A lack of standardized performance measurement tools and challenges in data collection have affected the ability of the Department to effectively report on pre-arrival services results in an in-depth manner and identify trends and address issues quickly, as they arise. For example, there are known issues of under-reporting, most acute for refugee clients, but also affecting other pre-arrival clients. A strengthened approach to performance measurement and results reporting for pre-arrival services will allow the Department better collect and use performance information, monitor and report program outcomes regularly, and inform program changes and directions.

Recommendation 5: IRCC should strengthen performance measurement and reporting for pre-arrival services, by:

• Developing key indicators and data strategies to support collection of performance information
• Considering developing a targeted Performance Information Profile for pre-arrival services which aligns with IRCC’s Settlement Program and Resettlement Program Performance Information Profiles

Appendix A: Detailed List of in-Person Pre-Arrival Service Locations (FY2015/16 – FY 2017/18)

North and South America

Europe

Africa

Asia

Source: Pre-Arrival Service Provider reports

Appendix B: Detailed Profiles

Table B1: Profile of Pre-Arrival Clients and Non-Clients Admitted to Canada between April 1, 2015 – August 31, 2017

Source: iCARE

Table B2: Profile of IRCC-funded Domestic Settlement Service Clients Admitted to Canada between April 1, 2015 – August 31, 2017

Source: iCARE

Appendix C: Detailed List of Pre-Arrival SPOs (FY2015/16 – FY 2017/18)

Source: GCS

Appendix D: IRCC Settlement Program Logic Model

Appendix E: List of Evaluation Questions

Relevance

1. Is there a continued need to provide Pre-Arrival Settlement Services to newcomers destined to Canada?

Design and Implementation

2. How effective are current tools and mechanisms at reaching potential participants and in promoting Pre-Arrival Settlement Services?
3. How effective are IRCC governance structures regarding Pre-Arrival Settlement Services? Are they appropriate?
4. Are in-person Pre-Arrival Settlement Services being offered in the appropriate locations?
5. To what extent is information provided through Pre-Arrival Settlement Services aligned with the needs of specific sub-populations or groups of newcomers?

Performance

6. To what extent is the information provided by Pre-Arrival Settlement Services appropriate, timely, and useful to clients?
7. To what extent are Pre-Arrival Settlement clients achieving expected settlement and integration outcomes? To what extent have Pre-Arrival Settlement Services contributed to achieving these outcomes?
8. To what extent do Pre-Arrival Settlement Services support partnerships and linkages to Settlement and other services to support newcomer settlement and integration, both overseas and in-Canada?
9. How efficient and effective are the various delivery models (i.e., in-person or web-based) for providing Pre-Arrival Settlement Services to newcomers?

Appendix F: Key Survey Frequency Tables

Table F1: Non-Refugees - Please indicate your level of agreement with the following statements

Table F2: Non-Refugees - Please indicate whether or not you had enough information on any of the following topics prior to coming to Canada

Table F3: Refugee clients - Please rate your level of agreement with the following statements

Table F4: Non-Refugees - During your first three months in Canada…