ARCHIVED – Evaluation of the Settlement Program

Research and Evaluation Branch
Immigration, Refugees and Citizenship Canada
November 2017

Technical Appendices are available upon request to Research-Recherche@cic.gc.ca.
Ci4-175/2018E-PDF
978-0-660-24707-6
Reference Number: E2-2016

On this page

• List of acronyms
• Executive summary
• Evaluation of the Settlement Program – Management Response Action Plan
• 1. Introduction
  • 1.1. Purpose of the Evaluation
  • 1.2. Program Background and Context
    • 1.2.1. IRCC Settlement Programming and Clients
    • 1.2.2. Eligibility and Client Uptake
    • 1.2.3. Other Stakeholders and Partners
    • 1.2.4. Overview of Settlement Programs in Other Countries
• 2. Settlement Program Profile
  • 2.1. Settlement Program Clients
  • 2.2. Settlement Program Services
  • 2.3. Settlement Program Service Provider Organizations
• 3. Methodology
  • 3.1. Evaluation Approach
  • 3.2. Evaluation Scope
  • 3.3. Data Collection Methods
  • 3.4. Considerations and Limitations
• 4. Relevance
  • 4.1. Continued Need and Relevance for Settlement Services
  • 4.2. Alignment with Federal Roles and Responsibilities
  • 4.3. Settlement as a Government of Canada and Departmental Priority
• 5. Achievement of Expected Client Outcomes
  • 5.1. Challenges in Identifying Impacts of the Settlement Services on Client Outcomes
  • 5.2. Clients Receive Appropriate Information and Services
  • 5.3. Clients Attain Awareness of Community Resources
    • 5.3.1. Clients Attain Awareness of Community and Other Resources
  • 5.4. Clients Gain Knowledge of Life in Canada, Including Laws, Rights and Responsibilities
    • 5.4.1. Clients Gain Knowledge of Life in Canada
    • 5.4.2. Clients Gain Knowledge of Laws, Rights and Responsibilities
  • 5.5. Clients Learn Official Language Skills
    • 5.5.1. Approaches to Assessing Language Skills Improvement
    • 5.5.2. Language Progression
    • 5.5.3. Language Skills Improvement
  • 5.6. Clients’ Use of Official Languages
  • 5.7. Clients Acquire Knowledge about Working in Canada and Participate in Canadian Labour Market
    • 5.7.1. Clients Acquire Knowledge about Working in Canada
    • 5.7.2. Clients Participate in the Canadian Labour Market
  • 5.8. Connections to Communities and Public Institutions
    • 5.8.1. Clients Have Connections to Communities and to Public Institutions
  • 5.9. Clients Participate in the Broader Community
  • 5.10. Making Informed Decisions about Life in Canada
  • 5.11. Impact of Taking Multiple Settlement Services
• 6. Service Delivery Enablers – Support Services and Local Immigration Partnerships
  • 6.1. Impact of Support Services
  • 6.2. Impact of Local Immigration Partnerships (LIPs)
• 7. Program Management
  • 7.1. Reach of IRCC-funded Settlement Services
  • 7.2. Governance and Program Delivery
  • 7.3. SPO Partnerships and Coordination within Communities
  • 7.4. Performance Reporting
  • 7.5. Gender Differences
  • 7.6. Cost of Providing Settlement Services
• 8. Conclusions and Recommendations
• Appendices

List of tables

• Table 1: Proportion of Adult Permanent Residents admitted in 2015 who used at least one Settlement Service (January 2015 - April 2017)
• Table 2: Settlement Program Service Profile Highlights (Adults), FY 2016/17
• Table 3: Language Assessment Results (CLB/NCLC) of Settlement Clients who were assessed in FY 2015/16 (First Language Assessment)
• Table 4: Number of IRCC-funded Settlement Service Locations by Province, FY 2014/15-FY 2015/16
• Table 5: Clarity of Guidance Released by IRCC On How to Report On Settlement and Support Services, By Type of Service
• Table 6: Contributions for the Settlement Program, FY 2011/12 - FY 2015/16
• Table 7: Settlement Program Contributions per Client and By Immigration Category, FY 2013/14 - FY 2015/16
• Table 8: Cost of Historical Settlement Utilization, FY 2014/15
• Table 9: Government of Canada Costs and IRCC FTEs to Administer the Settlement Program, FY 2015/16

List of acronyms

Executive summary

This report presents the findings of the evaluation of Immigration, Refugee and Citizenship Canada’s (IRCC) Settlement Program. The evaluation was conducted in fulfillment of requirements under the Treasury Board Policy on Results and section 42.1 of the Financial Administration Act and covered the period of fiscal (FY) years 2011/12 - 2016/17.

Overview of the Settlement Program

IRCC’s Settlement Program aims to support newcomers’ successful settlement and integration so that they may participate and contribute in various aspects of Canadian life.^{Footnote 1} Through contribution agreements, IRCC funds service provider organizations (SPO), such as immigrant-serving agencies, social service organizations or educational institutions to provide settlement services to newcomers under six main areas (Needs Assessments and Referrals, Information and Orientation, Language Assessments, Language Training, Employment-Related Services and Community Connections). The Settlement Program also funds six support services (i.e., Care for Newcomer Children, Transportation, Translation, Interpretation, Disability Support and Crisis Counselling) to help facilitate access to settlement programming as well as Indirect Services that support the development of partnerships, capacity-building and the sharing of best practices among Settlement service providers. Furthermore, as an essential part of the Settlement Program, IRCC engages provinces/territories, civil society, businesses and other stakeholders such as employer associations, sector councils, and credentialing bodies to mobilize resources to deliver responsive and coordinated settlement and non-settlement services to newcomers.

Summary of Conclusions and Recommendations

The Settlement Program has been effective at meeting a growing demand as service usage has risen over recent years, providing settlement services to more and more newcomers along their integration path. In addition, Settlement services coverage is widespread, with IRCC-funded SPOs offering permanent and itinerant Settlement services in all provinces and territories, both rural and urban centres within IRCC’s jurisdiction.

Overall, the main expected outcomes for the program are being met. Clients are having their settlement needs assessed, receiving referrals to appropriate services, and are increasing knowledge and skills to help them integrate into Canadian society. Specifically, the evaluation found that the majority of settlement clients are: gaining knowledge about life in Canada, improving their language ability, using official languages, acquiring knowledge about working in Canada, finding employment, forming connections to communities and public institutions, participating in the broader community and making informed decisions. Finally, the evaluation found that the management of the Settlement Program effectively supports program delivery.

The evaluation was able to assess the unique impact of Settlement Services on achieving client outcomes and found that each of the specific Settlement Program streams contributed to different client outcomes with varying degrees. Employment-related Services were found to positively impact the most client outcomes, including several beyond gaining knowledge of the Canadian work environment, whereas taking IRCC-funded Language Training, Information and Orientation, or Community Connections impacted fewer expected client outcomes.

The evaluation also found that the impact of Settlement Services was affected by client type, as, in general, clients with higher human capital (i.e., economic immigrants, those with a university degree or those with reported knowledge of English and/or French upon admission) derived greater benefits from IRCC-funded Settlement Services compared to other clients without these characteristics. This points to the need to ensure the proper balance among program components and to tie outcomes more specifically to the profile of different client groups.

The evaluation also found that IRCC-funded Local Immigration Partnerships has broadened community collaboration on, and profile of, newcomer issues by effectively engaging non-traditional newcomer service providers. Finally, the provision of six support services were found to be integral for ensuring that newcomers are able to access IRCC-funded settlement services.

Some areas for program improvements have been identified, and as such, this evaluation report proposes the following recommendations.

Language Training

Language training is helping newcomers improve their language skill, but not in the same way for all clients and not all skills (i.e., reading, writing, listening, speaking). Examining in more depth Settlement clients’ language learning needs, motivations and objectives, as well as the effectiveness of different language training delivery models used by service providers, will allow the Department to better determine what works for who and under what conditions and adjust its language training accordingly.

Recommendation 1

IRCC should review and assess its language training delivery and implement appropriate changes to improve its effectiveness. The assessment should:

• consider the needs of different groups of learners, and respective determinants of success;
• build on the strengths and weaknesses of existing approaches (curricula, modes of training, etc.);
• leverage best practices from adult education theory and practice, and the field of teaching English and French as a second language to adults; and,
• consider new and innovative approaches to language training for adult immigrants.

Employment-Related Services

The evaluation showed that Employment-Related Services have the most widespread positive impact on client outcomes, including, among others, improved language skills and use, gaining knowledge of the Canadian work environment, and helping clients learn more about life in Canada.

Recommendation 2

IRCC should develop and implement a plan to optimize the benefits of its Employment-Related Services and employment-specific language training.

Prioritization of services

Client needs and settlement objectives vary greatly, and the evaluation found that clients from different immigration categories and socio-demographic characteristics were taking differing amounts of time to achieve similar results. As such, there should be consideration given to exploring the prioritization of and access to services for clients while balancing needs and costs.

Recommendation 3

IRCC should review access to and duration of Settlement services and implement appropriate changes that achieve a balance between meeting the specific needs of different clients and available resources.

Support Services

Overall, there is a need for Support Services, as they enable clients to access the Settlement services necessary to assist in their integration journey. Some newcomers also need certain support services to access community or mainstream (non-Settlement Program) services that are not funded by IRCC and there are challenges with providing support for mental health issues through crisis counselling and in addressing clients’ support service needs beyond what IRCC funds. Finally, due to limited tracking, it is difficult to quantify the full cost of providing support services.

Recommendation 4

IRCC should clarify the Department’s expectations regarding the provision, use and reporting of Support Services. Specifically, the Department should review and clarify:

• the approach to the use of specific Support Services to access community services not funded by IRCC, and update and promote guidance as appropriate;
• the expectations regarding the provision of crisis counselling and clearly articulate what SPOs should be providing to clients as part of this particular Support Service; and,
• the financial and reporting requirements regarding Support Services to determine what needs to be tracked, and subsequently develop and issue new guidance and adjust financial reporting procedures as needed.

Local Immigration Partnerships

LIPs have made notable achievements, particularly in the areas of locally-relevant research, information sharing, partnership-building, and strategic planning. However, the current funding criteria and structure for LIPs may not be the appropriate model for certain communities. Challenges for LIPs in securing sustainable funding pose a risk to implementing strategic plan activities and projects. Finally, there is room for greater sharing of information between them and other LIPs, as well as IRCC.

Recommendation 5

IRCC should develop and implement a strategic plan to make best use of the potential contributions of Local Immigration Partnerships (LIP) to settlement and integration outcomes of immigrants and refugees. This plan should at the least:

• articulate its vision, expectations, rationale and role for supporting LIPs;
• confirm and communicate the criteria for funding new and existing LIPs; and,
• facilitate networking and knowledge sharing among LIPs by leveraging and disseminating research results and best practices generated by LIPs.

Data Systems

There are opportunities to examine the various data systems as a whole to ensure they work well together by allowing the department and external stakeholders to obtain the necessary information needed to support decision making while reducing reporting burden. In addition, repeating the client outcome survey developed as part of the evaluation and also administering it to a comparison group of non-users would allow the Department to augment its capacity to measure client outcomes.

Recommendation 6

IRCC should review its reporting requirements and systems for the Settlement Program and implement changes to streamline data collection and enhance analytical tools as required.

Recommendation 7

To strengthen its client outcomes data, IRCC should administer an outcomes survey on an ongoing basis to settlement users and non-users.

Evaluation of the Settlement Program – Management Response Action Plan

IRCC agrees with the recommendations and this Action Plan will be used to inform and guide Settlement Program development. The action items will support future program improvements and reflect consultations, are well aligned with Program directions and ongoing initiatives, and will also serve to inform the next Call for Proposals process.

Recommendation 1

IRCC should review and assess its language training delivery and implement appropriate changes to improve its effectiveness. The assessment should consider:

• the needs of different groups of learners and respective determinants of success;
• the strengths and weaknesses of existing approaches (curricula, modes of training, etc.)
• best practices from adult education theory and practice, and the field of teaching English and French as a second language to adults; and,
• new and innovative approaches to language training for adult immigrants.

Response #1

IRCC agrees with this recommendation.

Through the Settlement Program, the Department strives to help newcomers access the language training services that are most suited to their settlement needs, and as quickly as possible.

The Department will build on the evaluation findings as it continues its review of settlement language programming, with a focus on different clients’ needs, program parameters and best practices, testing new approaches and innovative delivery models.

The action items will consider and align with employment-related services and employment-specific language training, outlined for Recommendation 2.

IRCC recognizes the need for close cooperation with provincial and territorial ministries delivering English and French as a second language programming for adult immigrants and in consultation with key settlement language experts, academics, and stakeholders.

As a key action under the Government’s existing Federal-Provincial-Territorial (FPT) Vision Action Plan for Immigration, the Department is working together with all provinces and territories to advance a Pan-Canadian Language Strategy to improve coordination and efficiency in the design and delivery of language programming for adult immigrants. Enhancements are anticipated in the areas of employment-related language training better aligned to local labour market needs, clearer pathways for adult literacy learners, and better access to technology enhanced learning tools. These action items support Recommendation 1.

Action #1a

Complete a targeted evaluation of IRCC-funded Language Training provide recommendations to further improve language training delivery and effectiveness.

Accountability: Research & Evaluation Branch. Support: Settlement and Integration Policy Branch, Settlement Network Completion Date: Q3 2018/19.

Action #1b

Sign agreements for pilot projects to test the effectiveness of alternative models and explore innovative approaches for language training delivery.

Accountability: Settlement and Integration Policy Branch. Support: Settlement Network Completion Date: Q1 2018/19.

Action #1c

Identify any changes to language programming based on the results of the Language Training evaluation and the pilot project findings.

Accountability: Settlement and Integration Policy Branch. Support: Settlement Network, Research & Evaluation Branch Completion Date: Q4 2019/20.

Action #1d

Begin implementation of any changes to language training design and delivery in partnership with provinces and territories (PTs), service provider organizations, and other partners.

Accountability: Settlement and Integration Policy Branch. Support: Settlement Network, Provinces/Territories Completion Date: Q1 2020/21.

Recommendation 2

IRCC should develop and implement a plan to optimize the benefits of its Employment-Related Services and employment-specific language training.

Response #2

IRCC agrees with this recommendation.

The Department recognizes that while newcomers are an important contributors to the Canadian labour market, they continue to face individual and systemic barriers to integrating into the labour force. As such, IRCC is currently developing a policy diagnostic of labour market access support for newcomers, with a view to develop a federal strategy for newcomers’ employment to refine IRCC-funded Employment-related Services. The strategy will increase alignment with Employment and Social Development Canada (ESDC) and provincial/territorial labour market supports for a horizontal approach to better serve newcomers who are seeking employment.

The Department recognizes the potential benefits and positive impact of employment-specific language training on the economic outcomes of newcomers, and considers this to be a key area of focus in designing program enhancements. As outlined in the response to Recommendation 1, IRCC, in partnership with provinces and territories, and in consultation with key stakeholders, will enhance newcomers’ awareness of and access to the settlement services they need, including employment-specific language training.

Action #2a

Building on the settlement evaluation findings, complete a policy diagnostic of labour market program and services for newcomers, including IRCC-funded Employment-related Services, informed by key stakeholder consultations.

Accountability: Settlement and Integration Policy Branch. Support: Settlement Network Completion Date: Q4 2017/18.

Action #2b

Launch federal Newcomers’ Employment Strategy, including an implementation plan that will optimize the benefits of IRCC’s Employment-Related Services and leverage available supports, such as those provided by ESDC and Provinces/Territories.

The federal Newcomer Employment Strategy will complement the FPT Pan-Canadian Language Strategy.

Accountability: Settlement and Integration Policy Branch. Support: Settlement Network Completion Date: Q2 2018/19.

Action #2c

Sign agreements for pilot projects to test the effectiveness of alternative models of employment-specific language training.

Accountability: Settlement and Integration Policy Branch. Support: Settlement Network Completion Date: Q1 2018/19.

Action #2d

Identify best practices based on project outcomes.

Accountability: Settlement and Integration Policy Branch. Support: Settlement Network Completion Date: Q4 2019/20.

Action #2e

Based on project outcomes, implement changes to employment-specific language training models.

Accountability: Settlement and Integration Policy Branch. Support: Settlement Network Completion Date: Q1 2020/21.

Recommendation 3

IRCC should review access to and duration of Settlement services and implement appropriate changes that achieve a balance between meeting the specific needs of different clients and available resources.

Response #3

IRCC agrees with this recommendation.

In the context of the Department’s development of a new vision for settlement and integration, the Settlement Program model will be considered with a view to strengthening the Program’s ability to meet the diverse needs of different clients while maintaining cost effectiveness. Costing and modelling will be part of the analysis and options could involve program design changes and/or require new policy authority.

Engagement with provinces and territories, the settlement sector and other federal government departments are key to supporting this work.

Action #3a

Analyze evidence relating to client needs and access to settlement services, and costing to inform prioritization of services, and develop options.

Accountability: Settlement and Integration Policy Branch. Support: Finance, Strategic Policy and Planning, Settlement Network, Research & Evaluation Completion Date: Q2 2018/19.

Action #3b

Contingent on decisions made implement any changes to program design, in partnership with Provinces/Territories and the settlement sector.

Accountability: Settlement and Integration Policy Branch. Support: Settlement Network, Research & Evaluation, Settlement Sector, Provinces/Territories Completion Date: Q4 2018/19.

Recommendation 4

IRCC should clarify the Department’s expectations regarding the provision, use and reporting of Support Services. Specifically, the Department should review and clarify:

• the approach to the use of certain Support Services to access community services not funded by IRCC, and update and promote guidance as appropriate.
• the expectations regarding the provision of crisis counselling and clearly articulate what SPOs should be providing to clients as part of this particular Support Service; and,
• the financial and reporting requirements regarding Support Services to determine what needs to be tracked, and subsequently develop and issue new guidance and adjust financial reporting procedures as needed.

Response #4

IRCC agrees with this recommendation.

The establishment of a Support Services Working Group, to undertake analysis of existing policy and program guidance will require engagement with the settlement sector, regional officers and possibly provinces and territories to assess the impact of any policy changes.

With regard to sub-recommendation c), while the Department is prepared to review the financial and reporting requirements related to support services, any potential changes in these areas will need to be considered in the context of the reporting burden on IRCC’s Contribution Agreement (CA) holders, potential cost associated with changes to the Grants and Contribution System (GCS).

Following the development of the comprehensive Policy Guidance Document for Settlement Support Services, IRCC will update program guidance (e.g. Funding Guidelines, Negotiation Guidelines, iCARE policy documentation) for Settlement Officers and Service Provider Organizations (SPOs) on the scope and definition of IRCC Support Services as well as financial and reporting requirements.

Action #4a

Develop comprehensive Policy Guidance Document for Settlement Support Services.

Accountability: Settlement and Integration Policy Branch. Support: Settlement Network, Research & Evaluation, Finance, Medical Health Branch, Communications, Settlement Sector Completion Date: Q1 2018/19.

Action #4b

Develop and update related program materials (e.g. Funding Guidelines, Negotiation Guidelines, iCARE policy documentation and functional guidance.)

Accountability: Settlement Network. Support: Settlement and Integration Policy Branch, Research & Evaluation, Finance, Settlement Sector, Completion Date: Q2 2018/19.

Action #4c

Disseminate new program guidance to IRCC officers via information sessions and the settlement sector via the Settlement Network and National Settlement Council.

Accountability: Settlement Network. Support: Settlement and Integration Policy Branch, Research & Evaluation Completion Date: Q2 2018/19.

Action #4d

Identify scope of potential financial and reporting system changes and develop proposal accordingly.

Accountability: Settlement Network. Support: Finance, Settlement and Integration Policy Branch Completion Date: Q2 2018/19.

Recommendation 5

IRCC should develop and implement a strategic plan to make best use of the potential contributions of Local Immigration Partnerships (LIP) to settlement and integration outcomes of immigrants and refugees. This plan should at the least:

• Articulate its vision, expectations, rationale and role for supporting LIPs
• Confirm and communicate the criteria for funding new and existing LIPs; and,
• Facilitate networking and knowledge sharing among LIPs by leveraging and disseminating research results and best practices generated by LIPs.

Response #5

IRCC agrees with this recommendation.

The Department acknowledges the continuing need for and value of cross-sectoral participation and planning in support of newcomer integration, as well as the potential LIPs have for community coordination, engagement and information sharing.

A strategic plan for LIPs will bring greater clarity concerning their scope and expected role, and it will consider possible ways IRCC could better support the LIPs’ action plan implementation. Development of the plan will include a review of funding criteria and guidance and consideration of the IRCC role in support of LIPs.

Action #5a

Develop a strategic plan addressing the rationale, enhanced role in support of client outcomes, funding criteria, and expectations for LIPs.

Accountability: Settlement and Integration Policy Branch. Support: Settlement Network, Provinces/Territories, Municipalities, Local Immigration Partnerships Completion Date: Q4 2017/18.

Action #5b

Implement the LIPs strategic plan.

Accountability: Settlement Network. Support: Settlement and Integration Policy Branch Completion Date: Q2 2018/19.

Action #5c

As part of the strategic plan, review existing guidance documents for LIPs, including the funding criteria for new and existing LIPs, and update as required.

Accountability: Settlement and Integration Policy Branch. Support: Settlement Network Completion Date: Q4 2017/18.

Recommendation 6

IRCC should review its reporting requirements and systems for the Settlement Program and implement changes to streamline data collection and enhance analytical tools as required.

Response #6

IRCC agrees with this recommendation.

Over the last several years, IRCC invested heavily in development of tools and systems to enable data collection about the Settlement Program and from service provider organizations, including comprehensive client information, service usage, and financial information.

As tools were developed and rolled out and as the demand for timely, clear, accessible results data increases, it is an opportune moment to review the information collected to improve relevance, consistency and system/tool coherence – ranging from iCARE, APPR, narrative reports, GCS, and others as required. A strengthened approach to reporting will involve consideration of an integrated systems vision, including but not limited to iCARE and GCS.

Furthermore, data access is crucial and necessary to support departmental policy decisions, performance measurement and planning. Improving analytical capacity and access to data for internal users is key to ensure timely evidence is available to IRCC. As the Settlement Program continues to evolve it will be important to review and monitor its relevance on an on-going basis.

Action #6a

Finalize and implement iCARE analytical tools (i.e., data cube, complete module updates).

Accountability: Research & Evaluation Branch. Support: Settlement and Integration Policy Branch, Settlement Network, Strategic Policy and Planning, Solutions and Information Management Branch Completion Date: Phase 1 – Q3 2017/18 Phase 2 – Q4 2017/18.

Action #6b

Establish an integrated vision for iCARE-GCS-SAP and other systems and develop a business case to support the vision.

Accountability: Settlement and Information Sector, Research & Evaluation Branch. Support: Solutions and Information Management Branch, Finance Date: Q2 2017/18.

Action #6c

Complete iCARE-IMDB data linkage.

Accountability: Research & Evaluation. Support: Settlement and Integration Policy Branch, Settlement Network, Solutions and Information Management Branch Completion Date: Q4 2017/18.

Action #6d

Review and identify policy and program analytical, research and performance information needs.

Accountability: Settlement and Integration Policy Branch. Support: Research & Evaluation Branch, Settlement Network, Solutions and Information Management Branch, Grants and Contributions Financial Management, Strategic Policy and Planning Completion Date: Q4 2017/18.

Action #6e

Develop options and implement solutions for effective and timely reporting, analysis and research in support of the Settlement & Integration Sector.

Accountability: Settlement and Integration Policy Branch. Support: Research & Evaluation Branch, Settlement Network, Solutions and Information Management Branch, Grants and Contributions Financial Management, Strategic Policy and Planning Completion Date: Q2 2018/19.

Recommendation 7

To strengthen its client outcomes data, IRCC should administer an outcomes survey on an ongoing basis to settlement users and non-users.

Response #7

IRCC agrees with this recommendation.

The Department recognizes the importance of monitoring the Settlement Program, particularly client outcomes, highlighting the need for a regular outcomes survey. While evaluations, including the recent settlement evaluation with the Settlement Program Client Outcome Survey, enabled the Department to answer many client outcome questions, it is necessary that the client survey continue in order to inform performance measurement at the Departmental, program and project levels.

To augment its Settlement results information, IRCC also acknowledges the need for comparable results, which can be best obtained through analysis of ‘non-federal settlement clients’. This would further strengthen the ability of the Department to assess its contribution and attribution to settlement and integration outcomes. Administering a survey to this population would allow a deeper understanding of the fuller impacts of the Settlement program.

The results of these surveys will be used to inform policy analysis and program improvements.

Action #7a

Review and amend as required the current Client Outcomes Survey, including a non-client version.

Accountability: Research & Evaluation Branch. Support: Settlement and Integration Policy Branch, Settlement Network, Solutions and Information Management Branch Completion Date: Q4 2017/18.

Action #7b

Administer outcomes survey to Settlement clients and non-federal Settlement clients and analyze the results.

Accountability: Research & Evaluation Branch. Support: Settlement and Integration Policy Branch, Settlement Network, Solutions and Information Management Branch Completion Date: Q4 2017/18.

1. Introduction

1.1. Purpose of the Evaluation

This report presents the findings of the evaluation of Immigration, Refugee and Citizenship Canada’s (IRCC) Settlement Program for fiscal years (FY) 2011/12 to 2015/16. The evaluation was conducted in fulfillment of requirements under the Treasury Board Policy on Results and section 42.1 of the Financial Administration Act.^{Footnote 2}

As the Settlement Program is IRCC’s largest grant and contribution program, the scope of this evaluation contained in-depth analyses using multiple lines of evidence, focusing primarily on analysis and reporting according to the expected outcomes of the Program. The evaluation also assessed other elements such as the management of the Settlement Program, the provision of support services (i.e., care for newcomer children, transportation assistance, translation, interpretation, support for persons with disabilities and crisis counselling), and Local Immigration Partnerships.

1.2. Program Background and Context

IRCC’s Settlement Program aims to support newcomers’ settlement and integration, so that they may fully participate and contribute in various aspects of Canadian life. In this context, settlement refers to a short period of mutual adaptation between newcomers and the host society, during which the government provides support and services to newcomers, while integration is a two-way process for immigrants to adapt to life in Canada and for Canada to welcome and adapt to the newcomers. Ultimately, the goal of integration is to encourage newcomers to be fully engaged in the economic, social, political, and cultural life of Canada.^{Footnote 3}

1.2.1. IRCC Settlement Programming and Clients

Settlement services are delivered by organizations funded by IRCC to provide direct service delivery programming (i.e., services provided directly to newcomers and support services), and indirect services.^{Footnote 4} Projects are typically funded for three years following a national call for proposals process. Since FY 2014/15, IRCC has been responsible for the management of settlement services in all provinces and territories outside Quebec.^{Footnote 5}

Three type of services are offered.

1) Direct Service Delivery

Through contribution agreements (CA), IRCC funds service provider organizations (SPO) such as immigrant-serving agencies, social service organizations or educational institutions to provide the following types of services to newcomers. SPOs provide direct services in both permanent and itinerant locations.^{Footnote 6}

Needs Assessments and Referrals (NARS)

NARS are conducted to assess newcomers’ needs and link them to appropriate settlement and community-based services.

Information and Orientation services (I&O)

I&O services are offered to newcomers to provide relevant, accurate, consistent, and timely settlement-related information and orientation that is needed to make informed settlement decisions, as well as promoting an understanding of life in Canada. Examples of I&O include orientation sessions, general labour market information groups, general life skills development activities, etc.

Language Assessments (LA)

Assessments are conducted using a Canadian Language Benchmark (CLB) or Niveau de competence linguistique canadiens (NCLC) based tools to determine the official language ability of newcomers for placement in language training programs.

Language Training (LT)

Language training aims to provide adult newcomers with settlement content-based language instruction so that they may acquire English or French language skills they need to contribute to the Canadian economy and integrate into their communities. LT can be offered in a classroom setting, online, or in a blended environment, etc.

Employment-Related Services (ER)

Aims to equip newcomers with the skills, connections and support needed to enter into the labour market and contribute to the economy. Examples of ER include work placements, resume screening, employment networking, etc.

Community Connections (CC)

Includes activities to support the two-way process of integration and facilitate adaptation on the part of newcomers and their host communities. Examples of CC include peer support groups, community-based group events, conversation circles, matching newcomers with Canadians, cultural visits, field trips, etc.

2) Support Services

In order to help address barriers newcomers face in accessing settlement programing, IRCC funds six types of support services on a limited basis: Care for Newcomer Children, Translation, Transportation, Interpretation, Disability Support, and Crisis Counselling. Support services are expected to not comprise more than 20% of the funding for direct services within one contribution agreement, although IRCC can authorize funding in excess of 20% in special circumstances.

3) Indirect Services

These include projects that support the development of partnerships, capacity building and the sharing of best practices among SPOs. For example, indirect projects may focus on: developing new and innovative interventions, updating training content, conducting research, creating new tools as well as curricula, etc.^{Footnote 7} Examples of Indirect Services include community partnerships and networks for local planning and settlement coordination, namely Local Immigration Partnerships (LIP) and Réseaux en immigration francophone (RIF) which are the two largest components of IRCC funding for Indirect Services.^{Footnote 8} A LIP is a partnership network that aims to coordinate services for newcomers at the local level by bringing together various stakeholders outside of traditional settlement service providers including employers, school boards, health centres and networks, boards of trade, levels of government, professional associations, ethno-cultural organizations, faith-based organizations, and the community and social services sectors. RIF are important parts of the national strategy to strengthen Francophone Minority Communities, with the aim to enhance community capacity to recruit, welcome, integrate and retain French-speaking immigrants outside of Quebec.

1.2.2. Eligibility and Client Uptake

As per the Settlement Program Terms and Conditions, all permanent residents (PR) who have not yet become Canadian citizens, permanent resident applicants approved in principle (pending immigration security and health checks), protected persons as defined in Section 95 of IRPA, as well as some temporary residents, are eligible to receive direct settlement services funded by IRCC.^{Footnote 9},^{Footnote 10} As services are not mandatory, not all newcomers access IRCC-funded settlement services.

Within any given year, settlement clients can be made up of newcomers from various years of admission to Canada. As a result, in order to estimate client uptake, the evaluation examined the percentage of adult permanent residents that were admitted to Canada in 2015 that used at least one IRCC-funded Settlement service between January 2015 and April 2017.^{Footnote 11}

In examining the adult permanent residents who arrived in the 2015 admissions year, 39% of them used at least one IRCC-funded Settlement service by April 2017.

Source: GCMS and IRCC Settlement Client Continuum, April 2017.

1.2.3. Other Stakeholders and Partners

While IRCC-funded Settlement services are delivered primarily by SPOs, a variety of stakeholders and partners play key roles in supporting the overall integration process for newcomers.

• Provinces and Territories fund complementary settlement services and are responsible for education, health, and social services. Provinces/Territories provide settlement support and services in areas such as language training, labour market integration, recognition of foreign credentials, business development and youth integration. They also work with the Government of Canada on foreign qualification recognition issues.
• Municipalities provide additional support to newcomers (e.g., housing and public transportation).
• Other Federal Departments (e.g. Employment and Social Development Canada, Health Canada, Service Canada, Heritage Canada) fund various initiatives affecting newcomers, such as multiculturalism or foreign credential recognition initiatives and in some instances provide multi-lingual government services.
• Employers and Employer Associations, including Chambers of Commerce, Sector Councils play a role in supporting newcomer employment and the foreign credential recognition process.
• Regulators and Apprenticeship Authorities are responsible for licensure/trade certification.^{Footnote 12}

1.2.4. Overview of Settlement Programs in Other Countries

The evaluation conducted a limited review of publically available information on settlement programs in other countries, which included two international comparison studies conducted by the University of Toronto in 2010 and Ryerson University in 2016.^{Footnote 13}

Shields et al, noted that countries with a history of large in-migration (Australia, Canada, New Zealand and the United States) have had an extensive role for non-profit organizations in settlement service provision over many years. However, among these countries the role of the state in terms of funding these organizations has varied. Canada and Australia are at one end of the spectrum, where the federal government plays a comparatively large role in funding organizations to deliver settlement services on its behalf. At the other end are the United States and Britain which have adopted a more laissez-faire approach, in which immigrants are largely held responsible for their own integration and government intervention to support settlement programming is limited. With the exception of providing support to refugees, no single United States federal government institution is responsible for immigrant integration. While there is some funding from State governments for settlement services, Siemiatycki et al, note that these services are typically delivered on an ad hoc basis and funds that do flow to non-profit organizations for such purposes are both unstable and considerably lower than in the Canadian case.^{Footnote 14}

In Britain, given the absence of newcomer employment support programs and minimal central government funding for newcomer English language instruction, some local authorities and/or national governments such as Scotland have stepped in to create their own immigrant integration programs; however, some authors note that these interventions are often limited.^{Footnote 15}

The University and Toronto and Ryerson University reviews indicated that in many countries employment/labour market programs are the most common type of settlement services provided followed by services to support language acquisition, education (i.e., focused on the enrollment of newcomer children in the school system) and civic integration. Aside from Canada, only two countries (Belgium and France) have some form of pre-arrival services available for newcomers.

While a few countries have mandated curricula for settlement programming (i.e., Germany, France, the Netherlands and the Flanders region of Belgium), other countries are more flexible in terms of settlement programming. The reviews found that most settlement programming, including language courses, are usually offered free of charge or, in some cases, only a small payment is required. However, there are some exceptions, such as mandatory language and integration courses for newcomers in the Netherlands for which newcomers are required to pay for themselves and/or seek loans to help cover the costs.^{Footnote 16}

Other key highlights from comparable countries include:

Australia provides an Adult Migrant English Program, which allows for up to 510 hours of English-language instruction to eligible migrants and humanitarian entrants to help them learn foundation English language and settlement skills to enable them to participate socially and economically in Australian society. The Australian government also provides additional tuition through the Special Preparatory Program to eligible humanitarian entrants with low levels of schooling or who have had difficult pre-migration experiences such as torture and/or trauma. In 2017, changes were announced to the AMEP to provide access to a capped program of up to 490 hours of additional tuition for clients who have not reached functional English after completing their legislative entitlement of 510 hours and removing the funding cap applied to the Special Preparatory Program allowing all eligible humanitarian entrants to access additional training.^{Footnote 17} Australia also provides free translation and interpretation services to certain groups in order to facilitate communication with immigrants (e.g., healthcare workers, emergency services, trade unions, parliamentarians, local government authorities and community-based organizations involved in settlement services) as well as the Settlement Grants Program which provided funding for other settlement services (e.g., care for elderly, English classes for newcomers not eligible for Adult Migrant English Program, driving instruction, housing assistance, legal advice and counselling services).^{Footnote 18}

Germany provides integration courses which are made up of 900 hours of language instruction and 45 hours of civics lessons, focusing on Germany’s legal framework, history and culture.^{Footnote 19} An exam is held at the end of the process. Immigrants who pass the exam are awarded a permanent residency permit and a reduction in the residency period required for naturalization. These courses are mandatory for all newcomers from non-EU states that are unable to speak German at a basic level, who receive unemployment benefits or are deemed to have a special need for integration. In addition to these courses, the federal government also provides funding to state governments and municipalities to further develop settlement programming and adapt their services to meet the needs of newcomers.

France requires newcomers to sign integration contracts with the government upon arrival if they wish to obtain a residence card. By signing the contract, newcomers agreed to undergo language training (if necessary) and mandatory civic training course (i.e., teaching French values and the political and administrative organization of French society). The certificate awarded upon successful completion of these classes entitles the immigrant to a longer residence permit (i.e., 10 years). However, if the newcomer fails to earn the certificate, he or she may lose their residence permit or receive only a one-year renewal of their first residence permit. Newcomers are entitled to receive a maximum of 400 hours in a publically-funded language course.^{Footnote 20}

Several countries formally employ sports as a tool for integration. Although primarily aimed at children, sports programming may also benefit parents since they are provided with an opportunity to interact with non-immigrant parents during sports practices and matches which can play an important role in building networks, connections and social capital. According to Shields et al, Australia seems to be a leading country in terms of using sports as a tool to support newcomer integration.^{Footnote 21}

Several countries offer integration courses tailored to meet the specific needs of women and children. According to Shields et al, Germany offers innovative programming in this regard including courses for women to overcome the barriers that may keep them from joining integration courses. Similar to Canada, other countries such as Germany and France, offer child-minding services to facilitate the enrollment of women in language and other integration courses.^{Footnote 22}

2. Settlement Program Profile

2.1. Settlement Program Clients

According to iCARE administrative data, key characteristics of all Settlement Program clients who received at least one settlement service in FY 2016/17 include the following:

Overall unique clients

412,392 unique clients received at least one Settlement service, compared to 362,661 unique clients in FY 2014/15, and 401,446 in FY 2015/16.

Immigration category (top 3)

Refugees^{Footnote 23} (28%), Economic Spouses and Dependants (SD) (26%), and sponsored family (24%).

Gender

57% of unique clients were female.

Age

40% were 30-44 years of age, 23% were 15-29 years of age, 18% were 45-60 years of age, 14% were 0-15 years of age, 6% were 60-74 years of age, and 2% were 75 years of age or older or not stated.

Year of admission (top 5)

2016 (27%), 2015 (15%), 2014 (10%), 2013 (9%), 2012 (9%).

Self-declared Knowledge of Official Language

53% English, 40% Neither, 2% Both English and French, 2% French and 2% not stated.

Education Qualification

35% had Secondary or less, 17% had a Bachelor’s degree, 16% had no education.

Country of Citizenship (top 5)

China (16%), India (11%), Philippines (9%), Syria (9%), Iran (4%).

Intended province of destination (top 5)

Ontario (48%), British Columbia (15%), Alberta (13%), Manitoba (8%), and Quebec (5%).

Language spoken (top 5)

Arabic (12%), Mandarin (10%), Tagalog (8%), Punjabi (5%), and Spanish (5%).

A full profile of clients is presented in the technical appendices.

2.2. Settlement Program Services

Table 2 presents key characteristics of settlement program clients (18 years of age or older at the time of service) by settlement module for FY 2015/16. A full profile of Settlement services is presented in the technical appendices.

Table 2: Settlement Program Service Profile Highlights (Adults), FY 2016/17

Note: The Service Profile highlights for FY 2016/17 display the current population of Settlement clients, and not the population under review. A socio-demographic breakdown of the clients during the scope period can be found in the Technical Appendices.

Source: Settlement Client Continuum, September 2017.

2.3. Settlement Program Service Provider Organizations

IRCC funds a wide range of SPOs. This network of SPOs, most of which have been providing settlement services for many years, provides a wide range of settlement and non-settlement services to newcomers in a variety of locations to better reach newcomers and meet their various needs. More specifically, a survey of direct service IRCC-funded Settlement service providers indicated the following key characteristics of IRCC-funded SPOs:

Type of organization

• 51% of SPOs identified as an organization providing primarily social services to individuals and families
• 29% were an educational institution
• 8% primarily provide employment-related services
• 7% identified as a community, ethnic, fraternal, civic and social organization or association
• 3% were a health-care focused organization.^{Footnote 24}

IRCC-funded settlement services provided

• 78% of SPOs provide I&O
• 74% provide CC
• 67% provide NARS
• 55% provide LT
• 46% provide ER
• 22% provide LA.

Other services

• 70% of SPOs also offer non-settlement services (e.g., employment, education, or other social services for all Canadians, including newcomers).

Years providing settlement services

About half of the SPOs (46%) have been providing settlement services for 5 to 19 years

• 22% of SPOs have been for 20 to 29 years
• 16% for 30 to 39 years
• 14% for 40 or more
• 3% for less than 5 years.

Number of clients served

• 15% of SPOs provide Settlement services to less than 100 clients per year
• 12% to 100 to 199 clients
• 20% to 200 to 499 clients
• 15% to 500 to 999 clients
• 25% to 1,000 to 5,000 clients
• 12% to more than 5,000 clients.

Co-location with other organizations

• 27% of SPOs were co-located with other organizations offering settlement services, and 51% with other organizations offering other non-settlement services that are helpful to newcomers.

Location of service delivery

• 54% of SPOs deliver IRCC-funded services at multiple permanent service locations and 65% offer services at itinerant locations.^{Footnote 25}

Human resources

• 42% of SPOs have between 1 and 5 full time equivalent (FTE) supporting the provision of IRCC-funded settlement services,
• 17% have 6 to 10 FTEs,
• 13% 11 to 20 FTEs,
• 19% between 21 and 50 FTEs
• 9% more than 50 FTEs

Use of volunteers

• 79% of SPOs make use of volunteers to support the provision of IRCC-funded settlement services.

3. Methodology

3.1. Evaluation Approach

The evaluation scope and approach were determined during a planning phase, in consultation with IRCC branches involved in the design, management and delivery of the Settlement Program. The terms of reference for the evaluation was approved by IRCC’s Performance Measurement and Evaluation Committee in November 2015, and the evaluation was conducted by the IRCC evaluation team with the support of an external contractor from November 2015 to May 2017.

3.2. Evaluation Scope

The evaluation assessed the issues of relevance and performance of the Settlement Program for the period between FY 2011/12 and FY 2015/16, and was guided by the program logic model, which outlines the expected immediate and intermediate outcomes for the program.^{Footnote 26}

Specific evaluation questions were developed to address these core issues including:

• the continued need for the Settlement, its continued alignment with IRCC and Government of Canada priorities, as well as the appropriateness of federal role in supporting settlement services;
• the extent to which clients are achieving their settlement and integration outcomes, given their assessed needs and the services they have received from IRCC (immediate and intermediate outcomes);
• the relevance and performance of Local Immigration Partnerships and Support Services;
• the reach of the Program including the number of clients served as well as the geographic range of service provision;
• the extent to which program development, governance and management effectively support consistent and responsive program delivery; and,
• the cost of the Program, broken down by stream and client groups.^{Footnote 27}

While the evaluation did analyze the impact of specific settlement streams on achieving client outcomes, it did not focus on the delivery aspects of each of these settlement services, as consideration was given to the implementation of the ‘modernized approach’ to the Settlement Program, implemented in 2008. In addition, the evaluation did not focus on the following areas: indirect services (with the exception of LIPs), pre-arrival initiatives, and IRCC’s initiatives under the Roadmap for Canada’s Official Languages 2013-2018. An Evaluation of the Official Languages Minority Communities (OLMC) was conducted separately, and examined by Réseau en Immigration Francophone (RIF).^{Footnote 28}

3.3. Data Collection Methods

Multiple lines of evidence were used to gather qualitative and quantitative data from a wide range of perspectives, including settlement clients, stakeholders, and program officials.

These lines of evidence included the following:

3.4. Considerations and Limitations

Overall, the evaluation design employed numerous qualitative and quantitative methodologies. The different lines of evidence were complementary and reduced information gaps, and generally, the results converged towards common and integrated findings.

A few limitations to this evaluation included:

Lack of data on newcomers that do not access IRCC Settlement services

The primary line of evidence to assess clients’ outcomes was the Settlement Client Outcomes Survey (SCOS). Due to the lack of a comparison group of newcomers that did not access any IRCC Settlement services, the analysis was not able to fully isolate the unique impact of IRCC-funded Settlement services as all respondents had accessed at least one IRCC-funded settlement service. Despite this limitation, it was possible to compare the results of sub-groups within Settlement Program clients (i.e., clients that used specific IRCC-funded services as compared to Settlement clients that didn’t use these specific services, immigration categories, levels of education, etc.), which provide insights into which specific Settlement services or socio-demographic factors have the greatest impacts on client outcomes.

Other factors to which client outcomes may be attributed

While the evaluation used multivariate regression models to isolate the unique impact of IRCC-funded Settlement services and certain socio-demographic variables (e.g., age, gender, country of citizenship, year of landing in Canada, etc.) on influencing client outcomes, it was not possible to control for all contributing factors such as learner motivation or all the different delivery models and curricula used by SPOs. Despite this, statistically significant results were obtained which allowed for robust inferences to be made about the contribution of the program streams in achieving settlement outcomes.

Data availabilities

Although the Immigration Contribution Agreement Reporting Environment (iCARE) was launched in 2013 to replace the IRCC’s Immigration Contribution Accountability System (iCAMS), there was a transition period between the two platforms as functionalities were added to allow for reporting requirements up until October 2015. As such, data on the uptake of IRCC-funded Settlement services may not always be comparable year to year between iCAMS and iCARE, and a presentation of the data prior to the roll-out of the various iCARE modules may not always capture the full population served or service characteristics. In addition, data available in iCARE regarding Manitoba and British Columbia was not available for earlier years under review, as the federal government repatriated Settlement services in 2013 and 2014, respectively.^{Footnote 29}

Difficulty in analyzing some administrative data

Some challenges were encountered with the administrative data for the Settlement Program, including difficulties in obtaining data extractions in a timely fashion as well as incorrect information in a few iCARE datasets (e.g., minors receiving employment services). As a result, only high level information on program streams delivered to clients was available to assess the extent to which settlement services received have an impact client outcomes. In addition, qualitative data from the Narrative Reports and Annual Project Performance Reports (APPR) intended to be used a project level tool was challenging to analyze at the SPO level. Thus, the evaluation had to heavily rely upon a survey of service providers to obtain SPO-level information regarding service delivery.

Despite these limitations, the triangulation of the multiple lines of evidence, along with the mitigation strategies used in this evaluation were considered sufficient to ensure that the findings are reliable and can be used with confidence.

4. Relevance

4.1. Continued Need and Relevance for Settlement Services

Overall Need for Settlement Services

The need for formal settlement services is often dependent on the extent to which newcomers possess the necessary knowledge and skills upon arrival to settle and integrate quickly in Canadian society (e.g., ability to communicate in an official language, knowledge of the Canadian work environment, knowledge about life in Canada, etc.), and if not, whether there are strong networks of supports they can rely upon to obtain the necessary knowledge and skills to successfully settle and integrate (e.g., family and friends that can provide useful knowledge of life in Canada, strong social or employment connections).

Both literature reviewed and an analysis of data from IRCC’s Needs Assessment module confirmed the existence of gaps in knowledge as many newcomers face a variety of challenges and have a variety of needs upon arrival (i.e., dealing with cultural barriers, limited contacts or networks and unfamiliarity with Canadian institutions or how to find work or educational opportunities in Canada).^{Footnote 30} In addition to these, learning a new language was also cited as key need and challenge for newcomers as an analysis of clients’ language profile at landing revealed that approximately 87% of Canada’s permanent residents in 2015 had a mother tongue other than English or French and an estimated 23% of Canada’s PRs felt they could converse in neither official language.^{Footnote 31}

In terms of sources of information and support to help newcomers address these challenges, both literature and the Settlement Client Outcomes Survey confirmed that newcomers rely upon multiple sources of information including formal settlement services provided by governments, non-profit agencies and volunteers as well as other supports such as websites, informal networks of friends, family, teachers, and community members. However, since settlement services are specifically designed to assist newcomers through unique challenges faced when arriving in a new country, providing settlement remain an integral component of any immigration program.^{Footnote 32} A few interviewees mentioned that while family or friends may help a newcomer to learn a language or obtain knowledge about the Canadian work environment, they are not a substitute for formal language training or employment services delivered by staff with specific expertise. In addition, the provision of settlement services for newcomers in many other countries further suggests that they are needed, at least in some form. Finally, in terms of the overall level of need in the Canadian context, data also indicates an increasing need for providing settlement services as the number of unique clients accessing IRCC-funded Settlement services increased by 11% between FY 2014/15 and FY 2015/16 and by another 3% in FY 2016/17.^{Footnote 33}

Suite of Settlement Services

With regard to the suite of services provided, few studies focus on the specific supports required for newcomers, with some citing the need for language training, information provision, counselling, employment-programs, and fostering community partnerships.^{Footnote 34} Several studies have noted that the needs of newcomers are diverse and vary depending on various factors including “education, age, sex, country of origin, family status and knowledge of the dominant language in their country” and stress the need for flexible and tailored services to serve the unique needs of the newcomers.^{Footnote 35} This sentiment was echoed by key informants, as most stressed the importance of providing flexible services and responded that the six streams of direct services provided by IRCC are complementary. Furthermore, iCARE data from IRCC’s NARS module also indicates that the specific needs of newcomers is variable, with certain groups requiring information on certain topics. For instance, of those newcomers that received at least one needs assessment, 29% of sponsored family members identified a need for information on working in Canada as opposed to 47% of economic immigrants (see Section 5.1 for a further discussion on NARS).

Proportion of Newcomers, Specific Groups, and Service Duration

Despite the widespread consensus that settlement services should be provided in some form, it is more difficult to quantify the precise level of need (i.e., the proportion of newcomers that need settlement services, which groups of newcomers, and the length for which they should be provided).

Regarding the proportion of newcomers that need Settlement services, although roughly 39% of newcomers accessed at least one IRCC-funded Settlement service in their first 2 years in Canada, evidence from the Settlement Client Outcome Survey indicated that many Settlement clients make use of supports beyond IRCC-funded Settlement services (e.g., family and friends, participating in community events, informal training, websites and publications, etc.). This suggests that the same Settlement services are not needed for all newcomers to the same extent and for the same duration. As all IRCC-funded Settlement services are accessible without charge to all permanent residents regardless of income, the current design of the Settlement Program enables newcomers to make use of all available services at their disposal to improve their knowledge and skills.

Conversely, for the almost two-thirds (61%) of newcomers who did not access IRCC-funded Settlement services in their first two years of coming to Canada, it is possible that some were either unaware of their existence, unable to access them due to barriers (i.e., lack of transportation, need to find a survival job) or limited service offerings in their area. However, it is possible that at least some of these individuals that did not access IRCC-settlement services, did not feel they needed the services (i.e., their needs were met or fulfilled through other means/sources of information).

In terms of the specific groups with the highest needs for services, the composition of users in iCARE reveals there are specific newcomers that are more likely to use (and be in need of settlement services). As mentioned in Section 1.2.2, approximately 72% of resettled refugees use Settlement services within their first two years as compared to 31% of economic immigrant principal applicants. In line with iCARE data, all interviewees indicated that vulnerable populations had a greater need for settlement services. This notion of vulnerability covers a lot of different factors such as trauma and mental health problems, isolation, language, as well as other barriers to employment. Almost all interviewees mentioned that refugees formed a particular group in great need of settlement services. Families of highly skilled/economic immigrants were also frequently highlighted, as these key informants explained that these individuals may not have the same level of skills as the principal applicants. Although a lower proportion of economic newcomers use Settlement services, many still do which is explained by several studies that noted that even immigrants with high human capital have had difficulty finding appropriate levels of employment in host societies and may seek services to help them with labour market preparation or support with foreign credential assessment.^{Footnote 36}

While no literature was found which provided conclusive evidence for how long Settlement services should be provided, interviewees were split on this subject. Around half of interviewees suggested that the current window for services was somewhat insufficient and that some immigration groups should be able to access services past receiving citizenship. Conversely, the other half maintained that the current eligibility requirement to make services available until citizenship is appropriate as it provides newcomers with a reasonable timeframe to improve their knowledge and skills and is not unfair to Canadian citizens that are not provided with the same suite of services.

Both the length of time from admission to using IRCC-funded services and the number of years a newcomer will use Settlement services are variable and differ considerably by immigration category. This suggests further that certain groups of newcomers require Settlement services for more or less time than other groups. For example, for those who received Settlement services between January 1, 2014 and March 31, 2016, 68% used their first service within 90 days of admission, 10% between their first 90 and 180 days, 8% between 181 days and their first year and 5% beyond their first year after admission.^{Footnote 37} Of those who received services during this period, 83% of resettled refugees used services within their first 90 days as compared to 73% of economic immigrants.^{Footnote 38} In terms of those who received Settlement services in FY 2015/16, approximately 7% of were admitted to Canada in 2016, 21% in 2015, 13% in 2014, 11% in 2013, 10% in 2012, 9% in 2011 and around a quarter of clients (28%) in that year were admitted to Canada prior to 2011.^{Footnote 39} This figure differed by Settlement service type as in FY 2015/16, 12% of clients of Employment-related services were admitted to Canada prior to 2011 compared to 29% of clients receiving Information and Orientation services.^{Footnote 40}

4.2. Alignment with Federal Roles and Responsibilities

The Government of Canada has an obligation through the Immigration and Refugee Protection Act (IRPA) to, “promote the successful integration of permanent residents into Canada while recognizing that integration involves mutual obligations for newcomers and Canadian society.”^{Footnote 41} The importance of the federal role in the funding and administering of Settlement services was clearly highlighted in all interviewees, who cited the distribution of funds, national consistency, maintenance of financial oversight, and working with various partners to deliver cohesive services.

Appropriateness of Third-Party Service Provider Model

A cross-national scan reveals that countries providing settlement services to newcomers make use of non-profit service providers in immigrant settlement and integration programming to varying degrees. The current approach to settlement service delivery in Canada can be characterised as a public-private partnership model. According to literature, benefits of this type of model are that it reduces the size of government administration and allows the federal government to promote consistency of service delivery while still allowing for delivery flexibility based on needs determined at the local level. In addition, since non-profit agencies are often established in the immigrant communities they serve, they are often best qualified to provide services to immigrants due to the organizational skills and connections to communities that allow them to better identify newcomers’ needs.^{Footnote 42} According to literature, the major downside to this public/private model of settlement service delivery is that it can foster a competitive environment among service providers which may hinder partnerships, coordination of service delivery and the sharing of information and best practices.^{Footnote 43}

Complementarity of Provincial Settlement Services

IRCC engages with provinces and territories both at the multilateral and bilateral levels in efforts to ensure that funded settlement services are not duplicative of services provided by other jurisdictions; there are no gaps in services; and all stakeholders are engaged in development of settlement services.

The Forum of Federal-Provincial-Territorial (FPT) Ministers Responsible for Immigration is a primary mechanism through which intergovernmental collaboration occurs with key actions, as defined in the FPT Vision Action Plan (2016-2019) aimed at delivering a more effective and efficient immigration system.

IRCC also supports the work of the FPT Settlement Working Group to address common objectives across Canada. Meetings and activities of this Working Group are focused on the achievement of the stated vision and goals of the Federal-Provincial-Territorial Forum of Ministers Responsible for Immigration, particularly those contained in FPT Vision Action Plan 2016-2019, and in the FPT Settlement Working Group’s work plan.

At the bilateral level, the Department negotiates with provinces and territories settlement arrangements and governance mechanisms which address specific jurisdictional needs. These bilateral settlement arrangements articulate the joint vision, principles and objectives within the context of shared interests; and, define bilateral collaboration in information sharing; delivery of settlement services and areas of bilateral partnership.

Interviewees noted the department’s Settlement services were complementing the work of other government departments, provinces/territories, and stakeholders, and that there was continuous efforts to avoid duplication through communication and coordination between players, namely through these specialised working groups.

While in many cases provinces provide funding to support settlement services, they often do so to support newcomers that are ineligible for federally-funded services. For example, a 2016, a scan of FPT programming undertaken by the Settlement Working Group found that generally there is minimal overlap in the programming offered by the FPT Governments. The programming offered by the PTs tends to complement federal funding in order to make services available to clients who are ineligible for federally funded Settlement service. Moreover, the scan found that more needs to be done by all levels of government to reach vulnerable newcomers and newcomers residing in rural and remote communities.

The high reliance by SPOs on a multitude of sources for funding to support Settlement services speaks to the complementarity of federal and provincial funding to support settlement services.^{Footnote 44} In the survey of IRCC-funded Settlement Program service providers, of the 63% of SPOs that indicated they received funding for Settlement services from non-IRCC sources, 40% reported that this funding is used to serve only non-IRCC-eligible clients, 9% indicated that this funding is used to support only clients that are eligible for IRCC-funded Settlement services and 51% that this funding is used to support both eligible and non-eligible settlement IRCC clients. In addition, in the SPO survey, the most common non-IRCC eligible settlement clients supported by SPOs were naturalized citizens (87% of SPOs), refugee claimants (84% of SPOs) and Canadian-born citizens (73% of SPOs).^{Footnote 45}

According to internal research, in terms of specific programming differences between IRCC and Provincial-Territorial settlement programs include significant provincial funding for Bridge to Work Programs (e.g., Ontario, Nova Scotia, Alberta and British Columbia).^{Footnote 46} IRCC settlement language programming is complemented by provincial and territorial investments, but Province/Territory (PT) funded language training is often embedded within other programs geared towards academic preparation or job access. Eligibility criteria for PT-funded programs tend to be broader, and include clients who may not be eligible for federally funded training such as temporary residents and naturalized Canadian citizens.

Repatriation of Settlement Services

The provincial administration over Settlement services in Manitoba and British Columbia ended in 2013 and 2014, respectively, when the Federal government repatriated the Settlement Services under the intention of ensuring that all immigrants had the same access to Settlement services, regardless of their jurisdiction, as well as ensuring that each region in Canada received its share of appropriate federal settlement funding.^{Footnote 47}

Most key informants noted that the repatriation of services in British Columbia and Manitoba ensured a degree of consistency and uniformity at the national level. Some respondents also spoke to benefits in terms of collaboration and information sharing and providers gaining greater access to the national community of service providers. In terms of challenges, some IRCC respondents referred to the difficulties of operationalizing the repatriation, and others indicated that the process had generated tensions with PTs and local providers. Some external respondents felt the repatriation had raised significant challenges namely due to funding cuts and restrictions as well as administrative changes (e.g., methods of performance reporting and evaluation), while others felt service providers had lost a degree of flexibility and the possibility to tailor services to specific local realities.

4.3. Settlement as a Government of Canada and Departmental Priority

Both interviewees and key public documents confirmed that Settlement Program has been a high priority for the Government of Canada and the department over the past five years. Some interviewees mentioned that the Settlement Program has been the largest grants and contributions program of the department and that this amount of funding was further enhanced by Budget 2016 to increase Settlement programming.^{Footnote 48} The importance of settlement was also reiterated through the 2017 ministerial Mandate Letter for Minister Hussen, which listed the “delivery of high-quality Settlement services to ensure the successful arrival of new Canadians” as one of the minister’s top priorities. In addition, interviewees mentioned that the Government of Canada initiative to resettle 25,000 Syrian Refugees by February 2016 and a total of 44,800 refugees in 2016 alone, also highlighted the ongoing role that settlement plays in the integration process of refugees, as it assists them in having the means to facilitate their integration process.^{Footnote 49}

5. Achievement of Expected Client Outcomes

5.1. Challenges in Identifying Impacts of the Settlement Services on Client Outcomes

All IRCC-funded Settlement services aim to provide clients with knowledge and skills to function in the Canadian society. Although each stream typically targets a primary program outcome, many streams provide activities that support achievement of several outcomes, meaning that clients can derive similar benefits from multiple Settlement program streams. For example, clients can gain knowledge and skills about working in Canada mainly through Employment-related services, but also via the job search workshops provided under Information and Orientation services, through language training and through activities targeting access to the labour market under Community Connections services.

This approach to service provision has the benefit of being client-focused, recognizing that clients have different needs and different ways of learning. Although valuable from a client perspective, this blended approach to the provision of Settlement services makes it challenging to identify the primary impact of a stream in isolation and even more challenging to assess the secondary impact of a stream on clients’ outcome.

Despite these challenges in identifying impacts of the Settlement services, the evaluation was able to provide an overall view of the impact of the main Settlement program streams, simultaneously, towards achievement of program outcomes.

5.2. Clients Receive Appropriate Information and Services

SPOs provide Needs Assessment and Referral Services (NARS) to Settlement clients to assess their needs and link them to appropriate Settlement and other community-based services.^{Footnote 50} According to administrative data, 158,888 clients received a NARS in FY 2015/16; of which, 25% were Economic Principal Applicants (PA) and 25% were Sponsored Family.^{Footnote 51} Although NARS were most commonly obtained by clients who arrived in Canada in more recent years, 11% of NARS clients had been in Canada for almost ten years or more (i.e., arriving in Canada pre-2008).

According to Settlement Program documentation, NARS are universal practices in the settlement sector; however, there is considerable variation in the approaches used (e.g., intake procedures, tools and diagnostics). In most cases, the methods and tools that are used have been developed over a period of time and have been adapted to respond to the particular organizational and community context within which SPOs work.^{Footnote 52}

An analysis of NARS administrative data obtained through iCARE showed that 68% of Settlement clients admitted to Canada in 2014 and 2015 received at least one NARS.^{Footnote 53} Most NARS clients (62%) received only one NARS, and 36% received between two and five NARS. Findings from this analysis demonstrated that Settlement clients identify needs for a variety of IRCC and non-IRCC services.^{Footnote 54}

• 91% of those identifying IRCC program needs identified two or more of these needs, and 58% identified five or more (up to 11).^{Footnote 55}
• 48% of those identifying non-IRCC program needs identified two or more of these needs and 16% identified five or more (up to 10).^{Footnote 56}

Overall, NARS clients identified needs related to knowledge and information services most frequently. The top three needs identified were to increase knowledge of Community and Government Services, increase knowledge of Life in Canada, and increase knowledge of Education in Canada. Settlement clients identifying these needs were primarily Economic (PAs and Spouses and Dependants (SD)) and Refugees.

The NARS analysis examined the extent to which Settlement clients who identified a particular need received a corresponding Settlement service, or in the case of non-IRCC program needs, received a corresponding referral to a non-IRCC service.

In terms of receiving Settlement services, the NARS analysis found that many NARS clients received corresponding services. Overall, NARS clients identifying IRCC program needs related to increasing knowledge of community and government services (90%) and increasing knowledge of life in Canada (89%) had the highest frequency of receiving a corresponding Information and Orientation service. In addition, 60% of those identifying a need to improve their language skills received a corresponding IRCC Language service^{Footnote 57} and 43% of those identifying a need to find employment received a corresponding IRCC Employment-Related service.^{Footnote 58}

However, NARS clients identifying needs related to their social integration generally had a lower frequency of receiving a corresponding Community Connections service. For example, 22% of NARS clients identifying a need to increase their access to local community services received a corresponding Community Connections service, compared to 27% of those who identified a need to increase their level of community involvement, and 28% of those who identified a need to increase social networks.

In terms of referrals, the NARS analysis found that most NARS clients identifying non-IRCC program needs received a corresponding referral to a non-IRCC service (on average, 82% of NARS clients who identified a particular non-IRCC need). The rate of referral was highest for clients identifying needs related to non-IRCC community services at 91% and was lowest for those identifying needs related to non-IRCC family support at 71%.

Meeting the service needs of clients was also explored in the Settlement Client Outcomes Survey. When asked if they had had any problems or difficulties getting the services that they needed, 76% of Settlement clients surveyed indicated no problems or difficulties (n=15,917). Of those who reported having a difficultly or problem getting the services they need, the most frequent reason was not knowing about how or where to get services.

Settlement clients surveyed were also asked if there were areas where more information or services would have been helpful. Those who responded indicated the following:

• 29% cited employment (e.g., more information and resources to help find work, develop the right job-searching tools, learn about internships, co-op programs and career changes);
• 13% cited education (e.g., obtaining clear information about higher education, information on vocational, professional and adult training opportunities); and
• 7% cited accessing government services (e.g., information on different immigrant serving organizations, help with completing official applications, using public transit or securing a driver’s license).^{Footnote 59}

5.3. Clients Attain Awareness of Community Resources

Information and Orientation services aim to provide newcomers with relevant, accurate, consistent, and timely settlement-related information and orientation that is needed to make informed decisions and promote an understanding of life in Canada.

According to IRCC administrative data, a total of 243,116 Settlement clients, 18 years of age or older, received at least one Information and Orientation service in FY 2015/16.^{Footnote 60} Most obtained a service with regards to Sources of information (67%), followed by Important documents (54%). The most frequent way Information and Orientation services were delivered was in a one-on-one session with a settlement worker (78%), followed by group orientation (33%), and family orientation (26%).

5.3.1. Clients Attain Awareness of Community and Other Resources

The Settlement Client Outcome Survey was used to assess the extent to which Settlement clients attained awareness of community and other resources to deal with settlement issues. Clients rated their knowledge of service organizations in their community (e.g., where to find libraries, places to play sports, community centres, places of worship, such as churches, mosques, etc.).

Overall, the majority (92%) of Settlement clients reported having at least some knowledge of community and other resources. Two thirds (67%) felt they had quite a lot of knowledge (33%) or a great deal of knowledge (34%) about community and other resources, and close to a quarter felt they had some knowledge about it (24%).

Further analyses^{Footnote 61} were conducted to assess the impact of specific IRCC-funded Settlement services on clients' reported knowledge of community resources, when taking into account other characteristics at the same time. Out of the IRCC-funded services, Employment-related services, Information and Orientation services as well as Community Connection services positively impacted clients' reported knowledge of community and other resources. As such, Settlement clients who obtained Employment-related services, Information and Orientation services as well as Community Connection services have a higher level of knowledge about community resources. There were no statistical differences observed for the IRCC-funded Language Training.

The analysis also showed that the most important socio-demographic characteristics for greater knowledge of community resources were, by order of importance: country of citizenship, year of admission to Canada (i.e., length of time in Canada), and education. This can be explained by the fact that, as the more time a person has spent in Canada, the more likely they are to become familiar with their community and the resources it offers. Similarly, holding a university degree provides immigrants with greater personal resources, enabling them to become more aware of the services available in their community.

5.4. Clients Gain Knowledge of Life in Canada, Including Laws, Rights and Responsibilities

5.4.1. Clients Gain Knowledge of Life in Canada

In the SCOS, clients were asked to rate their level of knowledge about transportation, housing, health care, education and money and finances. These five elements of life in Canada were grouped together to form one factor: “Knowledge of Life in Canada”.

Almost all (95%) Settlement clients reported at least some knowledge of life in Canada topics with the majority reporting either a great deal of knowledge (22%) or quite a lot of knowledge (44%). The remainder reported having some knowledge (30%), not very much knowledge (4%) or no knowledge at all (0.5%).

Of IRCC-funded services, Employment-related services was the one Settlement stream that positively impacted clients' reported knowledge of life in Canada, when taking into account other characteristics at the same time. This indicates that Settlement clients who obtained Employment-related services had a higher level of knowledge about life in Canada. No statistical differences were observed between clients who received IRCC-funded Information and Orientation services compared to those who had not received such services^{Footnote 62}, i.e., both groups of clients had the same level of knowledge about life in Canada.

The analysis also showed that of the socio-demographic characteristics affected knowledge of life in Canada, including: country of citizenship, year of admission to Canada (i.e., length of time in Canada), and education. As suggested for the previous program outcome, these results can be explained by the fact that, the more time a person has spent in Canada, the more likely they are to become familiar with the various aspects of life in Canada, and that holding a university degree provides immigrants with greater personal resources, enabling them to learn more easily about life in a new country.

5.4.2. Clients Gain Knowledge of Laws, Rights and Responsibilities

In the SCOS, clients were asked to rate their level of knowledge about history, geography, culture of Canada, rights and freedoms in Canada, Canadian law and justice and becoming a Canadian Citizen. These three elements were grouped together to form one factor: “Knowledge of laws, rights and responsibilities”.

The majority (91%) of Settlement clients reported at least some knowledge of laws, rights and responsibilities with half reporting either a great deal of knowledge (15%) or quite a lot of knowledge (37%). The remaining clients reported not very much knowledge (9%) or no knowledge at all (0.7%).

When further analyses were conducted to assess the impact of specific IRCC-funded Settlement services on clients' reported knowledge of laws, rights and responsibilities, results indicated that of IRCC-funded services, Employment-related services is the only Settlement stream that positively impacted this program outcome, when taking into account other characteristics. This means that Settlement clients who obtained Employment-related services have a higher level of knowledge about laws, rights and responsibilities. No statistical differences were observed between clients who received IRCC-funded Information and Orientation services, Language Training or Community Connections services, compared to those who have not received such services, i.e., these groups of clients have the same level of knowledge about laws, rights and responsibilities.

The analysis also showed that the most important socio-demographic characteristics for greater knowledge of laws, rights and responsibilities, by order of importance: year of admission to Canada (i.e., length of time in Canada), country of citizenship, and immigration category. As for the other outcomes, these results could be partly explained by the fact that the longer you have been in Canada, the more time you had to acquire knowledge about this topic. In addition, refugees (GARs, PSRs and inland refugees) and family class immigrants had more knowledge about laws, rights and responsibilities than federal skilled workers. This may be attributable to specific support and orientation immigrants may have received under the Resettlement Assistance Program (RAP), or through sponsors or family members.

5.5. Clients Learn Official Language Skills

IRCC-funded Language Training services are offered for both of Canada’s official languages, English and French, and are delivered in a variety of formats, including classroom, online and blended formats. Clients can take IRCC-funded Language Training full-time or part-time. In addition, courses have different focus, depending on client need; they can be general in nature, occupation-specific, or focus on other objectives, such as academic or citizenship preparation or assisting clients with their daily life/basic needs.

IRCC also funds Language Assessment services. Language Assessments are conducted to assess language skills (in English or French) of clients for placement in a Language Training program, and use tools based on the Canadian Language Benchmarks (CLB) or the Niveaux de compétence linguistique canadiens (NCLC).^{Footnote 63}

In FY 2015/16, a total of 88,720 Settlement clients (18 years of age or older) received Language Assessment services, and 97,717 received Language Training services.^{Footnote 64} Most clients receiving these services did so to learn English. When English and French Language Assessment information in 2015/16 was taken into account, 97% of Language Assessment clients had received assessments for English only, 2% for French only and 1% for both English and French.^{Footnote 65} When English and French Language Training information in 2015/16 was taken into account, 99% of Language Training clients had received English training only, 0.8% French training only and 0.2% both English and French training.^{Footnote 66}

In terms of assessed language levels, most Language Assessment clients in 2015/16 (regardless of if they went on to pursue language training) were assessed at a level considered Basic (CLB 1 to 4) or Intermediate (CLB 5 to 8) in terms of their listening, speaking, reading and writing, as listed in Table 3.

Note: Not assessed refers to a skill that was not assessed, and not an individual who was not assessed.

Source: Settlement Client Continuum, March 2017.

In terms of the format and focus of the Language Training received, almost all clients (94%) received their Language Training in a classroom format, and the dominant focus of the training received for many clients (75%) was general in nature.^{Footnote 67} When all Language Training information was taken into account, Language Training clients received, on average, about 215 hours of training in 2015/16.

5.5.1. Approaches to Assessing Language Skills Improvement

In order to assess the extent to which Settlement clients learn official language skills, iCARE data on language progression (using CLB) through IRCC-funded Language Training was examined^{Footnote 68} alongside results from the SCOS where clients were asked to assess their own improvement in English/French since coming to Canada. While both are useful measures, each has its own advantages and disadvantages. While the iCARE data, on its own, permits an examination of clients’ progression through IRCC-funded Language Training, it is only possible to analyze the results of those that have taken these services. Given that there is no comparison group in iCARE, one cannot assess the relative impact of Language Training on clients’ progression. However, the SCOS data allows for a comparison of results between clients who have received IRCC-funded Language Training and those who have not, therefore allowing an analysis of the program impact i.e., program attribution. Part of this analysis is, however, more subjective in nature, as clients were asked to rate their own language progression.

5.5.2. Language Progression

In terms of language progression, 57% of clients who landed between January 1, 2014 and March 31, 2016 and accessed IRCC-funded Language Training progressed by at least 1 CLB (Listening, Speaking, Reading or Writing) with 11% progressing one CLB, 14% progressing 2 CLBs, 14% 3 CLBs and 18% progressing 4 CLBs. Nevertheless, there were still 43% of clients for whom no CLB progression was noted. The average number of hours needed to progress 1 CLB level in listening, speaking, reading and writing were: 441, 486, 609, and 562 respectively. However, these results differed by several characteristics, including:

Immigration category

Economic immigrants require the least number of hours on average to increase by one CLB level in all four language skills, compared to refugees and family class immigrants.

Age

Younger clients are more likely to improve at least one CLB level and need the fewest number of hours of instruction, on average, to improve one CLB level.

Education level

The likelihood of improving is positively related to level of education. Clients who have a bachelor degree and above need fewer hours of training, on average, to increase one CLB level.

Initial CLB level before training

Generally, the lower the client’s initial CLB level before training, the more likely they improve through language training.

Training types

Clients who utilized “occupation specific” language training are the most likely to improve and need the least number of training hours, on average, to increase 1 CLB level.

In addition, it was found that the more hours of Language Training a client receives, the more likely it is they will improve at least one CLB level, but the magnitude of the additional improvement diminishes as hours increase.

5.5.3. Language Skills Improvement

Clients were asked, in the SCOS, the extent to which their reading, understanding, writing and speaking English or French had improved since coming to Canada. These four elements of language improvement were grouped together to form one factor: “official languages skills improvement”.

Almost all (96%) of Settlement clients self-reported at least some improvement in their language skills since arriving in Canada, with most (73%) feeling that their language skills had improved quite a lot (39.3%) or improved a great deal (33.2%). The remainder reported that their language skills had not much improved (4%) or not at all improved (0.7%).^{Footnote 69}

Analyses were conducted to assess the impact of specific IRCC-funded Settlement services on clients' reported official language skills improvement, when taking into account other characteristics at the same time. Of IRCC-funded services, Employment-related services positively impacted clients' reported language skills improvement. In addition, language skills also improved slightly with the number of hours spent in Language Training.

Furthermore, the analysis demonstrated that the most important socio-demographic characteristics for greater language skills improvement were, by order of importance: country of citizenship, age and year of admission in Canada (i.e., length of time in Canada). Immigrants from China were the ones who had improved their language skills the least, but they are also a group for which the linguistic distance to English and French is considerable. Consistent with the results presented above in terms of CLB progression of Language Training clients, younger clients improved to a greater extent their language skills, suggesting that language acquisition is easier the younger people are. The more time immigrants have spent in the country also affects the potential for language skills improvement, as immigrants learn official languages through various means and venues over time. Those who have been in Canada for a longer period of time improved their language skills to a greater extent than those who have come to Canada more recently.

In order to better understand the impact of Language Training for specific client sub-groups, additional analyses using the SCOS were conducted by immigration category, level of education and knowledge of official languages upon admission.

Separate analyses by immigration categories demonstrated that Language Training had a greater impact on language skills improvement for economic immigrants (Federal Skilled Workers (FSW), Canadian Experience Class (CEC), Provincial Nominee Program (PNP) – PA and SD) than for immigrants from other immigration categories (Government Assisted Refugees (GAR), Privately Sponsored Refugees (PSR) and Family Class).

For the economic class as a whole (FSW, CEC and PNP, including principal applicants and spouses and dependants), Language Training had a positive impact on language improvement.^{Footnote 70} However, Language Training had no observable impact on language skills improvement for immigrants admitted under other categories (GARs, PSRs and Family Class), i.e., those who received Language Training did not improve their language skills any more than those who did not take Language Training.

Further regression analyses on level of education and knowledge of official languages indicated that Language Training had a greater impact on language skills improvement for Settlement clients with greater human capital (i.e., those who had a university degree and those who reported knowing English and/or French upon admission).

5.6. Clients’ Use of Official Languages

The SCOS was used to assess the extent to which Settlement clients use official languages outside their home. Clients were asked how often they speak, read, and write in English or French outside their home. These three elements of language use were grouped together to form one factor: “official language use”.

The majority (88%)^{Footnote 71} of Settlement clients reported using English or French about half of the time or more outside their home, with 69% using it most of the time (28%) or always (41%). The remainder reported using English or French around half the time (18%), sometimes (12%) or never (0.9%).

Further analyses were conducted to assess the impact of specific IRCC-funded Settlement services on clients' reported use of official languages outside their home. Of IRCC-funded services, Employment-related services is the Settlement stream that positively impacted clients' reported use of official languages outside their home. Clients who took Language Training reported using official languages outside their home less frequently than those who have not accessed such services. As clients with English or French as their mother tongue were less likely to take language training and had higher level of proficiency in at least one of the official languages, and as language training clients may have been less proficient and comfortable expressing themselves in one of the official languages, it was expected that this group would report using English or French outside their home less frequently.

Analysis also showed that certain socio-demographic characteristics factor into use of official languages outside the home (e.g., country of citizenship, knowledge of official languages at admission, and education). Use of official languages outside the home increases with length of time spent in Canada; with more time in Canada, immigrants may have acquired greater official languages skills and more comfort with the language leading to a greater use outside the home.

Additional analyses by immigration category, level of education and knowledge of official languages upon admission demonstrated that regardless of those attributes, those who have taken IRCC-funded Language Training used official languages outside their home less frequently than those who did not take such services; and that clients who have received IRCC-funded Employment-related services used official languages outside their home more frequently compared to those who have not received such services.

5.7. Clients Acquire Knowledge about Working in Canada and Participate in Canadian Labour Market

IRCC-funded Employment-Related services have a short-term and longer-term focus, and also include some referral services. Short-term Employment-Related services include services such as employment counselling (e.g. one-on-one training to enhance interview skills, job search and resume writing skills), resume screening and matching services, and networking opportunities at job fairs, “meet and greets” and various types of information sessions. Longer-term services include employment-related mentorships, services to help clients prepare for credential recognition or licensure, and work placements (including paid or unpaid internships), practice firms (workplace simulation) and other direct work experience opportunities. To access Employment-Related services, clients generally need to have at least basic proficiency in one of Canada’s official languages.

According to IRCC administrative data, a total of 31,386 Settlement clients, 18 years of age or older, received Employment-Related services in FY 2015/16.^{Footnote 72} Most obtained services with a short-term focus (88%), primarily employment-related counselling (81%). Networking opportunities was the second most frequent type of short-term Employment-Related service, accessed by 36% of clients. Considerably fewer clients accessed Employment-Related services with a longer-term focus: only 6% of clients accessed mentoring services and work placement opportunities. Lastly, 11% of clients received referrals to other non-IRCC services related to employment.

5.7.1. Clients Acquire Knowledge about Working in Canada

The SCOS was used to assess the extent to which Settlement clients obtain knowledge about working in Canada. Clients were asked to rate their knowledge of several topics related to the Canadian work environment, including looking for a job (i.e., job searches, prepare resume, participate in an interview, etc.); establishing contacts, connections and networks with others; getting educational and/or professional qualifications assessed; using “soft skills” or “people skills” in the workplace; using computers to create documents, send emails and do other work-related tasks; and acting on their rights and responsibilities as a worker (e.g., health and safety, employment insurance, holidays, etc.). These elements were grouped together to form one factor: “Knowledge about Working in Canada”.^{Footnote 73}

The majority (87%) of Settlement clients reported some knowledge of the Canadian work environment. About half (52%) felt they had quite a lot of knowledge (37%) or a great deal of knowledge (16%) about the work environment. The remainder reported having some knowledge (35%), not very much knowledge (12%) or no knowledge at all (2%).

More detailed analyses were conducted to assess the impact of specific IRCC-funded Settlement services on clients' reported knowledge about working in Canada, isolating different characteristics. Of IRCC-funded services, Employment-related services had by far the greatest impact on clients’ level of knowledge about working in Canada. Information and Orientation services as well as Community Connection services also positively impacted clients' reported knowledge about working in Canada, although to a lesser extent. Clients who have not taken Language Training reported more knowledge about working in Canada than those who have taken these services.

5.7.2. Clients Participate in the Canadian Labour Market

The Settlement Client Outcome Survey was used to assess the incidence of employment among Settlement clients by asking all respondents if they were working at the time of the survey.^{Footnote 74},^{Footnote 75}

The majority (62%) of survey respondents were working at the time of the survey.

• Of those who indicated working, 73% of Settlement clients indicated working full time, while 27% were working part time at the time of the survey;
• 12% were self-employed;
• 86% indicated holding one job at the time of the survey, while 15% were employed in more than one job; and,
• 12% of Settlement clients indicated having been hired for a job in Canada before coming.

Comparatively, of the 38% who indicated not currently working:

• 43% indicated having worked in Canada in the past and 57% indicated never having worked in Canada in the past; and
• 53% were looking for employment at the time of the survey.

Further analyses^{Footnote 76} were conducted to assess the impact of specific IRCC-funded Settlement services on clients' employment status at the time of the survey, when taking into account other characteristics at the same time. Of IRCC-funded services, Employment-related services was the one Settlement stream that positively affected clients' chances of being employed at the time of the survey. Clients who took Language Training and Community Connection services were both less likely to be working at the time of the survey than those who have not obtained such services. Further analysis showed that the most important socio-demographic characteristics for being employed at the time of the survey were, by order of importance: country of citizenship, year of admission in Canada (i.e., length of time in Canada), and gender. Again, the more time immigrants have been in Canada, the more time it gave them to familiarize themselves with the labour market and to find a job. Results related to gender are aligned with other studies on economic outcomes of immigrants showing a higher incidence of employment for men, and may reflect that women are not working to take care of their families while men are working to support the family.

5.8. Connections to Communities and Public Institutions

Community Connections activities support the two-way process of integration and facilitate adaptation on the part of newcomers and their host communities. These services are intended to address vulnerabilities by removing barriers to full economic, social, political and cultural participation. Services can be grouped as Targeted Community Connections or Community based, and can include various activities such as group workshops in local community settings (e.g., after school programs, conversation circles, etc.), cultural visits and community events, etc.

According to IRCC administrative data, a total of 39,592 Settlement clients, 18 years of age or older, received at least one Community Connections service in FY 2015/16.^{Footnote 77} Clients obtained targeted matching more often than community-based services (26,867 versus 20,095 clients respectively).^{Footnote 78}

Increasing knowledge of Canadian society was the most frequent topic/focus of community-based Community Connections received, with 16% of all Community Connections clients accessing this service, followed by Access to local community services (11%). With regards to targeted Community Connections, Language learning was the most frequently provided to clients (13%), followed by Connecting with settled immigrants or landed immigrants (12%) and Access to local community services (12%).

Differences did not appear between men and women who accessed services, but differences emerged between immigration categories with regards to targeted Community Connections. Economic immigrants accessed services to Access the labour market most frequently (16%), while sponsored family accessed Language learning services (17%) and refugees most frequently accessed Access to local community services (17%).

5.8.1. Clients Have Connections to Communities and to Public Institutions

The SCOS was used to assess the extent to which Settlement clients felt a connection to their community and to public institutions. First, clients were asked to rate their sense of belonging to the local community where they live (city, town, neighbourhood), the province/territory where they live, and to Canada. These were grouped together to form the factor “Connections to Communities”.

Clients were also asked to rate the confidence (trust) they have in a number of institutions, namely the police, the justice system and courts, the school system, the political system (i.e., Federal parliament), banks, local businesses, the Canadian media, the health-care system, the Social security/social insurance system (e.g., social assistance, old age benefits, etc.), and the Canadian government. The answers to these questions were grouped together to form the factor “Connection to Public Institutions”.

The majority (88%) of Settlement clients reported having either a very strong (40%) or somewhat strong (48%) connection to communities in Canada. The remainder reported having a somewhat weak (10%) or very weak connection to their communities (2%).

In addition, the majority (73%) of Settlement clients reported having either a great deal of confidence (25%) or quite a lot of confidence (48%) in Canadian public institutions. The remainder reported having some confidence (25%), not very much confidence (2%), or no confidence at all (0.2%).

When compared to the results from the General Social Survey on the Canadian population, Settlement clients tended to report about the same or a higher level confidence in public institutions. Based on the 2013 General Social Survey on Social Identity, three in four Canadians (76%) have either a great deal or some confidence in the police, making it the institution with the highest level of public confidence. Next highest were the school system (61%), banks (59%), and the justice system and courts (57%).^{Footnote 79}

Results of more detailed analyses indicated that there were no statistical differences observed between clients and non-clients of each of the four streams of IRCC-funded Settlement services, meaning that clients who have received IRCC-funded Settlement services from each of the different program streams do not show greater connections to communities and to public institutions than those who have not taken the services from each of those program streams.^{Footnote 80} For example, clients who received community connection services did not show greater connections to communities and to public institutions than those who did not receive community connection services. The same was true for those who received Information and Orientation services, Employment-related services and Language training.

Refugees reported a higher sense of belonging to communities than those admitted as Federal Skilled Workers, suggesting that the primary reason for immigrating to Canada may impact on the extent they feel connected to their new communities. Sense of belonging also increases with time spent in Canada.

Results related to age indicated that immigrants who were older at the time of the survey had greater confidence in public institutions than younger individuals. These results are aligned with what was found in the General Social Survey in 2013 where older Canadians generally had the highest levels of confidence in government and institutions.^{Footnote 81}

5.9. Clients Participate in the Broader Community

The following questions from the SCOS were grouped together to form the factor “Participate in the Broader Community”:

In the past 12 months, were you a member or participant in any of the following in Canada: (Check all that apply)

• A youth organization
• A cultural, educational or hobby organization (such as a theatre group, book club or bridge club)
• A school group, neighbourhood, civic or community association (such as PTA, alumni, block parents or neighbourhood watch)

Close to a quarter (22%) of clients reported participating in at least one of the three types of organizations identified above. However, the majority (78%) of Settlement clients reported not participating in any of the three types of organizations.

Results from the 2013 General Social Survey indicated similar results, where 20% of Canadians reported participating in a cultural, educational or hobby organization, a little under 20% in a school group, neighbourhood, civic or community association (17%), and 5% in a youth organization.^{Footnote 82}

Additional analyses were conducted to assess the impact of specific IRCC-funded Settlement services on clients' participation in the broader community, when taking into account other characteristics at the same time. Of IRCC-funded services, Community Connections services is the only Settlement stream that positively impacted clients' chances of participating in an organization. There were no statistical differences observed for the IRCC-funded Information and Orientation services, Language Training and Employment-related services, meaning that clients who have received these IRCC-funded Settlement services were not more likely to participate in community organizations than those who received Community Connections services.

5.10. Making Informed Decisions about Life in Canada

The SCOS was used to assess the extent to which Settlement clients are comfortable making informed decisions about life in Canada. The following survey questions were grouped together to form the factor “Making Informed Decisions”:

Thinking about all the information you have learned about Canada, to what extent do you feel comfortable making decisions about your life in Canada regarding each of the items listed below:

• Health care (e.g., making decisions about when to see a doctor or dentist, choosing a doctor or dentist, etc.)
• Money and finance (e.g., making decisions about investments, whether or not to take out a loan, etc.)
• Education (e.g., making decisions about where to send your children to school, whether or not to enroll in courses, etc.)
• Housing (e.g., making decisions about where to live, what type of housing you need, whether to buy or rent, etc.)

Just over half (52%) of Settlement clients reported being either extremely comfortable (12%) or very comfortable (40%) making informed decisions about life in Canada. The remainder reported being somewhat comfortable (38%), not very comfortable (9%) or not at all comfortable (0.6%).

Additional analyses were conducted to assess the impact of specific IRCC-funded Settlement services on clients' comfort making informed decisions about their life in Canada, when taking into account other characteristics at the same time. Of IRCC-funded services, Employment-related services was the only Settlement stream that positively impacted clients' comfort making informed decisions about their life in Canada. While there were no statistical differences observed for the IRCC-funded Community Connections services, clients who did not take Language Training and Information and Orientation services reported more comfort making informed decisions about their life in Canada than those who have received such services.

Analysis showed those who had been in Canada the longest reported a higher degree of comfort making informed decisions about life in Canada. In addition, knowledge of official languages may help in gathering the information needed to make informed decisions about various aspects of life in Canada, partly explaining why those who knew at least one of Canada’s official languages were more confident making decisions.

5.11. Impact of Taking Multiple Settlement Services

One of the expected results of the Settlement Program modernization was to strengthen links between Settlement services and improved client outcomes. With this view, additional analyses were conducted to assess the effects of taking combinations of IRCC-funded Settlement services for each of the expected client outcomes.

The following combination of services were found to provide additional contribution to the achievement of improving language skills and gaining more knowledge about working in Canada:

• Taking both IRCC-funded Language Training and Employment-related services had a greater impact on language skills improvement than Employment-related services or Language Training alone.
• Taking both IRCC-funded Language Training and Employment-related services had a greater impact on knowledge about working in Canada than Employment-related services or Language Training alone.

However, although this combination of services brought an additional impact to clients towards the achievement of two targeted program outcomes, there were no observable differences from other combinations or by simply taking a greater number of Settlement services – which suggests the greatest impacts were specifically from Language Training and Employment-related services.

6. Service Delivery Enablers – Support Services and Local Immigration Partnerships

Both Support Services and Local Immigration Partnerships (LIPs) are expected to enable and/or enhance the delivery of services for newcomers.

Support Services are designed to enable clients to access IRCC-funded direct Settlement services. IRCC provides funding to service providers for six different Support Services (i.e. care for newcomer children (CNC), transportation, translation, interpretation, crisis counselling and provisions for persons with disabilities) to enable eligible clients to access IRCC-funded direct services.

Local Immigration Partnerships (LIPs) are community-based partnerships designed to:

• Foster a systematic approach to engage SPOs and other institutions to integrate newcomers;
• Support community-based knowledge-sharing and local strategic planning; and,
• Improve coordination of effective services that facilitate immigrant settlement and integration.

LIPs do not provide direct services to newcomers.^{Footnote 83} Rather, they seek to increase the absorptive capacity of host communities by engaging a range of stakeholders including employers, school boards, health centres, levels of government, Service Provider Organizations, professional associations, ethno‐cultural and faith-based organizations, and the community and social services sectors to enhance collaboration and strategic planning at the community-level. The ultimate goal of LIPs is to leverage new partnerships and community knowledge to adapt services to better serve newcomers.

Key results of the case studies on Support Services and LIPs are presented in this section.

6.1. Impact of Support Services

A case study of was conducted to assess the need for Support Services and impact of support services on the ability of newcomers to access Settlement services; the extent to which clients have access to support services, including gaps in access; and the cost of delivering them.

In terms of impacts, key informant interviews and focus groups with settlement clients both confirmed that support services were necessary for clients in regard to reducing some of the barriers clients face in accessing Settlement services as many newcomers, especially women, would not have been able to take language classes, attend appointments, workshops, or understand information being provided to them without support services. SPOs rated all Support Services as “very important” to access Settlement services and identified them as an important enabler to deliver Settlement services.

In terms of areas for improvement, key informants and a review of program documentation revealed that IRCC has had challenges defining crisis counselling as a Support Service and determining what should be funded. Key informants noted that there is a high level of need for mental health support for newcomers, and that need has been growing; however SPOs are often not equipped to deal with the mental health issues that newcomers face. Results from the SPO survey and key informant interviews confirmed that of the roughly half (46%) of all SPOs that deliver crisis counselling, most due so as part of the normal delivery of Settlement services (e.g., by a settlement worker during a NARS or Information and Orientation session, by a language teacher during language training). However, key informants noted that in a few cases, SPOs have used funding for crisis counselling to hire specialized counsellors (e.g., social workers) to provide support to clients.

The case study did not identify any support services that were needed beyond the current six being offered and funded. While nearly all IRCC and SPO interviewees as well as focus group participants agreed that support services are important to access SPO services, some suggested that there is a need for newcomers to have access to support services beyond those funded by IRCC, particularly for translation, interpretation, and transportation to access community services. These individuals noted that while some SPOs attempt to fill these gaps by using volunteers or non-IRCC funding, the level of support is limited which presents challenges for many newcomers. For example:

• Interpretation and translation for medical-related services is one of the biggest needs, but there was also a need for this support to access services related to housing, school, government programs and banking.
• Transportation support to attend medical and school appointments was also identified as a large need for some clients.

In the survey of service providers, SPOs were asked about access gaps for each support service (i.e., more clients needing a support service than accessing it). About two-thirds of SPOs reported a need among their clients for long-term and short-term care for newcomer children and transportation (68% and 63% of SPOs, respectively). About two-thirds (60%) of SPOs reported having more clients needing crisis counselling services than accessing this type of Support Service, and a little over half the SPOs reported a gap in access for translation (53%) and interpretation (51%) services. The Support Service with the least amount of SPOs reporting a gap in access was disability support (42%).

An analysis of financial data for IRCC-funded support services revealed that of the $2.9 billion was expended on Settlement services between FY 2011/12 and FY 2015/16. Of this, 6% ($185.7M) was allocated to support services.^{Footnote 84} However, as the expenditures for interpretation, translation, crisis counselling, and disability support are recorded in the IRCC Grants and Contribution System (GCS) with “other program delivery costs”, the exact amount of funding that IRCC provides for these support services is unknown. Anecdotal information indicates that much of the funding for support services is allocated to Care for Newcomer Children (CNC), as it is the most expensive service to provide since it requires infrastructure and dedicated staff.^{Footnote 85}

Further findings and results of the case study on support services are contained in the technical appendices.

6.2. Impact of Local Immigration Partnerships (LIPs)

A case study of LIPs was conducted to assess the extent to which LIPs have successfully engaged stakeholders, developed and implemented strategic plans and improved service delivery in their communities, as well as the continued need for LIPs and the role of the Federal government in supporting these partnerships.

LIPs were able to engage diverse partners and stakeholders to coordinate service delivery. Key informants reported that they were successful in securing interest in the LIP from a broad range of partners, and obtaining some commitment from them to participate in LIP meetings and activities.

According to a survey of LIP coordinators:

• All LIPs included Settlement service providers on their main councils or working groups.
• All LIPs included mainstream service partners (such as hospitals, schools, justice services etc.). All LIPs also included regional or municipal government partners.
• Two-thirds of LIPs included provincial government partners and nearly half included federal government partners.
• The majority of LIPs (>85%) included employers or employment bodies on their main councils or working groups, however 70% indicated it was somewhat or a major challenge to secure their involvement in the LIP.
• More than 85% of LIPs included research/academic organizations or umbrella organizations within their partnership.

However, uncertainty about future funding was the single biggest concern cited by the LIPs, with two thirds noting it as a challenge, while many LIPs noted that uncertainty about funding inhibited the implementation of projects.

Most LIPs were reported to be conducting research on community needs and gaps as across all LIPs surveyed; 87% of LIPs had completed initial research and 85% of survey respondents indicated that the LIP had a moderate or strong impact on the development and sharing of immigrant-related research. Despite this, several LIP stakeholders noted that there are opportunities for greater sharing of information between IRCC and LIPs to disseminate research or leverage lessons learned across communities. These stakeholders expressed that there may be potential for information produced by LIPs to feed into policy-making or decision-making.

In terms of contribution to cultural competence and innovation, in a survey of LIP coordinators, 62% of LIPs reported a strong or moderate improvement in cultural competence among mainstream service providers, while 56% reported that LIPs facilitated the development of innovative methods or models for service delivery (see Section 7.3 for additional information on SPO partnerships). Examples of such innovations included an audit tool to access the accessibility of community services to newcomer customers, communal pot lucks for newcomers and community members and discount passes which provide free or discounted admission to a variety of cultural, arts and recreational organizations and programs for newcomers.

Finally, the growth in the number of LIPs between 2008 and 2016 proceeded without comprehensive application of standard IRCC criteria defining when a LIP should be funded. Although internal documentation from 2013 reviewed suggested four criteria for expansion,^{Footnote 86} in a few instances, LIPs have been funded in communities that have not met all of these criteria or have been able to meet these criteria during their first few years. For instance, in a few cases, LIPs have been funded in communities with very small newcomer populations. This creates challenges for these LIPs and places them at risk of venturing into direct services or services that do not specifically target newcomers. Second, in a few cases LIPs had not established working relationships with provincial/territorial/municipal governments which can make it difficult for them to implement projects or to leverage funding.

Further findings and results of the case study on LIPs are contained in the technical appendices.

7. Program Management

The evaluation assessed the extent to which program management outcomes have been achieved. As per the Settlement Program logic model, program development, management and governance are expected to support consistent and responsive program delivery and Settlement partners and stakeholders are expected to be engaged in settlement and integration strategies to address newcomers’ needs. These issues were explored via key informant interviews with program and external stakeholders. The evaluation also conducted a GIS mapping exercise to assess program reach and a survey of service provider organizations to identify key delivery challenges.

7.1. Reach of IRCC-funded Settlement Services

Settlement service locations (primary and itinerant^{Footnote 87}) and Settlement clients were mapped to help assess the geographic range of services provided through the Settlement program. Both iCARE data and the survey of SPOs confirmed that SPOs provide services using permanent and itinerant locations in all regions of the country.

In the survey of SPOs, a little over half (54%) indicated delivering IRCC-funded services at multiple permanent service locations (e.g., a main office and satellite sites), while a little under half (46%) reported having only one permanent service location. Many SPOs (65%) indicated offering services at itinerant locations (i.e. offer services, but not on a regular basis). Around a quarter of SPOs (27%) indicated being co-located with other organizations offering Settlement services, and 51% indicated being co-located with other organizations offering other non-Settlement services that are helpful to newcomers.

As indicated in Table 4, service level data from IRCC data systems from FY 2014/15 and FY 2015/16 reveals that service coverage is comprehensive, as IRCC-funded Settlement services were offered in 3,609 locations across the country.

Source: Settlement Client Continuum

Waiting for Language Training

Although service coverage is widespread, not all SPOs have capacity to provide Settlement services to all clients at the same time. As a result, there have been waitlists for some IRCC-funded Settlement services such as language training and support services such as childminding. During focus groups with clients, some clients expressed that they waited anywhere from 2 weeks to 6 months to receive a spot in language training (either due to a waitlist for language training, lack of available care for newcomer children, or both).

An internal study of wait times associated with IRCC-funded language training found that between January 1, 2015 and April 30, 2017, 11,221 clients, or 11% of those who received referrals to IRCC-funded or co-funded courses were waiting for language training.^{Footnote 88}

Multiple reasons exist for why a client cannot be placed immediately in a language class, including:

• No spaces available in classes offered by the client’s preferred SPO
• No spaces available at the appropriate CLB level or skill
• No spaces available at the client’s preferred schedule or timeslot
• Limited support services available (e.g., no childminding available, or transportation issues)
• The client has been placed in a class which is not their first choice, but is waiting for their preferred class.

In addition, it is important to note the difference between lack of available classes and clients’ personal reasons/preferences for not accessing language training – these are often conflated as overall ‘waitlists’.

While waitlists are not a new issue, the delivery of language training through the Settlement program is intended to be flexible, allowing for SPOs to place clients in classes suited to their needs. In addition, the recent Syrian Refugee Initiative has added an influx of refugees and has exacerbated the waitlist issue in some locations. However, the Department continues to review and validate the waitlists of SPOs with large numbers of clients waiting for language training and long wait times (i.e., more than 90 days) and is currently developing an action plan to reduce wait times.

7.2. Governance and Program Delivery

Governance and Coordination

Of the internal and external key informants who were able to speak about program governance, most seemed satisfied with the organizational structure of the Settlement Program within IRCC; however, a few respondents felt there was still a lack of clarity with regards to role definition (e.g., between national headquarters and regions, between policy and delivery, etc.). In particular, a few external respondents said their organizations coordinated well with IRCC at a national level, but less so with regional offices. Besides discrepancies noted between the national and regional levels, a few respondents noted contradictions between the operations and policy branches within IRCC which suggest a need for greater coordination. However, in the survey of SPOs, 86% reported to a large extent or to a moderate extent that coordination between IRCC and SPOs is effective.

Only IRCC interviewees were asked to comment on the committee structures. Half the respondents who answered this question indicated that committee structures, including the National Settlement Council, the Settlement and Resettlement Management Committee, and the specialized working groups were effective (e.g., effective information sharing, right people at the table, timing and frequency, etc.). However, a few respondents felt local managers should be more involved in the discussions, especially pertaining to operations.

Policy Guidance and Program Flexibility

There were mixed views regarding how clearly the Program is defined and understood by stakeholders (i.e., the various streams of direct services, what constitute eligible expenditures) as well as the balance of flexibility and consistency to support program delivery.

When asked specifically about the definitions of each of the six streams of the Settlement Program, SPOs most or at least half strongly agreed or agreed that the definition of the following was clear: language training (80%), language assessment (72%), information and orientation services (67%), needs assessments and referrals (63%), employment-related services (56%); and, community connections services (53%). When asked specifically about the definitions of each support services, generally at least half of SPOs strongly agreed or agreed that the definition of the following support services was clear: care for newcomer children (74%), transportation assistance (67%), interpretation services (61%), translation services, (59%), support for persons with disabilities (49%), and crisis counselling (42%).^{Footnote 89} These results suggest that there are opportunities to improve the guidance around definitions of Settlement Program streams and particular Support Services to ensure a clearer understanding among SPOs.

While IRCC interviewees agreed that there is sufficient flexibility within the program and stressed the importance of having some level of consistent service delivery across regions, external respondents noted that there was a need to revisit and refine the Settlement Program Terms and Conditions to provide more flexibility for service providers. A few external interviewees highlighted that the contribution agreements were too prescriptive, which may deter innovation. In the SPO survey, less than half (44%) strongly agreed or agreed that there is sufficient flexibility within the contribution agreement to facilitate service delivery.

Innovation

Key informants were asked about new or innovative approaches to Settlement Program delivery. Most of the respondents who answered this question expressed a need to dedicate funding for innovation and to define precisely what is meant by the term. On the other hand, a few respondents felt service providers had to innovate frequently to respond to new needs on the ground. Some respondents spoke of potentially fruitful ways to improve Settlement services through innovation such as:

• better defining the innovation stream or designing a special Call For Proposals with innovation as a focus;
• making better use of technology to reach newcomers in remote areas and overcome other barriers to service provision;
• new delivery strategies to provide some services/information directly to clients instead of them going to organizations for services; and,
• expanding cooperation with the private sector to provide resources through funds and private sponsorship programs.

7.3. SPO Partnerships and Coordination within Communities

Most SPOs described working closely with other partners. In the survey of IRCC-funded SPOs, 69% reported actively participating in a LIP while 13% reported participating in a Réseau en immigration francophone (RIF). Of these SPOs that participated in their local LIP, 90% reported indicated participating in LIP council / working groups and using research, analysis or other information produced by the LIP while 17% also indicated being the LIP Contribution Agreement holder.

For the SPOs that participated in a RIF, 88% indicated participating in RIF council / working groups, 78% reported using research, analysis or other information produced by the RIF while 23% of SPOs participating in RIFs also indicated being a RIF Contribution Agreement holder.

Beyond LIPs and RIFs, the majority of SPOs reported maintaining active partnerships with:

• community organizations (94%);
• other service provider / immigrant-serving agencies (91%);
• an association of service provider organizations / immigrant serving agencies (86%); and,
• municipal organizations (76%).

When asked about the benefits of these partnerships, the majority of SPOs reported that the partnerships they maintained have to a large extent or to a moderate extent:

• provided newcomers with access to a wider range of services (95%)
• helped their organization deliver services that better address the needs of newcomers (94%)
• provided an opportunity to bring forward a unified voice on issues related to newcomer settlement and integration (88%)
• helped other, non-settlement organizations understand the needs of newcomers (86%)
• contributed to their organization’s capacity to plan and coordinate Settlement services locally (82%); and,
• resulted in innovative approaches in service delivery (80%) to a moderate or a large extent.

However, SPOs indicated that collaboration has fallen short with regard to dealing with employment issues. Just over half (58%) of SPOs reported to a large extent or a moderate extent that collaboration with employers and industry has been effective in identifying solutions to newcomer employment issues while 57% reported to a large extent or a moderate extent that collaboration with employers and industry has resulted in more opportunities for employment.

In addition, when asked about collaboration in their community, over three quarters of SPOs reported to a large extent or to a moderate extent that collaboration between settlement organizations is effective in identifying solutions to newcomer integration issues (83%) and the collaboration between settlement and non-settlement organizations is effective in identifying solutions to newcomer integration issues (81%).

These results align with those LIPs surveyed that strongly agreed that partnerships and community collaboration have been helpful to support integration goals within their respective communities.

7.4. Performance Reporting

SPOs providing direct service delivery are required, under their contribution agreements, to submit the following to IRCC:

• Monthly client service data into iCARE (i.e., which clients accessed different IRCC-funded Settlement services and support services).^{Footnote 90}
• Financial claims typically every quarter, but can be required on monthly or semi-annual basis.^{Footnote 91}
• Narrative Reports, required at the same frequency as financial claim reporting, containing a mix of closed and open-ended questions on client needs, support services, service delivery challenges, group sessions, success stories, and partnerships.
• Annual Project Performance Reports^{Footnote 92} for each contribution agreement at the end of each fiscal year, containing a mix of closed and open-ended questions on points of service, non-IRCC inputs (e.g., volunteers), client and non-IRCC eligible clients served, client needs, client groups, support service gaps, service delivery factors, service delivery best practices, client feedback, success stories, partnerships, contributors, other beneficiaries.

In some cases, the Department may also request direct service delivery SPOs to carry-out project-level performance monitoring and assessment activities such as project evaluations or surveys.

For indirect service projects such as LIPs and RIFs, organizations are required to manually^{Footnote 93} submit Annual Performance Report for Community Partnerships (APRCP) which contain a mix of closed and open-ended questions that focus on the results of the activities under partnerships, with specific attention to outcomes for newcomers and communities.

Guidance on Reporting Requirements

When asked in the survey, SPOs generally felt that the guidance released by IRCC on how to report on Settlement and Support Services in iCARE was clear, with over 90% of SPOs reporting the guidance was somewhat clear or very clear for most service types. As shown in Table 5, the Settlement and Support Services with the highest share of SPOs reporting that the guidance on how to report was not at all clear were crisis counselling services (17%) and support for persons with disabilities (13%).

Source: SPO survey

SPOs were mostly positive about the APPR and Narrative Reports as 75% of SPOs surveyed either strongly agreed or agreed that the guidance from IRCC on how to report in the APPR and Narrative reports was clear. However, 25% of SPOs surveyed either neither agreed or disagreed, disagreed, or strongly disagreed which suggest that guidance on APPR and Narrative reporting could be clearer.

Overall Approach to Performance Reporting

Interviewees (both internal and external) noted performance reporting was an area that could be improved in order to better facilitate communication among stakeholders as well as better “telling the story” of the impacts the program has been generating. While IRCC provides SPOs with monthly iCARE reports based on the data previously inputted, just under half of SPOs (48%), strongly agreed or agreed that the information produced was useful for their organization.

In addition, some external interviewees expressed that the reporting process was too heavy, long, complicated or onerous to providers and suggested that there might be ways to streamline multiple reporting methods. When asked about reporting requirements in the survey of service providers, just over half of SPOs (57%) either strongly agreed or agreed that reporting requirements to IRCC are reasonable.

Finally, some interviewees (both internal and external) mentioned that the department has been too focused on outputs rather than outcomes and noted that newcomer surveys should continue to be used as a key source of data to assess outcomes. Similarly, external respondents raised concerns regarding a lack of understanding of the data available for use regarding the Settlement Program and a perception that data is not being shared consistently with stakeholders across the Settlement Program.

Other Data Challenges

Other issues noted during the course of the evaluation focused on the reconciliation of data between the various data systems:

• In some cases, all IRCC-funded activities for a SPO are listed under one single contribution agreement. In other cases, one SPO will hold multiple agreements, with multiple activities under each. As SPOs are required to report data for each CA and not as a single organization, it is difficult to analyze results in a consistent way. Especially as the number of Narrative Reports submitted by a SPO is variable and each cover a different time period, it is difficult to reconcile information at a national level.
• APPR, Narrative Reports and APCRP data fields are mostly qualitative which makes it challenging to conduct an analysis of the results at a national level. SPOs often interpret open-ended questions slightly differently, making it difficult to roll up results. In addition, APCRPs are only submitted in PDF format which presents additional challenges for the consistent analysis of results.
• There are some instances of duplication in terms of similar data collected in various systems which makes it difficult to reconcile information across systems. For example, many of the same fields are collected in Narrative reports and APPRs (e.g., information about support services and partnerships). In addition, some financial information is captured in GCS, SAP, APPR and Narrative reports, each in slightly different forms.

7.5. Gender Differences

A presented in Section 2.2., iCARE data indicated that women accounted for a larger proportion of Settlement clients than men as well as for each specific Settlement Program stream. In addition, women are greater users of support services such as care for newcomer children. Focus group participants and key informant interviews as part of the Support Services Case Study both noted that women are adversely affected in terms of accessing settlement services when care for newcomer children is unavailable, as in most cases, women will stay home to take care of children while men either access settlement services or seek employment opportunities.

In addition, when examining the client survey results by gender, some differences were observed between male and female Settlement Program clients in terms of the achievement of program outcomes. On the one hand, results indicated that male clients improved their language skills to a lesser extent than female clients, and were less likely to participate in community organizations than their female counterparts. On the other hand, male clients had more knowledge about the work environment, life in Canada as well as laws, rights and responsibilities, had greater connection to communities and public institutions, were more likely to be employed and felt more comfortable making informed decisions about life in Canada than female clients. The only two outcomes for which no gender differences between Settlement clients users were related to awareness of community and other resources, as well as official language use.

7.6. Cost of Providing Settlement Services

The costs to provide Federal Settlement services include contributions to other parties (i.e., SPOs, British Columbia and Manitoba)^{Footnote 94} and the administrative costs within IRCC associated with managing the Settlement Program.

Cost of Contributions

In terms of Settlement Program contributions, between 2011/12 and 2015/16:

• The total amount of contributions for the Settlement Program was $2.9 billion with an average cost of $581 million per year. As Settlement services were repatriated in Manitoba in 2013/14 and British Columbia in 2014/15, these costs were reallocated into the various IRCC cost categories in the years following repatriation.
• Over the 5-year period, of the 6 direct services, language training accounted for the largest proportion of Settlement Program contributions (31%).^{Footnote 95}
• The other direct services comprise smaller proportions of Settlement Program contributions: Information and Orientation (15%), Community Connections (10%), Employment-related services (8%), NARS (8%) and Language Assessments (3%).
• Administrative costs for SPOs, support services and indirect services accounted for 11%, 7% and 5% of the overall settlement contributions, respectively.^{Footnote 96}
• From 2011/12 to 2015/16, the three most significant changes (in terms of percentage) were for indirect Settlement services which grew by 88% (from $15M to $28M), Information and Orientation which grew by 73% (from $55M to $96M) other Settlement contribution costs which fell by 66% (from $14M to $5M). In terms of shares of the total cost of Settlement contributions, between 2011/12 and 2015/16, the various cost categories remained relatively stable with only minor fluctuations (e.g., ±2-3% for each category).

Source: SAP

As displayed in Table 7, when Settlement Program contribution costs are compared with the number of unique clients served by the Settlement Program, the cost per client has fallen by 18% from 2013/14 to 2015/16. Costs per client were also calculated for each immigration category by dividing the cost of providing settlement services by the average number of Settlement services used by each immigration category. In terms of specific immigration categories, between 2013/14 and 2015/16, refugees accounted for the highest cost per capita.

Source: SAP

An internal cost analysis conducted by IRCC which examined the year of admission of Settlement Program clients in 2014/15 found that the majority of Program costs (between 67-79%) are incurred during the first 5 years of being admitted to Canada, however this percentage varied by immigration category with Humanitarian and Compassionate immigrants incurring 79% of their costs within their first five years of being admitted to Canada compared to Family Class immigrants that incurred 67% of costs within their first five years of being admitted to Canada.

Source: SAP

Costs to Administer the Settlement Program

In addition to the funding provided to service providers, the IRCC Cost Management Model (CMM) provides information on the cost of salaries and employee benefits plan for IRCC and other federal government departments allocated to the Settlement Program. Specifically, in FY 2015/16, the Government of Canada resources associated with administering the settlement program was $47 million and 341 FTE positions within IRCC.

FTEs for other government departments are not tracked through this model.

Source: IRCC Cost Management Model, FY 2015/16

8. Conclusions and Recommendations

As newcomers face a variety of challenges when settling in Canada, there continues to be a need to provide services to support their integration. IRCC-funded Settlement services can be a key source of support that newcomers rely upon to gain information and skills necessary to integrate into the Canadian society and economy.

Settlement service usage has risen over recent years, as the number of clients accessing services has continued to grow, meaning the program has been able to provide settlement support to more and more newcomers along their integration path.

Overall, the Settlement Program has been effective in providing assessments of clients’ needs and clients are receiving appropriate referrals and services to meet their needs and settlement objectives.

The Settlement Program results story is diverse and nuanced, as the program is complex and there are a variety of factors that contribute to client outcomes. The array of program streams (interventions) and flexibility in how Settlement services are delivered can make it difficult to assess their full impact, especially when also considering that other factors (e.g., learner motivation) can also influence successful integration. Despite these challenges, the evaluation used multiple lines of evidence and robust analysis to demonstrate the impact of the Settlement Program on clients.

The evaluation found that the Settlement Program has contributed to different client outcomes with varying degrees of impact on different client types. This points to the need to ensure the proper balance among program components and to tie outcomes more specifically to the profile of different client groups.

Overall, Settlement clients reported improved language skills and frequent use of official languages outside their home. Moreover, close to 60% of the language training clients have demonstrated progression in their language skills. The impact of IRCC-funded Language Training varies by several factors: it has a greater impact on younger clients, as well as those with higher human capital (i.e., economic immigrants, those with a university degree, those with reported knowledge of English and/or French upon admission). Also, language training services had a greater impact on language skills improvement for economic immigrants (FSW, CEC, PNP) than for immigrants from other immigration categories (GARs, PSRs and Family Class).

When considering the effect of IRCC-funded services in isolation, employment-related ones had the greatest impact on language skills improvement and use; whereas language training played a smaller role in language skills improvement and use. However, the positive impact increased significantly when taking both Employment-related services and Language Training together, and analysis also showed that taking ‘occupation specific’ Language Training resulted in greater impacts for clients.

As the evaluation found that the impact of IRCC-funded Language Training varies by specific sub-groups and client characteristics, there is a need to further assess its relative impact while taking into consideration other success factors and approaches. Language training is helping newcomers improve their language skill, but not in the same way for all clients and not all skills (i.e., reading, writing, listening, speaking). Examining settlement clients’ language learning needs, motivations and objectives, as well as the effectiveness of different language training delivery models used by service providers, will allow the Department to better determine what works for who and under what conditions and adjust its language training accordingly.

Recommendation 1

IRCC should review and assess its language training delivery and implement appropriate changes to improve its effectiveness. The assessment should:

• consider the needs of different groups of learners, and respective determinants of success;
• build on the strengths and weaknesses of existing approaches (curricula, modes of training, etc.);
• leverage best practices from adult education theory and practice, and the field of teaching English and French as a second language to adults; and,
• consider new and innovative approaches to language training for adult immigrants.

The evaluation showed that Employment-Related services have a positive impact on many client outcomes, including, among others, improved language skills and use, gaining knowledge of the Canadian work environment, and helping clients learn more about life in Canada. Employment was also the primary area where a majority of clients identified a need and reported wanting more information and services. Despite this, Employment-related services was the least-accessed service by clients.

Positive impacts from employment-related were also shown in occupation-specific Language Training, as these clients improved their language levels at higher rates compared to other language learners.

Given the many benefits this particular stream provides to Settlement clients, the need among clients, and the importance of employment to integration, maximizing Employment-related services should be a key area of focus when considering program improvements.

Recommendation 2

IRCC should develop and implement a plan to optimize the benefits of its Employment-Related Services and employment-specific language training.

One of the major strengths of the Settlement Program design is its flexibility and accessibility – services, in any combination and for any duration, are available until clients they obtain Canadian citizenship (at which point they become ineligible).

Client needs and settlement objectives vary greatly, and the evaluation found that clients from different categories and with characteristics were taking differing amounts of time to achieve similar results. This was most apparent in language training, for example, where the number of hours required to improve language skills was lower for economic immigrants than refugees. Analysis also showed that there are diminishing returns to language training beyond a certain number of hours, and that over one quarter of all Settlement Program clients who accessed services in FY 2015/16 had been in Canada for five years or longer.

As such, there should be consideration given to exploring the prioritization of and access to services for clients, while balancing needs and costs.

Recommendation 3

IRCC should review access to and duration of Settlement services and implement appropriate changes that achieve a balance between meeting the specific needs of different clients and available resources.

Overall, there is a need for Support Services, as they enable clients to access the Settlement services necessary to assist in their integration journey. Certain groups of settlement clients have a continued and growing need for support services (especially refugees and those with lower levels of education) to access IRCC-funded Settlement services, and some newcomers also need certain support services to access community or mainstream (non-Settlement Program) services that are not funded by IRCC.

The evaluation found that support services provided through IRCC are generally appropriate, however there are challenges with providing support for mental health issues through crisis counselling and in addressing clients’ support service needs beyond what IRCC funds.

While SPOs rely primarily on IRCC funding to provide certain support services, they use other resources, such as volunteers to provide support services to clients (especially interpretation and translation). While evidence indicates that the overall cost to SPOs of providing support services exceeds the amount of funding provided by IRCC, it is not possible to know the full cost of providing support services as they are not all tracked in a consistent manner.

Recommendation 4

IRCC should clarify the Department’s expectations regarding the provision, use and reporting of Support Services. Specifically, the Department should review and clarify:

• the approach to the use of specific Support Services to access community services not funded by IRCC, and update and promote guidance as appropriate;
• the expectations regarding the provision of crisis counselling and clearly articulate what SPOs should be providing to clients as part of this particular Support Service; and,
• the financial and reporting requirements regarding Support Services to determine what needs to be tracked, and subsequently develop and issue new guidance and adjust financial reporting procedures as needed.

LIPs have broadened the collaboration on, and profile of, newcomer issues in most communities by effectively engaging non-traditional newcomer service providers (“mainstream” services). LIPs have made notable achievements, particularly in the areas of locally-relevant research, information sharing, partnership-building, and strategic planning, which has resulted in a greater focus on newcomers and their issues in many communities. However, the current funding criteria and required structure for LIPs may not be the appropriate model for certain communities. Challenges for LIPs in securing sustainable funding pose a risk to implementing strategic plan activities and projects. Finally, although LIPs have generated a lot of new knowledge and research on newcomer issues, there is room for greater sharing of information between them and other LIPs, as well as IRCC.

Recommendation 5

IRCC should develop and implement a strategic plan to make best use of the potential contributions of Local Immigration Partnerships (LIP) to settlement and integration outcomes of immigrants and refugees. This plan should at the least:

• articulate its vision, expectations, rationale and role for supporting LIPs;
• confirm and communicate the criteria for funding new and existing LIPs; and,
• facilitate networking and knowledge sharing among LIPs by leveraging and disseminating research results and best practices generated by LIPs.

There is a wide array of detailed information on the Settlement Program collected by the department through its various data systems. While SPOs were generally positive in terms of guidance received from IRCC in terms of reporting, there were also some key challenges and limitations noted in this area.

The way in which some of the data systems have been designed makes it difficult to reconcile information between the various systems in a timely fashion for consistent performance reporting. Often these systems differ in terms of the unit of analysis or reporting period covered, while some information is captured in several different systems, each in slightly different forms.

There are opportunities to examine the various data systems as a whole to ensure they work well together by allowing the department and external stakeholders to obtain the necessary information needed to support decision making while reducing reporting burden at the same time.

Recommendation 6

IRCC should review its reporting requirements and systems for the Settlement Program and implement changes to streamline data collection and enhance analytical tools as required.

As part of this evaluation, a Settlement Client Outcomes Survey was developed, which allowed the Department to gather a significant amount of client outcomes-related data for the first time. This data, particularly when linked with client demographic profiles and services usage, formed the basis of extensive analysis in the evaluation and will also provide the department a valuable source of results information on Settlement Program clients. The data collected also enables rich reporting and an assessment of the contribution of IRCC-funded services towards client outcomes. Repeating the survey and using a comparable tool would allow the Department to augment its capacity to measure client outcomes regularly in support of ongoing program and policy analysis and design. In addition, expanding the survey to non-clients would provide an even stronger comparison group against which to measure Program effectiveness.

Recommendation 7

To strengthen its client outcomes data, IRCC should administer an outcomes survey on an ongoing basis to settlement users and non-users.

Appendices

A set of Technical Appendices have been prepared to support the Evaluation which are available upon request. The list of appendices are listed below.

• Appendix A: Client and Service Profiles, FY 2015/16
• Appendix B: Settlement Program Logic Model
• Appendix C: Complete Client Survey Regression Results Tables
• Appendix D: Detailed Findings from the Support Services Case Study
• Appendix E: Detailed Findings from the LIP Case Study
• Appendix F: Evaluation Matrix