Evaluation of the Skills and Partnership Fund

Project design and delivery

Partnerships: SPF agreement holders worked with public, private, and non-profit sector partners to design and deliver interventions, and to connect participants to work experience and employment opportunities. Partners generally also committed cash and in-kind contributions to the projects. In fact, half of the projects funded through the 2016 call for proposals indicated that a quarter or more of project resources would be provided by partners.

Objectives: while the ultimate objective of most projects was to help participants to obtain and maintain employment, most also sought to support pre-employment outcomes for a large portion of their participants. In other words, the immediate outcome sought for many participants was increased employability.^{Footnote 2}

Sectors targeted by SPF projects: the most common sectors targeted for training and employment were trades (60%), construction (38%), and mining (30%).

Participants: The program reached individuals in rural or remote areas (58%), and in urban areas (40%). Most participants in SPF interventions were status First Nations (75%), male (64%), and single (62%). Participants had relatively low educational attainment; over a quarter (29%) had not completed high school and only 1% had obtained a university degree.

Post-participation labour market outcomes

The outcome analysis focused on the post-participation labour market outcomes of the 5,003 participants who started an SPF-funded intervention in 2017 and 2018.^{Footnote 3}

Outcomes should be interpreted with caution as they may be influenced by broader trends and shifts in the labour market and economy. Because participant outcomes were not compared to those of non-participants, they cannot be attributed solely to program participation. The post-participation period overlapped with COVID-19 disruptions to the economy and labour market, and with pandemic response measures that may have influenced participants' employment decisions and use of income supports. Moreover, the analysis captures the outcomes of the first cohorts of participants following the launch of the projects; the outcomes of these first cohorts may not be representative of overall participant outcomes

Overall participant outcomes

Participants improved their labour market attachment after participating in SPF-funded interventions:

• they increased their earnings by $4,150 per year
• the proportion of participants earning less than $10,000 decreased by 7.4 percentage points while the proportion earning more than $30,000 increased by 6 percentage points
• they increased their incidence of employment by 4 percentage points

Outcomes by intervention type

Participant outcomes varied across different types of interventions:

• Skills Development Apprentices had the strongest increase in incidence of employment (+12 percentage points), while Targeted Wage Subsidies had the highest increase in earnings (+$10,494)
• of the 2 types of interventions taken by participants with weaker labour market attachment (Essential Skills Training and Job Creation Partnerships), only Essential Skills Training had stronger than average earnings outcomes

Outcomes by type of participant

Of the 3 participant types (active EI claimants, former EI claimants, and non-EI claimants), non-EI claimants, who accounted for 62% of participants, had the strongest outcomes; they increased their incidence of employment by 14 percentage points and their earnings by $7,089.

Outcomes by Indigenous group

Non-status First Nations, Inuit, and Métis participants had relatively strong outcomes, while those of status First Nations were much lower.^{Footnote 4} Prior to participation, status and non-status First Nations had similar levels of labour market attachment, but their post-participation outcomes differed:

• the earnings of non-status First Nations increased by $10,860 while those of status First Nations increase by $1,692
• the incidence of employment of non-status First Nations increased by 24 percentage points while that of status First Nations decreased by 0.5 percentage point

Outcomes by gender

Women had much weaker outcomes than men:

• women increased their earnings by $2,253 while that of men increased by $5,455
• the incidence of employment of women decreased by 0.5 percentage points while that of men increased by 7 percentage points

Outcomes by age

The earnings and incidence of employment of older participants decreased, while those of younger participants increased. Participants who were aged 30 or younger increased their earnings by $9,430 and increased their incidence of employment by 15 percentage points.

Outcomes by type of location

The outcomes of participants in rural and remote areas were positive but much lower than those of participants living in urban areas:^{Footnote 5}

• participants in rural and remote areas increased their earnings by $2,667, compared to $6,110 for those living in urban areas
• the incidence of employment of participants in rural and remote areas decreased their 2 percentage points, while that of participants in urban areas increased by 12 percentage points

Broader project impacts

The evaluation identified several impacts that extended beyond program participants to their communities. Broadly, these included:

• supporting communities' economic, human resources, social and cultural development
• helping participants to access employment opportunities without having to move away from the community
• creating a sense of opportunity and possibility for community members, and
• strengthening families and households

Challenges and promising practices

The evaluation identified a variety of challenges that hindered project delivery and outcomes:

• project proposal and administrative processes can be difficult to manage for organizations that are not well-established or that have limited human resources capacity; providing support prior to calls for proposals and support throughout the duration of the project can be helpful
• many projects were forced to rethink their training or service delivery approach in response to COVID-19 related disruptions and response measures, such as lockdowns, social distancing, or closing communities to travel; the move to online delivery disadvantaged those who did not have access to highspeed internet, sufficient bandwidth, or computers
• staff turnover was a common challenge; for those in northern and remote communities, where it is often necessary to recruit individuals who do not live in the area, the loss of an employee or instructor can have severe impacts on project delivery and outcomes
• while working in partnership allows many projects to achieve positive outcomes, when partnerships do not proceed as planned or dissolve, the impacts on project delivery and outcomes can be severe; for agreement holders in rural and remote areas where there are fewer partnership opportunities, finding new partners can be challenging

The evaluation also identified a number of promising practices:

• the most common response to COVID-19 related disruptions was to move service delivery and training online; while this approach was not appropriate for all context and types of interventions, it provided many projects with the opportunity to reach more participants^{Footnote 6}
  • in some cases, agreement holders were able to provide internet access and computers to participants
• a common thread among projects that had positive outcomes working with multi-barriered participants was the provisions of ongoing support throughout their path to employment; the agreement holders leading these projects emphasised that paths to employment are often not linear and that participants may need help to overcome a broad range of barriers

Performance measurement and data assessment

The program follows the best practice of collecting information at the individual level in a manner that enables the integration of program data with other administrative data sources and, potentially, with survey data. This approach supports the validation of short and medium-term post-participation employment outcomes while mitigating the administrative burden on agreement holders.

Agreement holders' ability to collect and report data were, at time, hindered by pandemic related disruptions, such as office closures. Despite these challenges, collected administrative data was found to be of sufficient quality and integrity to support the creation of participant profiles and post-participation labour market outcome analysis. Moreover, the data is of sufficient quality and integrity to support incremental impact analysis based on a 5-year post-participation period when the required taxation data becomes available in the 2027 to 2028 fiscal year.

Northern and remote agreement holders and projects

The data assessment and document review identified contextual challenges and vulnerabilities relating to the organizational capacity of agreement holders located in northern and remote regions and to the limited partnership opportunities present in northern and remote areas.

Recommendation 1: ESDC is encouraged to explore options and take actions to offer further support to SPF agreement holders in rural and remote areas through each stage of the projects.

Data collection and performance measurement

The program is following the best practice of collecting individual-level data. The data collected and reported for projects funded through the 2016 call for proposals was of good quality and integrity overall. The document review and the literature review, which included reports summarising Indigenous partner engagement findings, found that the data collection and reporting requirements continue to be challenging for some agreement holders.

Moreover, some projects reported on the cash and in-kind contributions they received from their project partners throughout the project. Some also reported on their experience of working with their partners, including the challenges and benefits. This information, particularly when collected across all projects, can be valuable for program and project design and delivery, program management, program evaluation, and performance measurement.

Recommendation 2: ESDC is encouraged to explore options and take actions to:

• continue to prioritize data integrity, including validating data uploads, and providing support to projects experiencing data collection and reporting challenges; and
• collect consistent partner contribution data across all projects

Management Response

Management agrees with this recommendation. Employment and Social Development Canada recognizes that some organizations, particularly in more rural and remote communities, may have additional barriers to applying for funding through Call for Proposals (CFP) processes and in implementing projects. The program and officials will continue to explore how to better support potential project applicants and recipients throughout the project life cycle as part of the ongoing SPF renewal process.

Management Response

Management agrees with this recommendation. Employment and Social Development Canada is committed to collecting and maintaining quality program data to support practical outcomes analysis for program participants. The report indicates that the program is following the best practice of collecting individual-level data and that the data collected was deemed of good quality and integrity overall. The program intends to continue to ensure data integrity and will seek to work proactively with organizations experiencing data collection and reporting challenges.

Lines of evidence

Outcome analysis

The outcome analysis captures the post-participation outcomes of participants who began an SPF intervention in 2017 and 2018. These participants include the first cohorts of participants following the launch of the projects.^{Footnote 11},^{Footnote 12} The outcomes of the first participant cohorts may not be representative of overall participant outcomes.

Outcome analysis provides an assessment of changes in participants' labour market outcomes over time. However, outcome analysis does not use comparison groups to compare the outcomes of participants to those of similar non-participants. Accordingly, it does not measure to what extent changes in participants labour market outcomes can be attributed to the intervention they received.

Labour market outcomes are influenced by changes in the economy, inflation, and the labour market. Several contextual factors should be taken into account when interpreting participants' labour market outcomes. The post-program participation period overlaps with the COVID-19 pandemic and its disruptive impact on the labour market. Moreover, several temporary income support measures were introduced during the pandemic that may have influenced participants' use of income supports and employment related decisions. These measures included, but were not limited to:

• the Canada Emergency Response Benefit from March 2020 to September 2020
• the Canada Recovery Benefit from September 2020 to October 2021

Moreover, available administrative data does not include income supports provided by Indigenous Services Canada through the On-reserve Income Assistance Program.

Accordingly, outcomes relating to the use of EI, use of SA, and the proportion of participants' incomes from government income supports should be interpreted with caution.

To mitigate these limitations, a follow-up study on impacts should be undertaken for this cohort of projects when the data required becomes available.^{Footnote 13}

Assessing partner contributions

Some, but not all, projects reported on the cash and in-kind contributions they received from their project partners, and some also reported on their experience of working with their partners. For these projects, the reported data enabled:

• the tracking of partner contributions
• the identification of reliable and committed partners, and
• the identification of challenges and benefits of working with partners

Because partner contribution data was not consistently available in the project reporting documents, it was not possible to confirm the full extent of partner contributions across all projects.

Key informant interviews and case studies

The evaluation took place 1 to 4 years after the end of projects funded through the 2016 call for proposals. During this interim, agreement holder and project partner staff who had worked on SPF projects had begun working on other projects or had left the organization. This increased the challenge of scheduling interviews with SPF agreement holder staff who recalled the projects in detail. As a result, fewer interviews were completed than originally planned.

Engaging the partners of SPF agreement holders in evaluation activities proved to be challenging. While agreement holders are expected to participate in evaluation activities, project partners are not and have little incentive to do so. As a result, most of the key informant interviews conducted were with agreement holders, meaning that the perspectives of project partners are less well represented.

While the case studies provide valuable insight, they are not representative of the projects funded through the 2016 call for proposals. Overall, they include projects that:

• targeted participants who were further along in the employment integration continuum
• had private sector partners and received a large share of their project resources from partners
• had lower costs per participants, and
• had stronger employment outcomes

More information on the characteristic of the 5 case study projects and on how these compare to the average across the 52 SPF projects are provided in Appendix C.

Due to the challenges encountered in collecting qualitative data directly from agreement holders, the evaluation included more information collected by ESDC from agreement holders as part of program administration activities. In particular, the evaluation incorporated findings drawn from a review of the funding agreements and reporting documents of the 52 funded projects.

Labour market development and economic reconciliation

In addition to ongoing labour market development priorities, the federal government has identified several priorities intended to support economic reconciliation with Indigenous Peoples.^{Footnote 29} These include working with Indigenous communities and organizations to:

• support Indigenous Peoples to increase their educational attainment
• support Indigenous Peoples' participation in the economy, and
• increase the benefits that Indigenous communities derive from major projects in their territories

The SPF supports these government priorities by helping Indigenous individuals to:

• improve their educational attainment and labour market attachment, and
• build partnerships that enable them to benefit from emerging economic opportunities and major projects

Labour market development needs and barriers to employment

The program responds to the labour market needs of Indigenous Peoples and to some of the barriers to employment that they face.

The socio-economic profile section of this report identifies inequities between Indigenous and non-Indigenous Canadians with respect to their educational attainment, employment rates, unemployment rates, and average annual income. Moreover, while the educational and labour market outcomes of all Canadians worsen as remoteness from urban areas increases, the impact of remoteness is more pronounced among Indigenous Peoples.

With respect to barriers to employment, the most common barriers identified by participants were lack of work experience, low educational attainment, and lack of marketable skills.^{Footnote 30} Other common barriers included lack of transportation and living in a rural area with few job opportunities. For women in particular, lack of access to dependent care was a common barrier to employment.

The SPF responds directly to these needs and barriers by supporting Indigenous-led projects that:

• support participants in their efforts to increase their level of education and training
• support participants to increase their employability and work experience
• help connect participants to existing and emerging job opportunities and to increase their employment income
• help communities and Indigenous service providers to build and maintain working relationships with employers, and
• reach Indigenous individuals who live on-reserve, in rural areas, and in remote communities

Finally, the SPF seeks to respond to the needs of multi-barriered individuals by providing the flexibility to design programs and services that:

• respond to a broad range of barriers, such as access to childcare and transportation, resources to buy basic equipment, and access to mental health supports, and
• allow projects to support participants who are further from the labour market as they progress along the path to employment

Relevant and informed program design

In their complementarity, the SPF and the ISET Program reflect lessons learned from previous evaluations of ESDC labour market integration programs for Indigenous Peoples and input provided by Indigenous partners.

A previous evaluation of Indigenous labour market programs funded through ESDC yielded important lessons relating to project-based funding for employment programs:^{Footnote 31}

• projects focusing on specific industrial developments, such as a particular mining project, yield stronger employment results than projects with a broader focus, such as the mining sector in general
• partnering with employers and tailoring training to their needs creates a training-to-employment path for project participants yielding stronger outcomes for participants and employers
• partnerships with greater employer engagement, particularly those including formal agreements for in-kind or cash contributions, yield stronger employment results, as employer partners are more invested in the outcomes

Engagement with Indigenous partners also yielded valuable insight relating to the design and delivery of labour market training programs for stronger, broader, and longer lasting impact:^{Footnote 32}

• longer-term funding supports the maintenance and development of experienced service providers with knowledgeable staff who produce strong results
• stronger connections are needed between Indigenous service providers and potential employers in local industries to:
  • better align training opportunities with in-demand skills and long-term employment opportunities
  • help develop the local economy and help participants to stay in their communities with their families
• greater flexibility is needed for service providers to design programming to meet the complex needs of their people and communities, such as:
  • pre-employment interventions to increase employability, such as life and essentials skills training
  • wraparound services and individualised supports
  • supports to complete and skills up-grading to move beyond entry-level positions

Overall, with ISET supporting a stable network of experienced service providers across the country, the SPF aims to respond to these lessons learned and insights by providing timely funding to develop strategic partnerships with employers for training-to-employment projects and to test flexible approaches to addressing complex issues through community-based service delivery.

Key informants interviewed as part of the evaluation added that the SPF is part of a broader ecosystem of programs aiming to improve the labour market integration of Indigenous Peoples in Canada. While these programs have similar objectives and offer similar types of interventions, they differ in their approach. Key informants specified that the SPF contributes to the broader effort by supporting the development and maintenance of cross-sectoral partnerships, including employers, training institutes, and post-secondary education institutions to take advantage of existing and emerging employment opportunities; these findings were further confirmed through the document review.

SPF agreement holders

Most of the projects were led by Indigenous bands, councils, and governments (42%), Indigenous led non-profit organizations (33%); 15% were led by Indigenous education institutes, and 10% were led by Friendship Centres.

Agreement holders were located in each of the 13 provinces and territories. However, 70% were located in 4 provinces and territories: Ontario (31%), BC (19%), the Northwest Territories and Quebec (10% each).

Text description: Figure 4

Map of Canada showing the location of SPF projects and table showing the number of projects per province and territory.

Province or territory	Number of projects
Alberta	2
British Columbia	10
Manitoba	3
New Brunswick	1
Newfoundland	2
Northwest Territories	5
Nova Scotia	1
Nunavut	2
Ontario	16
Prince Edward Island	1
Quebec	5
Saskatchewan	3
Yukon	1

SPF funding streams

Of the 52 projects funded through the 2016 call for proposals, 60% were funded through the Innovation Stream and 40% through the Training to Employment Stream.

Scope and objectives

The objectives and intended outcomes of each project were identified in their funding agreements. While all were ultimately aimed at improving participants' labour market participation, most projects included progress along the pre-employment continuum as an immediate project outcome for a large portion of their participants.

Overall, the most common immediate outcomes sought by projects were:

• participants find employment (98%)
• participants complete occupational skills training (80%)
• participants reach the pre-employment outcome of increased employability (76%)
• participants complete pre-employment training (64%)

Populations targeted by SPF projects

Economic sectors targeted

SPF projects provided training aimed at helping participants to secure employment in a wide variety of sectors. The most common sectors were:

• trades (non-specific) (60%)
• construction (38%)
• mining (30%)
• hospitality and tourism (28%)
• healthcare (22%)
• carpentry (22%)

Programs and services provided

SPF projects provided a broad variety of services, ranging from light-touch employment assistance services to intensive pre-employment training and support to complete university degrees:

• 96% of projects provided interventions aiming to improve participants' general employability; for example:
  • 69% provided essential skills training
• 62% provided pre-employment development interventions aimed at improving participants employability, and
• 60% provided various work experience programs
• 87% of projects provided various types of occupational skills training, including certificates, diplomas, and degrees from recognised post-secondary institutions, apprenticeships, and industry recognised certificates
• 63% of projects offered some type of employment assistance services, such as:
  • employment counselling (52% of projects)
  • career research and exploration (40% of projects), and
  • job search supports (22% of projects)
• 23% of projects provided ongoing wrap-around supports for participants facing multiple barriers to employment

Many projects incorporated Indigenous approaches, methods, and practices into their training programs. Agreement holders noted that this practice increased participant engagement, created a supportive learning environment, and helped to support participants' wellbeing.

Average cost per participant

Contextual factors should be taken into account when considering the cost per participant of interventions. For the SPF, the document review found that projects with higher costs per participant were affected by the following factors:

• the higher cost of delivering labour market programs in rural, remote, and northern areas (compared to urban areas), and
• the higher cost of supporting individuals who face persistent barriers to employment, as they may require interventions over a longer period of time, broader and more intensive support, and may not have a linear path to employment

For all projects funded through the 2016 call for proposals, the anticipated average cost per participant prior to program delivery was $11,263. The actual cost per participant once the program had been delivered was $14,437.^{Footnote 36}

Taking the contextual factors into account, the anticipated cost per participant was relatively low, and the actual cost per participant was within the range expected for these types of programs. It is expected that some projects will encounter unforeseen challenges that will increase their cost per participant. The impacts of the pandemic on participant numbers, project delivery, the labour market and the economy may also have influenced costs.

Evaluation found significant variation in the average cost per participants across projects. This variation reflects, in part, the higher costs associated with delivering programs in northern, rural, and remote areas, and with serving individuals facing persistent barriers to employment who may require multiple interventions. Nevertheless, ESDC may wish to closely examine projects with relatively high average cost per participants with a view to ensure that projects approved support the greatest number of Indigenous workers or jobseekers and represent the most effective avenue to provide employment support and services.

Types of partners

SPF agreement holders worked with a variety of partners in the public, non-profit, and private sectors. The number of partners per project ranged from 1 to over 20. Overall:

• 71% of projects partnered with Indigenous bands, councils, and governments
• 62% of projects partnered with private sector firms, corporations, and businesses
• 56% of projects partnered with post-secondary education institutions
• 56% of projects partnered with Indigenous-led non-profit organizations

Nearly 60% of the projects that identified private sector partners worked with partners in the mining, quarrying, and oil and gas extraction sector.

Partner contributions

Partner contributions took many forms, including:

• supporting project design
• delivering training
• providing work placements
• commitments to hiring trained participants, and
• providing cash and in-kind resources

Funding agreements identified the planned sources of funding for each project, including the anticipated cash and in-kind resources to be provided by partnering organizations. Overall:

• 54% of the funding across all projects was to be provided by the SPF
• partners were to contribute 13% of project resources as cash contributions, amounting to over $50 million across all projects
• partners were to provide 33% of project resources as in-kind contributions, amounting to an estimated value of over $126 million across all projects

Funding and in-kind resources to be provided by partners ranged from 0% to 89% of total resources per project. A quarter (25%) of the projects were to receive at least 50% of their project's resources from their partners. Another quarter (25%) were to receive 25% to 49% of their project resources from their partners.

Although many partnerships evolved as planned, the document review reveals that some agreement holders experienced challenges in securing anticipated contributions from their partners. It was not possible to confirm the actual amount of cash and in-kind resources provided by partners, nor to confirm to what extent partners had honoured their initial commitments across all projects.^{Footnote 37}

Benefits and challenges of working with partners

Barriers to labour market participation

At the start of their SPF-funded intervention, participants were asked to identify which barriers affected their ability to find a job and to stay employed. Figure 6 presents the barriers identified by participants. Given that participants may have chosen not to report their barriers to employment or may have been uncomfortable reporting some types of barriers, findings should be understood to represent a conservative estimate.

• Source: SPF administrative data.

Text description: Figure 6

Barrier	Percentage of participants (n=10,929)
Physical or mental health barrier	4%
Lack of access to dependent care	4%
Lack of transportation	13%
Living in a remote area with few jobs	16%
Lack of resources to purchase equipment	16%
Long absence from the labour market	18%
Lack of marketable skills	29%
Low educational attainment	29%
Lack of work experience	32%

Reflecting the high proportion of participants who were youth, and the high proportion who had low educational attainment, the 3 most common barriers identified by participants were:

• lack of work experience (32%)
• low educational attainment (29%)
• lack of marketable skills (29%)

Long absence from the labour market (18%), limited resources (16%), living in a remote area (16%) and lack of transportation (13%) were also identified as barrier by at least 15% of participants. With respect to remoteness, key informants noted that the shift toward online training that began in response to the COVID-19 pandemic facilitated training for participants who had access to highspeed internet with sufficient bandwidth. For those who did not, the move to online training became a major obstacle to participation, particularly for those living in communities that closed during parts of the pandemic.^{Footnote 39}

Only 4% of participants identified lack of access to dependent care as a barrier. However, there was variation by gender; 8% of women who participated in SPF-funded interventions identified dependent care as a barrier, compared to 2% of men.

Barriers to employment were identified when participants began their interventions, and most began their interventions before the onset of the COVID-19 pandemic. The closure of schools, childcare centres, and other dependent care services had a disproportionate impact on women and on single parents, meaning that dependent care may have become a more common barrier during the pandemic.^{Footnote 40}

General participant outcomes

Table 6 presents the pre and post-participation labour market outcomes for 2017 and 2018 SPF participants.

Table 7 presents the pre and post-participation proportion of participants who fell in 6 different earning ranges of $10,000, from $0 to $60,000 or more per year. It also presents the post-participation change in the proportion of participants who fell in each earning range.

As shown in Table 6 and Table 7, after participating in SPF-funded interventions, participants:

• increased their earnings by $4,150 per year
  • the proportion of participants earning less than $10,000 decrease by 7 percentage points while the proportion earning more than $30,000 increased by 6 percentage points
• increased their incidence of employment by 4 percentage points
  • based on the results reported by SPF agreement holders, 48% of participants were employed following participation (including 1% who were self-employed)
• increased their use of EI benefits by $2,000, their use of SA benefits by $530, and increased the proportion of their income that came from government income supports by 12 percentage points

The increase in use of EI benefits is not necessarily negative as it may reflect post-participation improvements in participants' labour market attachment. In other words, having worked more, participants became EI eligible and were able to access EI benefits.

The labour market outcomes cannot be attributed solely to SPF participation. However, an incremental impact analysis completed as part of the 2020 evaluation of the SPF found that the program had positive impacts. Compared to similar non-participants, individuals who started an SPF intervention between January 2011 and December 2012 had higher annual earnings in the post-participation period; active EI claimants earned $5,000 more, former EI claimants earned $6,010 more, and non-EI claimants earned $1,240 more.^{Footnote 43}

Findings by SPF intervention type

Outcomes were produced for participants in 6 types of SPF interventions.^{Footnote 44} Overall, the findings reveal variation with respect to:

• the labour market characteristics of participants in different types of SPF interventions, and
• the labour market outcomes of participants in different types of SPF interventions

Findings by type of participant

Post-participation outcomes were produced for 3 types of participants:^{Footnote 47}

• active EI claimants, who accounted for 15% of participants
• former EI claimants, who accounted for 23% of participants, and
• non-EI claimants, who accounted for 62% of participants

Each type includes individuals with different levels of pre-participation labour market attachment (earnings and incident of employment). As shown in Table 10, active EI claimants had the strongest pre-program participation labour market attachment while non-EI claimants had the weakest attachment.

The differences in the level of labour market attachment of the 3 types of participants are maintained in the post-participation period. In other words, active EI claimants continue to have the highest annual incomes and incidence of employment, while non-EI claimants continue to have the lowest. However, non-EI claimant participants had stronger post-participation increases in earnings and employment. In fact, while active and former claimants decreased their incidence of employment, non-EI claimants increased theirs by 14 percentage points.

Findings by participant subgroup

Outcomes were produced for participant subgroups based on:

• Indigenous population group (status First Nations; non-status First Nations; Inuit; Métis)
• gender (men; women)
• age (30 and younger; 31 to 54; 55 and older)
  • youth with no high school diploma
• type of location (rural; urban)

Within these participant subgroups, the outcome analysis found differences in participants pre-participation labour market attachment and post-participation outcomes. The outcome analysis does not account for differences in the composition of subgroups. For example, there may be differences in the level of education, age and type intervention received across different Indigenous population groups, which could account for some of the differences relating to their labour market attachment and outcomes. For this reason, findings should not be compared across subgroups. Nevertheless, some insight can be gained by interpreting outcomes in conjunction with subgroups' pre-participation level of labour market attachment. Differences in outcomes that cannot easily be explained may reveal areas that would benefit from further exploration.

Community economic and human resources development

Projects that help participants to live and work in their community can support the economic development of the community as a whole. For example:

• projects that support job creation opportunities, either through the creation of a self-employment business or through the development of local services, can lead to the creation of employment opportunities for more community members
• projects that connect participants to job opportunities in surrounding communities or in the region can also build ties and paths leading to employment for more community members
• projects that focus on skills development, for example in trades and healthcare, increase the skills of community members

Community development

Projects that help participants find work in or near their community, or that enable them to live in the community while working elsewhere part of the time help participants to:

• maintain their community ties
• remain involved in the cultural life of their communities
• continue to participate in traditional harvesting activities such as fishing and hunting, and to pass these skills to the next generation

When youth and families are able to stay and thrive within their community they contribute to the ongoing growth, development, and vitality of the community as a whole.

Opportunity effect

Aside from the creation of new paths to labour market integration, projects can create an opportunity effect. In other words, seeing participants successfully complete studies, obtain new skills, gain employment and return to their community can increase community members' sense of what is possible and of what is within their reach.

Strengthening families and households

Life and essential skills training have benefits that extend beyond increased employability to other areas of participants' lives, including family life and household management.

Skills development and other interventions that help connect participants to sustainable, long-term employment reduces families' vulnerability to housing and food insecurity. In some cases, agreement holders reported that the improvements were such that families who had been separated, largely due to the consequences of poverty, were able to be reunited.

Examples of broader project impacts

Example: The move to online service delivery and increased access to training

Having a high school diploma or General Education Diploma is a pre-requisite for many occupational skills training programs. Because there are usually no transportation and housing supports to help people who live in remote communities to travel to urban centres in order to obtain their General Education Diploma, those who have not completed secondary school are often unable to qualify for entry into training.

Due to the pandemic, an agreement holder was forced to move the General Education Diploma portion of their program online, and participants were able to obtain their Diploma without having to leave their communities. The completion rate increased significantly after the program was moved online, and the agreement holder has decided to maintain online delivery of the program. Participants who obtained their General Education Diploma became eligible to participate in occupational skills training programs.

For individuals who did not have access to computers and a high-speed internet connection with sufficient bandwidth, the move to online delivery represented a greater challenge. In some cases, agreement holders were able to connect participants to the resources they needed, but this was not always the case.

Administrative processes and SPF agreement holder capacity

Key informant interviews, the case studies, and previous engagement exercises with Indigenous partners identify challenges relating to SPF administrative processes for funding applicants that are not well established or that have limited human resources capacity. For example:^{Footnote 52}

• the investment of time and effort required to submit a competitive proposal can present a considerable challenge for smaller organizations
• securing well-defined cash and in-kind contributions from potential new partners can be challenging for organizations that do not have an established partner network
• reporting requirements can be challenging for smaller organizations, particularly when they experience staff turnover, when their main government contact changes, or when they rely on partners to collect and report data

Program officials reported that providing support before the call for proposal is released and throughout the duration of the project can be helpful.

Staff and partner recruitment and retention

Issues relating to staff retention and the difficulty of filling vacant job positions were widespread throughout the country during the pandemic. A common challenge faced by the projects with lower employment outcomes was the difficulty of recruiting and retaining staff, instructors and partners.

Staff and instructor turnover had a particularly negative impact on projects located in northern and remote areas. These communities tend to have small populations with very limited human resources pools. They often have to recruit from outside of their community and find individuals willing to move to their community. This process can take a long time and require significant resources. These challenges are compounded in communities that are not connected to highspeed internet, as shifting to online training is not an option. Therefore, the loss of a single staff member or instructor can have a severe impact on projects outcomes.

Similarly, SPF agreement holders in northern and remote areas have a limited pool of potential partners to choose from. In some communities, there is only 1 large employer. Consequently, when a partnership dissolves or does not go as planned, finding new partners can prove very challenging. When it is not possible to find a new partner, or when new partners have much smaller labour force needs than the initial partner, project outcomes are negatively impacted.

These challenges, although compounded by the pandemic, are part of the reality of serving communities in northern and remote areas. In some cases, when given the flexibility to adjust project design, and additional time to secure new staff or partners and to reach their targets, agreement holders were able to overcome setbacks and achieve positive results.

Ongoing participant support

A common thread among the SPF agreement holders that achieved positive outcomes was the need to provide ongoing support to participants. Projects were designed and delivered around the understanding that paths to employment may not progress in a linear fashion, may encounter setbacks, and may require course adjustments and multiple interventions. Service providers were prepared to offer or connect participants with a broad range of supports.

Capturing post-participation results

Results are collected after program participation and can include, for example, whether or not participants found employment or returned to school. To collect this information, project holders must contact former participants to confirm whether they found employment or returned to school after the end of their intervention.

Several factors can hinder agreement holders' ability collect this information. For example, after program participation, some participants may move, change their telephone numbers, or become unreachable for other reasons. Accordingly, it is expected that agreement holders will not be able to confirm results for all the project participants.

The assessment of the programs' administrative data found that, in the period following participation, agreement holders were able to collect and report results for 85% of participants.

Capturing progress along the pre-employment continuum

Many types of interventions provided through SPF projects are not conceived to result in employment in the short-term post-program period. Rather, such interventions aim to support participants' progress along the pre-employment continuum. Although employment is the end goal for all participants, agreement holders emphasise the importance of capturing pre-employment outcomes.

SPF agreement holders can use several indicators to capture pre-employment outcomes; these include the number of participants who:

• increased their employability
• completed pre-employment training, and
• gained work experience

The document review found that the meaning of employability and the approach to measuring increased employability varied across projects.

Identifying barriers to employment

Individuals can face many different barriers that prevent them from entering or remaining in the workforce, such as lack of transportation, lack of access to childcare, and lack of work experience, for example.

Agreement holders ask participants to identify which types of barriers are affecting their ability to find and maintain employment. Participants are not obliged to provide this information and may choose not to report 1 or more of the barriers they face. Participants may also be more comfortable reporting on some barriers than others. For example, they may be comfortable revealing that they lack work experience, but uncomfortable revealing that they have a health or mental health issue.

The administrative data assessment confirmed that some participants chose not to provide information on some types of barriers. For example, information on disability status was missing for 42% of participants.

Accordingly, while the administrative data on reported barriers can be used to gain insight into commonly experienced barriers to employment, barriers of a more sensitive or personal nature may be underreported.

Assessing partner contributions

Developing and maintaining partnerships with public, non-profit and private sector organizations to support program and service design, delivery, and outcomes is an integral component of the SPF program. As part of their proposal, organizations must identify and quantify the cash and in-kind resources that their partners will contribute to the project. Projects that can secure at least half of their project resources are given priority consideration during the project selection process. Yet, agreement holders are not systematically requested to report:

• to what extent their partners honoured their commitments, and
• whether they developed new partnerships and how these contributed to their projects

The document review identified a few instances of agreement holders reporting that partners had not contributed to the projects as expected. While this may not be a widespread challenge, in the absence of systematic reporting, the actual contribution of project partners to the projects cannot be assessed.

Northern and remote agreement holders and projects

The data assessment and document review identified contextual challenges and vulnerabilities relating to the organizational capacity of agreement holders located in northern and remote regions and to the limited partnership opportunities present in northern and remote areas.

Recommendation 1: ESDC is encouraged to explore options and take actions to offer further support to SPF agreement holders in rural and remote areas through each stage of the projects.

Data collection and performance measurement

The program is following the best practice of collecting individual-level data. The data collected and reported for projects funded through the 2016 call for proposals was of good quality and integrity overall. The document review and the literature review, which included reports summarising Indigenous partner engagement findings, found that the data collection and reporting requirements continue to be challenging for some agreement holders.

Moreover, some projects reported on the cash and in-kind contributions they received from their project partners throughout the project. Some also reported on their experience of working with their partners, including the challenges and benefits. This information, particularly when collected across all projects, can be valuable for program and project design and delivery, program management, program evaluation, and performance measurement.

Recommendation 2: ESDC is encouraged to explore options and take actions to:

• continue to prioritize data integrity, including validating data uploads, and providing support to projects experiencing data collection and reporting challenges; and
• collect consistent partner contribution data across all projects

Case study 1

Stream: Training-to-Employment

Location: Western provinces (urban, near urban)

Population(s) served: First Nations, Inuit, Métis

Types of partners: Indigenous educational institute(s), non-profit organization(s), sector council(s) and trade association(s), provincial government, private sector

Training sector(s): trades, apprentices, construction

Intervention type(s): employment assistance services, essential skills training, work experience, occupational skills training (including apprentices)

Case study 2

Stream: Training-to-Employment

Location: Western provinces (urban, near urban)

Population(s) served: First Nations

Types of partners: non-profit organization(s), sector council(s) and trade association(s), private sector (mining)

Training sector(s): trades, apprentices, construction, electrical, forestry, mining, welding

Intervention type(s): essential skills training, occupational skills training (including apprentices)

Case study 3

Stream: Training-to-Employment

Location: Central provinces (northern, remote, Inuit self-government communities)

Population(s) served: Inuit

Types of partners: non-profit organization(s), Indigenous government(s), provincial government, private sector (mining)

Training sector(s): trades, apprentices, carpentry, construction, mining

Intervention type(s): employability training, employment assistance services, essential skills training, work experience, occupational skills training (including apprentices)

Case study 4

Stream: Training-to-Employment

Location: Atlantic provinces (northern, remote, including on and off reserve and Inuit self-government communities)

Population(s) served: First Nations, Inuit, Métis

Types of partners: post-secondary institution(s), provincial government, private sector (mining)

Training sector(s): trades, apprentices, mining

Intervention type(s): essential skills training, employment assistance services, work experience, wage subsidies, occupational skills training (including apprentices)

Case study 5

Stream: Innovation

Location: Atlantic provinces (urban, near urban)

Population(s) served: First Nations

Types of partners: post-secondary education institution, Indigenous government(s), non-profit organization(s), provincial government, municipal government(s), private sector

Training sector(s): arts and culture, green energy, information and communications technology

Intervention type(s): employability training, employment assistance services, essential skills training, work experience, occupational skills training (including apprentices)