Evaluation of the Canada – Yukon Labour Market Development Agreement

The LMDA investment

In fiscal year 2020 to 2021, Canada transferred approximately $4.7 million (including nearly $389,000 in administration funds) to Yukon.

Table i provides an overview of the share of funding allocated to EBSMs. From the 2010 to 2012 time period to the 2020 to 2021 fiscal year,^{Footnote 1} investments in SDEB decreased by 11 percentage points. The largest increase in investments went to EAS, which saw its share of funding increase by 24 percentage points.

Sources: EI Monitoring and Assessment Reports for fiscal years 2010 to 2011, 2011 to 2012, and 2020 to 2021.

Evaluation objectives

Building on the success of previous LMDA evaluation cycles, the aim of this evaluation is to fill in knowledge gaps about the effectiveness, as well as design and delivery of EBSMs in Yukon.

Evaluation methodology

The findings in this report are drawn from 3 evaluations studies carried out at the territorial level. These studies examine issues related to program effectiveness, design and delivery. A mix of qualitative and quantitative methods are used, including:

• incremental impact analysis
• outcome analysis
• a questionnaire
• document review

The labour market outcomes are estimated for 2 types of EI claimants:

• active claimants are participants who started an EBSM intervention while collecting EI benefits
• former claimants are participants who started an EBSM intervention up to 3 years after the end of their EI benefits^{Footnote 4}

Key findings

Across Yukon, nearly 950 active and former EI claimants began participating in Canada-Yukon LMDA programs and services between 2009 and 2012.

Supplemental studies

A supplemental study addresses information gaps previously identified in LMDA evaluations regarding the design and delivery of Research and Innovation. An additional study examines the outcomes for participants in SDEB-Apprentices.

Recommendations

Since 2012, several qualitative and quantitative studies addressed issues and questions related to EBSM design, delivery and effectiveness at the national level. The quantitative studies provided insight into participants’ post-participation labour market outcomes. The qualitative studies identified specific challenges, lessons learned and best practices associated with the design and delivery of EBSMs. Each study included key considerations for program and policy development or recommendations.

The recently completed evaluation of the Workforce Development Agreements complements the LMDA qualitative studies. This evaluation was also supported by literature reviews and provided unique insights into challenges and lessons learned to assist persons with disabilities, immigrants and those further removed from the labour market.

Most results from this evaluation stem from the conduct of medium-term outcome analysis predicated on having access to high quality administrative records, thereby confirming the importance of the capacity to leverage and integrate relevant administrative data.

From these main findings, 2 key recommendations emerge.

Recommendation #1: Yukon is encouraged to share and discuss lessons learned, best practices and challenges associated with the design and delivery of programs and services. Discussions are encouraged with ESDC, at the bilateral or multilateral levels.

Recommendation #2: Yukon is encouraged to pursue efforts to maintain and strengthen data collection provisions in support of reporting, performance measurement and data-driven evaluations at the national and territorial levels.

Employment benefits^{Footnote 14}

Skills Development Employment Benefit (SDEB) provides direct financial assistance to individuals to select, arrange, and pay for training. Training is tailored to the needs of participants through counselling and career orientation. It can include adult-based education, literacy and essential skills, language training, short-term training and occupational training leading to certification from an accredited institution.

Support measures

Employment Assistance Services (EAS) support individuals as they prepare to enter or re-enter the workforce or assist them to find a better job. Examples include job search services, career counselling, and résumé writing assistance. These services are light touch interventions due to their very short duration and can be provided on a one-on-one basis or in a group setting. A typical intervention lasts less than 1 day, but a participant may receive multiple short interventions over a few weeks. These services are generally provided in combination with more intensive interventions.

Research and Innovation funds research projects aimed at identifying labour market information or testing new approaches to labour market development.

Eligible participants covered in this study

The incremental impacts and labour market outcomes are estimated for active and former EI claimants:

• active claimants are participants who started an EBSM intervention while collecting EI benefits
• former claimants are participants who started an EBSM intervention up to 3 years after the end of their EI benefits^{Footnote 15}

Average EBSM share of funding

Table 1 provides an overview of the share of funding allocated to EBSMs. From the 2010 to 2012 time period to the 2020 to 2021 fiscal year, investments in SDEB decreased by 11 percentage points. The largest increase in investments was for EAS, which saw its share of funding increase by 24 percentage points.

Sources: EI Monitoring and Assessment Reports for fiscal years 2010 to 2011, 2011 to 2012, and 2020 to 2021.

Incremental impacts analysis^{Footnote 18}

Program effectiveness is assessed by estimating the incremental impacts from EBSM participation on participants’ labour market experience. That is, earnings from employment and self-employment, incidence of employment, use of EI, use of social assistance (SA), and dependence on government income supports after participation.

In Yukon, incremental impacts were estimated for active EI claimant participants in EAS. Incremental impacts for SDEB and Targeted Wage Subsidies were not produced due to the small samples.

The role of the incremental impact analysis is to isolate the effects of participation from other factors. In order to achieve this, the incremental impact analysis compares the labour market experience of participants before and after their participation with that of similar non-participants. Figure 1 presents an example of incremental impact calculation.

Text description – Figure 1

Incremental impacts are the difference in the pre-post participation outcomes between participants and similar non-participants.

For example:

• if the average annual earnings of participants are $30,000 before participation and $38,000 after participation, the change in earning for participants is an increase of $8,000
• if the average annual earnings in the comparison group are $31,000 before participation and $36,000 after participation, the change in earnings for the comparison group is an increase of $5,000
• the incremental impact, that is the change attributed to program participation, is $8,000 minus $5,000, which equals an increase of $3,000

The main estimator used is propensity score kernel matching technique combined with difference-in-differences estimator. Moreover, 3 different state-of-the-art estimation techniques (Inverse Probability Weighting, Nearest Neighbour and Cross-sectional Matching) were carried out separately for each type of EBSMs and EI claimants in order to validate the impact estimates.

As for previous LMDA evaluation studies, the Action Plan Equivalent is the unit of analysis used. Action Plan Equivalents regroup all EBSMs received by an individual within less than 6 months between the end of one EBSM and the start of the next. Action Plan Equivalents are categorized based on the longest EBSM they contain, except for EAS-only Action Plan Equivalents which include only EAS interventions.

The analysis includes Action Plan Equivalents that consist only of LMDA interventions. Action Plan Equivalents that include a combination of LMDA and other labour market programs funded by ESDC, were excluded from the participant sample.

The matching of participants and comparison group members used up to 75 socio-demographic and labour market variables observed over 5 years before participation. For active claimants in EAS, incremental impacts were measured relative to a comparison group of active EI claimants who were eligible to, but did not, participate in LMDA-funded EBSMs during the reference period.

Impacts are generated for active EI claimant participants in EAS over the 5 years following participation.^{Footnote 19}

Medium-term outcome analysis

The study reports on the labour market outcomes for active and former EI claimants SDEB and EAS. When the number of participants was sufficient, outcomes are examined for active and former EI claimants who are female, male, youth (30 years old and under), older workers (55 years old and older), and Indigenous.

The study includes up to 100% of participants who began their EBSM participation between April 1, 2009, and December 31, 2012. The timeframe was selected in order to assess the outcomes of EBSMs in the years following participation. Overall, the analyses were conducted over a 10 to 11 year period:

• a pre-participation period of 5 years for SDEB and EAS
• a participation period of 2 years for SDEB, and of 1 year for EAS
• a post-participation period of 4 years for SDEB and EAS

The labour market outcomes regarding employment earnings, EI and SA benefits are presented in constant 2010 dollars.

Strengths and limitations of the studies

One of the key strengths from the studies is that all quantitative analyses are based on administrative data rather than survey responses. Compared to survey data, administrative data are not subject to recall errors or response bias.

The outcome analysis provides an assessment of how the labour market situation of participants evolved over time but does not permit inference regarding the extent to which those changes were due to EBSM participation. For example, increases in employment earnings over the period examined could be partly due to an increase in wages.

The propensity score models used to match participants and non-participants for the incremental impact analyses are judged to be robust. In part this is because they were based on 5 years of pre-participation data. Moreover, these models are based on a vast array of variables including sociodemographic characteristics, location, skill level related to last occupation, and indicators of labour market attachment.

However, the matching process can be further refined for specific subgroups if the following information is available in the future:

• persons with disabilities: the type and severity of the disability, and the capacity/willingness to work full-time
• recent immigrants: the country of origin, the proficiency in English or French, and the relevance of credentials and work experience
• visible minorities: place of birth; individuals who are born outside of Canada face different challenges compared to those born in Canada

Refining the matching process for population subgroups could broaden the scope for greater Gender-based Analysis Plus.

Sensitivity analysis and the use of alternative estimation methods have increased confidence in the incremental impact estimates. However, one limitation with the propensity score matching techniques is that no one can be fully sure the impacts are not influenced by factors not captured in the data.

Overview of the studies summarized in this report

The findings in this report are drawn from 3 separate studies:

• Estimation of the medium-terms incremental impacts from 2009 to 2017
• Examination of the medium-term outcomes from 2009 to 2017
• Design and delivery of Research and Innovation in Yukon

Using a mix of qualitative and quantitative methods, these studies examine issues related to program effectiveness, design and delivery. Appendix A presents an overview of each study.

Profile of participants

Between 2009 and 2012, nearly 950 active and former EI claimants participated in LMDA-funded programs and services in Yukon.

The profile of participants is presented in Table 2 by gender, age, sociodemographic groups, marital status, educational attainment, occupation and industry. Information about their educational attainment, occupation and industry are based on the latest job they held prior to applying for EI benefits. Information about sociodemographic groups is self-reported.

Notes: Values may not equal 100% due to rounding or missing information; < 10 = fewer than 10 participants.

Top 3 occupational groups for EBSMs participants

Active claimants:

• other manual workers = 15%
• semi-skilled manual workers = 14%
• semi-professionals and technicians = 12%

Former claimants:

• other manual workers = 16%
• semi-skilled manual workers = 14%
• intermediate sales and service personnel = 13%

Top 3 industries for EBSMs participants

Active claimants:

• public administration = 18%
• construction = 14%
• accommodation and food services = 12%

Former claimants:

• public administration = 21%
• construction = 13%
• accommodation and food services = 12%

As presented in Table 3, in the year before program participation, former claimants had lower levels of employment and annual earnings than active claimants. Former claimants also had a higher dependence on SA.

Incremental impacts for active EI claimants in EAS^{Footnote 20}

Labour market outcomes for active and former EI claimants

Overall, following participation in SDEB and EAS, program participants have:

• higher average annual earnings
• lower incidence of employment, which may be due to retirement decisions
• decreases in the use of EI benefits, but experience increases in the use of SA benefit

In Yukon, labour market outcomes are examined for active and former EI claimant participants in SDEB and EAS. The outcomes describe the average changes in participants’ earnings, incidence of employment, and use of EI and SA benefits that occur from before to after program participation.

The labour market outcomes are based on individuals who began their participation during the 2009 to 2012 period. Statistics focus on 5 years prior to participation, 1 or 2 years during participation and 4 years post-participation.

Between 2009 and 2012:

• 238 active EI claimants and 168 former EI claimants participated in SDEB
• 270 active EI claimants and 238 former EI claimants participated in EAS

Skills Development Employment Benefit participant outcomes

Approximately 400 active and former EI claimant participants, between 2009 and 2012, participated in SDEB.

Active claimants

As shown in Figure 2, SDEB active EI claimant participants increase their average earnings from $22,261 in the fifth year pre-program to $36,474 in the fourth year after program participation. On average, active claimants earn $9,629 more per year after participating in SDEB.

Text description – Figure 2

Year in pre-post participation period	Average earnings for active claimant participants in SDEB
5 years pre-program	$22,261
4 years pre-program	$22,971
3 years pre-program	$26,297
2 years pre-program	$27,404
1 year pre-program	$29,879
Program start year	$19,820
Second year of participation	$25,405
1 year post-program	$33,409
2 years post-program	$36,157
3 years post-program	$35,526
4 years post-program	$36,474

As shown in Table 5, the change in proportion of active claimants employed from before to after SEDB participation declines by 2 percentage points remaining on average at 92% during the post-program period. The decline can be attributed, in part, to participants’ retirement decisions. This conclusion is based on the fact that there is an increase in participants receiving Canada Pension Plan (5 percentage points), Canada Pension Plan-Disability (1 percentage point) and Old Age Security benefits (1 percentage points) in the post-program period.

Participants also decrease their use of EI benefits by 5 percentage points, but experience increases of 3 percentage points in the use of SA benefits.

Table 5. Pre and post-participation change in labour market outcomes for active claimants in SDEB

Indicator	SDEB active claimants
Incidence of employment (percentage points)	-2
Employment earnings ($)	9,629
EI benefits ($)	-5
SA benefits ($)	3
Dependence on income support (percentage points)	0
n=	238

Former claimants

As shown in Figure 3, SDEB participants increase their average earnings from $18,601 in the fifth year pre-program to $32,043 in the fourth year after program participation. On average, former claimants earn $11,246 more per year after participating in SDEB.

Text description – Figure 3

Year in pre-post participation period	Average earnings for former claimant participants in SDEB
5 years pre-program	$18,601
4 years pre-program	$20,962
3 years pre-program	$21,696
2 years pre-program	$19,894
1 year pre-program	$16,206
Program start year	$20,321
Second year of participation	$27,155
1 year post-program	$29,108
2 years post-program	$30,873
3 years post-program	$30,847
4 years post-program	$32,043

As shown in Table 6, the change in proportion of former claimants employed from before to after SEDB participation declines by 6 percentage, remaining on average at 87% during the post-program period. The decline can be attributed, in part, to participants’ retirement decisions. This conclusion is based on the fact that there is an increase in participants receiving Canada Pension Plan (7 percentage points), Canada Pension Plan-Disability (1 percentage point), and Old Age Security (3 percentage points) after participation.

Participants decrease their use of EI benefits by 20 percentage points, but experience increases of 2 percentage points in SA use.

Table 6. Pre and post-participation change in labour market outcomes for former claimants in SDEB

Indicator	SDEB former claimants
Incidence of employment (percentage points)	-6
Employment earnings ($)	11,246
EI benefits ($)	-20
SA benefits ($)	2
Dependence on income support (percentage points)	-5
n=	168

Employment Assistance Services participant outcomes

Approximately 510 active and former EI claimant participants, between 2009 and 2012, participated in EAS.

Active claimants

As shown in Figure 4, active claimants in EAS increase their average earnings from $19,632 in the fifth year pre-program to $24,874 in the fourth year after the program participation. On average, active claimants earn $1,720 more per year after participation.

Text description – Figure 4

Year in pre-post participation period	Average earnings for active claimant participants in EAS
5 years pre-program	$19,632
4 years pre-program	$21,509
3 years pre-program	$22,969
2 years pre-program	$24,284
1 year pre-program	$26,986
Program start year	$18,602
1 year post-program	$22,563
2 years post-program	$25,321
3 years post-program	$26,426
4 years post-program	$24,874

As shown in Table 7, the change in proportion of active claimants employed from before to after EAS participation declines by 8 percentage points, remaining on average around 85% in the post-program period. The decline can be attributed, in part, to participants’ retirement decisions. This conclusion is based on the fact that there is an increase in participants receiving Canada Pension Plan (8 percentage points) and Old Age Security (5 percentage points).

These participants also decrease their use of EI benefits by 5 percentage points, but experience increases in the use of SA benefits of 3 percentage points in the post participation period.

Table 7. Pre and post-participation change in labour market outcomes for EAS active claimants

Indicator	EAS active claimants
Incidence of employment (percentage points)	-8
Employment earnings ($)	1,720
EI benefits ($)	-5
SA benefits ($)	3
Dependence on income support (percentage points)	2
n=	270

Former claimants

As shown in Figure 5, former claimants increase their average earnings from $18,951 in the fifth year pre-program to $23,824 in the fourth year after EAS participation. On average, former claimants earn $2,994 more per year after participation.

Text description – Figure 5

Year in pre-post participation period	Average earnings for former claimant participants in EAS
5 years pre-program	$18,951
4 years pre-program	$20,264
3 years pre-program	$21,860
2 years pre-program	$20,246
1 year pre-program	$16,217
Program start year	$15,184
1 year post-program	$21,340
2 years post-program	$21,739
3 years post-program	$23,102
4 years post-program	$23,824

As shown in Table 8, the change in proportion of former claimant participants employed from before to after EAS participation decreases by 9 percentage points, remaining on average around 84% in the post-program period. The decline can be attributed, in part, to participants’ retirement decisions. This conclusion is based on the fact that there is an increase in participants receiving Canada Pension Plan (8 percentage points) and Old Age Security (3 percentage points) in the post-program period.

These participants also decrease their use of EI benefits by 16 percentage points, but experience increases in the use of SA benefits by 6 percentage points.

Table 8. Pre and post-participation change in labour market outcomes for EAS former claimants

Indicator	EAS former claimants
Incidence of employment (percentage points)	-9
Employment earnings ($)	2,994
EI benefits ($)	-16
SA benefits ($)	6
Dependence on income support (percentage points)	1
n=	238

Labour market outcomes by subgroups of participants

Research and Innovation^{Footnote 36}

Research and Innovation provides funding for research and demonstration projects. These projects aim to collect and disseminate labour market information and to test new approaches in labour market development.

Between 2015 and 2020, Yukon used the program intermittently. Research and Innovation spending varied between 1% ($51,000) to 5% ($187,000) of annual LMDA expenditures in the territory. Program officials report that the amount allocated to Research and Innovation is influenced by government priorities and labour market issues.

Skills Development Employment Benefit-Apprentices

The objective of the program is to help apprentices become skilled tradespeople and to increase their labour market attachment. Program participants have generally chosen a career and are already attached to the labour market. The apprenticeship process involves on-the-job learning and technical training in a classroom setting.

Apprentices who have worked enough hours to qualify for EI can apply to receive EI Part I benefits while on training. The program provides financial assistance to EI eligible apprentices to help them offset the costs they incur while they attend technical training. The level of funding is based on the needs of apprentices, the location of the training, and any fees paid by the apprentices.

The profile of active EI claimant participants in SDEB-Apprentices is presented in Table 19.^{Footnote 37} Information about their educational attainment, occupation and industry is based on the last job they held prior to applying for EI Part 1 benefits. Information about sociodemographic groups is self-reported.

Notes: Values may not equal 100% due to rounding or missing information; < 10 = fewer than 10 participants.

Top 3 occupational groups for SDEB-Apprentices participants

Active claimants:

• skilled crafts and trades workers = 82%
• other manual workers = <10 participants
• skilled sales and service personnel = <10 participants

Top 3 industries for SDEB-Apprentices participants

Active claimants:

• construction = 67%
• other services (excluding public administration) = <10 participants
• public administration = <10 participants

Labour market outcomes

As shown in Figure 6, active claimant participants in SDEB-Apprentices increase their average earnings from $19,238 in the fifth year pre-program to $56,585 in the fifth year after the program start year. They earn on average $26,457 more per year during the post participation period compared to before participation.

Text description – Figure 6

Year in pre-post participation period	Average earnings for active claimant participants in EAS
5 years pre-program	$19,238
4 years pre-program	$21,654
3 years pre-program	$22,739
2 years pre-program	$25,780
1 year pre-program	$34,976
Program start year	$33,369
1 year post-program	$38,405
2 years post-program	$44,754
3 years post-program	$50,860
4 years post-program	$53,138
5 years pre-program	$56,585

As shown in Table 20, the change in proportion of active claimants employed from before to after participation declines by 2 percentage points, remaining on average around 95% after program participation.

Participants also increase their use of EI benefits by 8 percentage points in the post program period compared to before participation. However, these participants experience decreases in EI use from 100% in program start year to 25% in the fifth post-program year.

Participants decrease their use of SA benefits by 1 percentage point in the post participation period compared to before participation.

Table 20. Pre and post-participation change in labour market outcomes for SDEB-Apprentices active claimants

Indicator	SDEB-Apprentices active claimants
Incidence of employment (percentage points)	-2
Employment earnings ($)	26,457
EI benefits ($)	8
SA benefits ($)	-1
Dependence on income support (percentage points)	-1
n=	163

Summary of findings

The medium-term outcome analysis completed as part of the evaluation provides an assessment of how the labour market situation of participants evolved over time.^{Footnote 38} Overall, the outcome analysis shows that following program participation active and former EI claimant participants in SDEB and EAS have:

• higher average earnings
• lower employment levels, mainly due to retirement decisions
• lower use of EI benefits, however, they increase their receipt of SA benefits

Where the number of participants was sufficient, a subgroup analysis was conducted for female, male, youth, older workers and Indigenous participants. The analysis shows that labour market outcomes for most of subgroups of active and former participants in SDEB and EAS are largely consistent with those for all participants.^{Footnote 39}

A separate incremental impact analysis was conducted on active EI claimants who participated in EAS. The incremental impacts are not statistically significant compared to similar non-participants.

Supplemental studies address information gaps previously identified in LMDA evaluations for Research and Innovation. A separate study examined outcomes for SDEB-Apprentices.

Overall, the studies found that:

• Yukon uses the Research and Innovation support measure to fund labour market research projects
• after participating in SDEB-Apprentices, active EI claimants increase their employment earnings and decrease their dependence on government income supports

Recommendations

Since 2012, several qualitative and quantitative studies addressed issues and questions related to EBSM design, delivery and effectiveness. The quantitative studies provided insight into participants’ post-participation labour market outcomes. The qualitative studies identified specific challenges, lessons learned and best practices associated with the design and delivery of EBSMs. Each study included key considerations for program and policy development or recommendations.

The recently completed evaluation of the Workforce Development Agreements complements the LMDA qualitative studies. This evaluation was also supported by literature reviews and provided unique insights into challenges and lessons learned to assist persons with disabilities, immigrants and those further removed from the labour market.

Most results from this evaluation stem from the conduct of advance causal analysis whereby impacts found could be attributed to a specific EBSM. These analyses are predicated on having access to high quality administrative records, thereby confirming the importance of the capacity to leverage and integrate relevant administrative data.

From these main findings, 2 key recommendations emerge:

Recommendation #1: Yukon is encouraged to share and discuss lessons learned, best practices and challenges associated with the design and delivery of programs and services. Discussions are encouraged with ESDC, at the bilateral or multilateral levels as well as with service delivery network if necessary.

Recommendation #2: Yukon is encouraged to pursue efforts to maintain and strengthen data collection provisions in support of reporting, performance measurement and data-driven evaluations at the national and territorial levels.

Overview of studies included in this synthesis report

Study 1: Examination of medium-term outcomes from 2009 to 2017
Evidence generated: profile of active and former EI claimants in Yukon; outcomes by claimant type and by subgroup
Methods: before and after results of program participation
Reference period: 2009 to 2012 participants
Observation period: Up to 12 years (5 years before participation, 1 to 2 years of participation, and up to 5 years after participation)

Study 2: Estimation of medium-term incremental impacts from 2009 to 2017
Evidence generated: incremental impacts for active and former EI claimants in Yukon; incremental impacts by subgroup; profile and socio-demographic characteristics of participants
Methods: Non-experimental method using propensity score matching in combination with Difference-in-Differences; Statistical profiling
Reference period: 2009 to 2012 participants
Observation period: Up to 7 years (1 to 2 years in program, and up to 5 years after participation)

Study 3: Design and delivery of the Research and Innovation Support measure in Yukon
Evidence generated: Program design and delivery; Challenges and lessons learned
Methods: Document review; Questionnaire validated by provincial program officials
Reference period: 2017 to 2020
Observation period: Design and delivery at the time of the data collection