Language selection

Government of Canada / Gouvernement du Canada

Search

Incremental Impact Analysis of the Youth Employment Strategy – Career Focus Stream

Technical report prepared under the Fourth Summative Evaluation of the Youth Employment Strategy (YES)

By: Andy Handouyahia, Essolaba Aouli, Leeroy Rikhi and Azadeh Heydarirad.

On this page


List of acronyms
YES
Youth Employment Strategy
SL
Skills Link
CF
Career Focus
SWE
Summer Work Experience
GBES
Group-Based Employability Skills
ESWE
Employability Skills Work Experience
WE
Work Experience
ESEN
Employability Skills Entrepreneurship
ISE
Individual Skills Enhancement
CSGC
Common Systems Grants and Contributions
AP
Action Plan
SEB
Skills and Employment Branch
POB
Program Operation Branch
SA
Social Assistance
EI
Employment Insurance
SIN
Social Insurance Number (masked)
CRA
Canada Revenue Agency
PIF
Participant Information Form
STEM
Science, Technology, Engineering, and Mathematics
EAS
Employment Assistance Service
List of tables
List of figures

Executive summary

This report presents the results of the fourth summative evaluation of the Career Focus stream of the Youth Employment Strategy (YES) program. The analysis focused on the socio-economic profiles of participants, medium-term labour market outcomes, and incremental impacts. The costs and benefits from participation are also considered. The Youth Employment Strategy is a horizontal initiative led by ESDC, and delivered in collaboration with 10 other federal departments and agencies. The YES received approximately $520M in 2017 to 2018.

The YES consists of the following 3 program streams, all of which involved youth aged 15 to 30. Each stream has distinct objectives and targets youths with different needs:

This study concentrates on participants who started an ESDC Career Focus intervention during January 2010 to December 2011. It examines their labour market outcomes up to 5 years post-participation. Specifically, this study addresses the following questions:

Methodology

For the ESDC Career Focus intervention, we used the reference period of January 1, 2010 to December 31, 2011 to define the participant cohorts for the analysis. The analyses concentrated on profiles (that is, socio-economic characteristics), labour market outcomes, incremental impacts, and economic benefits. The incremental impacts were measured relative to a comparison group. These comparison cases had the same observed socioeconomic characteristics as Career Focus participants, but had received only minor services through minor Employment Assistance Services (EAS) interventions as part of the Labour Market Development Agreements (LMDA). The comparison group also started their participation in EAS during the same 2010 to 2011 reference period.

We assessed the net impact of Career Focus year-by-year over a 6-year span, (1 year in-program and 5 years post-participation) including both average and cumulative 5-year impacts. The net impact is measured for the following 5 labour market indicators:

In addition, we conducted further analyses to assess the influence of field of study on net incremental impacts. These additional analyses focused on measuring the impact of CF for individuals who worked in science, technology, engineering, and mathematics (STEM) industries during the program participation period versus those in non-STEM fields. We did this additional analysis to examine potential biases that may arise from lack of information on the field of study.

Key findings

About 2,744 youth participated in an ESDC Career Focus intervention during the January 1, 2010 to December 31, 2011 period. The highest proportion of participants were those aged 25 to 29, representing 49% of participants, followed by those aged 20 to 24. Female participants (56%) were notably more prevalent than male participants (43%).

The incremental impact analysis indicated that relative to the EAS comparison group, Career Focus participants had higher earnings and incidence of employment. They also experience a small decrease in their reliance on income support relative to the comparison group.

The cost-benefit analysis showed that the total societal economic benefit (government + participant benefit) was $46,293 per participant.

The cost-benefit sensitivity analysis on the discount rate and marginal cost of public funds showed small fluctuations in outcomes in terms of net present value and rate of return. However, sensitivity analysis with respect to field of study, that is STEM versus non-STEM, revealed substantial variation in the cost-benefit analysis results. The average annual social rate of return for participants who worked in STEM industries during the in-program period is approximately 25%, compared to 13% for those who did not work in STEM industries and 18% overall.

 

1. Introduction

The ESDC-led, horizontal Youth Employment Strategy represents the government’s commitment to helping to equip youth with the skills, work experience and abilities they need to transition into the labour market. This evaluation will assess whether and to what extent the program is meeting these objectives for the 2010 to 2011 cohort of participants by examining results on:

In addition, we used the incremental impacts as the basis for cost-benefit analysis of the program.

The report is structured as follows:

1.1 YES program overview

The Youth Employment Strategy is a horizontal initiative led by ESDC and delivered in collaboration with 10 other federal departments and agencies. The YES received approximately $520M in 2017 to 2018. In order to be eligible to participate in YES, youth must meet all of the following criteria:

YES is divided into the Career Focus (CF), Skills Link (SL) and Summer Work Experience (SWE) program streams. Each of these streams has a different youth target population and corresponding objectives. This report concentrates solely on the Career Focus stream and those who participated in the ESDC initiative during the January 2010 to December 2011 period.

Career Focus (CF) provides funding for employers and organizations to design and deliver activities that enable youth to make informed career decisions and develop their skills. Specifically, CF activities are designed to:

1.2 Modernization of the Youth Employment Strategy program

ESDC has re-designed the YES program to address multiple labour market challenges faced by youth, particularly those facing barriers to employment. The new initiative, which began in June 2019, is called the Youth Employment and Skills Strategy (YESS) program. The YESS continues to be led by ESDC and delivered in conjunction with 10 departments and agencies across the Government of Canada. YESS aims to provide more flexible employment services tailored to each individual and enhanced supports for all young Canadians. It also aims to broaden eligibility and offer a wider range of programs to help young people gain the skills, abilities, and experience they need to obtain good-quality jobs.

2. Previous research and evaluation studies

Today’s economic landscape is highly dynamic, and is rapidly and continuously reshaped by factors such as increasing global competition and technological innovation. These forces have resulted in a swiftly-evolving labour market that presents numerous and growing opportunities for both industries and individuals. However, the current pace of change has also led to an increased likelihood of supply-demand mismatches between job requirements and prospective workers’ skill sets. These mismatches making it progressively more challenging for young people in Canada and elsewhere to find consistent and meaningful work.Footnote 2

Moreover, it is not uncommon for those with a post-secondary education to struggle with translating their credentials and training into satisfactory work. These individuals would be deemed “underemployed” in the service industry for some time, with few opportunities for advancement, before settling into more stable, career-related employment (Hoskins et al., 2018). In addition, Dhakal et al. (2018) point out other potential barriers to employment of youth (including those with PSE), such as employer preferences for employees with better credentials and work experience (which are typically accumulated with age), and the absence of adequate transitional support mechanisms (from an institutional, policy perspective).

In Canada, there have been 2 previous summative evaluations of the youth employment strategy, which focused on the 2003 to 2006 and 2008 to 2009 program cohorts. The overarching conclusion from these studies was that some streams of YES targeting disadvantaged youth did not achieve their objectives. In comparison for the Career Focus stream, the incremental impact was highly positive and significant. Another recent Canadian study suggests that co-op programs can help ease the transition to the labour market for recent graduates; as well as, positively impact their income in both the short and long term (Wyonch, 2020).

3. Data and methodology

3.1 Scope

The objective of this study is to measure the incremental impacts and economic benefits resulting from participation in the Career Focus stream of the YES. We estimate the program effect through a counterfactual approach. In this approach, we compare the participants’ labour market outcomes to what would have happened had they not received CF interventions. This study also includes a statistical analysis of Career Focus participants’ socioeconomic profiles and outcome trends.

This study examines Career Focus participants who received an intervention during the January 2010 to December 2011 period. This reference period allows us to measure program impacts on labour market outcomes over a period of at least 5 years following the completion of the intervention.

We examine the following evaluation questions as part of this study:

We addressed questions 1 and 2 by examining descriptive statistics on the socio-demographic characteristics and labour market outcomes of the participants. We addressed question 3 by conducting an incremental impact analysis on:

Finally, we answered question 4 through a cost-benefit analysis, using the incremental impact findings as input.

3.2 Data

We used the integrated Labour Market Program Data Platform (LMPDP) as our main data source. The Platform contains program administrative data collected through the Common System for Grants and Contributions, linked to the EI benefits databank and Canada Revenue Agency (CRA) taxation files. The Platform includes a large number of variables that are closely linked to individuals’ labour market experiences, such as:

3.3 Unit of analysis

Throughout this report, the unit of analysis was the individual, with respect to their CF intervention, as all participants received only one intervention.

3.4 Outcome indicators

We measured outcome trends and incremental impacts for the following individual-level labour market indicators:

3.5 Methods for estimating incremental impacts

We built the methodological framework for this study using the same quasi-experimental methods as the evaluation of the LMDAsFootnote 3. The procedure used here measures the effectiveness of the program by comparing the actual outcomes for the participants to their counterfactual outcomesFootnote 4. Because we cannot observe a given individual’s outcomes under both participation and non-participation, an appropriate comparison group is needed to “stand in” as the counterfactual.

We use specialized statistical methods to render the comparison group exchangeable with the participant group. These methods control for all observable factors that can influence both program participation and outcomes, so that the only difference remaining between the 2 groups is participation itself. This exchangeability criterion allows the estimated impacts to be attributed to program participation rather than differences in other factors across the 2 groups. Such differences could include pre-existing socioeconomic characteristics and labour market histories.

For Career Focus, we constructed our comparison group using a subset of individuals aged 15 to 30 who have similar observed characteristics as CF participants. However, the comparison group only received a low-intensity treatment through minor EAS interventions as part of the LMDAs.

We produced the net impact results using a state of-the-art combination of difference-in-differences estimation and propensity score kernel matching methods (Caliendo, 2005). Given the breadth of background variables available in our data, we can be reasonably confident that the impact estimates are robust (Lechner & Wunsch, 2013) in the context of ALMP evaluations. However, there may be some unobserved, relevant characteristics (for example, field of study, confidence, motivation, mental health, etc.) that are not captured in the data. Details on the estimation methods as well as the various validation tests conducted are provided in Appendix A.

This study also investigated how the incremental impact of participating in Career Focus varied among sociodemographic subgroups, such as gender and age (Appendix C). Moreover, we conducted further analyses (discussed in section 3.6) to assess the influence of field of study on net incremental impacts. In particular, these analyses focused on measuring the impact of CF for individuals who worked in science, technology, engineering, and mathematics (STEM) industries versus those who did not.

3.6 Robustness analysis

We conducted a robustness check to examine potential biases that may arise from lack of access to data on the field of study. To do this, we created a proxy for field of study using CRA data on the industries that individuals worked in during their in-program period, broadly categorized in terms of STEM versus non-STEM industries. The methodology for estimating the impacts within these industry subgroups was the same as that used for the overall impact analysis of Career Focus. We examined the distribution of the 5-year annual averages for post-program earnings to check for imbalances related to the selected groups of industry. Moreover, other outcome indicators, such as employment insurance benefits received, income taxes paid and social assistance usage were re-estimated for STEM and non-STEM groups for additional cost-benefit analysis.

We also further partitioned the groups working in the STEM and non-STEM industries by their level of education (that is, college vs. university), and estimated impacts within those categories.

A recent study published by the C.D. Howe Institute (Wyonch, 2020) found that on average, individuals in STEM earned more than their counterparts in other fields of study. This finding led to the conjecture that the overall incremental impact might be levered primarily by the STEM group. Thus, adjusting for field of study should give more robust estimates of program impact. As a result of these findings, we developed a proxy for individuals who (possibly) studied in STEM fields by using industry codes from income tax data to identify participants who worked within STEM-related industries during their CF program participation period. The assumption here is that participants who worked in STEM industries are those who were more likely to have graduated from a STEM field of study. For these participants the estimated treatment effect inform an upper bound of the incremental impact. Conversely, the treatment effect for participants in other fields constitute the lower bound of the incremental impact. The following industry codes were used as proxies for STEM fields of study:

3.7 Strengths and limitations

The primary strength of this study is its reliance on the rich administrative data in the Labour Market Development Platform. The Platform allows for numerous socioeconomic and labour characteristics to be accounted for in the analysis. Additionally, we used a combination of machine learning and econometric techniques to build an appropriate comparison group that is closely matched to the group of CF participants.

A potential limitation is that the universal eligibility of the Career Focus stream under the YES program creates challenges for comparison group selection. This is because there is no readily available data where we can draw comparison cases. For example, in the case of the LMDAs, EI claimants who do not to take interventions naturally form the comparison group. Another limitation is that this analysis requires linking participants’ administrative records to CRA and EI databases and, for privacy reasons, could only be done for ESDC participants but not other government department.

An additional limitation is the lack of access to data on the field of study for participants and comparison cases, which may result in potential biases. To address this limitation, we conducted further analyses based on a proxy for field of study using CRA data on the industries that individuals worked in during their in-program period, broadly categorized in terms of STEM vs. non-STEM industries.

Lastly, using labour market indicators in assessing program effectiveness does not measure unobservable characteristics of youth that might also affect participation and outcomes. Such characteristics may be confidence, motivation, mental health, etc.

4. Participant’s profile and outcome trends

4.1 Career Focus profile

The socio-demographic profiles of Career Focus participants are presented in Table 1 below, which shows CF participants were more likely to be female (56%) than male (43%). The highest proportions among youth aged 25 to 29 (49%), followed by the 20 to 24 age group. Relative to the Canadian youth population at-large, a relatively lower proportion of Career pocus participants self-identified as a visible minority or a person with a disability, whereas the share of participants who self-identified as Indigenous persons was roughly comparable.

Table 1a: Gender distribution of CF participants and their EAS-only comparison group
Gender* Participants
(N=2744)		 Comparison group
(n=5,588)
Male 43% 45%
Female 56% 54%

Table 1b: Age distribution of CF participants and their EAS-only comparison group
Age Participants
(N=2744)		 Comparison group
(n=5,588)
Under 20 3% 3%
20-24 44% 44%
25 and over 54% 53%

Table 1c: Education level of CF participants and their EAS-only comparison group
Education level Participants
(N=2744)		 Comparison group
(n=5,588)
Secondary incomplete 2% 2%
Secondary complete 2% 3%
Post-Secondary incomplete 3% 4%
Post-Secondary complete 90% 88%
Missing/Unknown 2% 3%

Table 1d: Other characteristics of CF participants and their EAS-only comparison group
Other socio demographics characteristics Participants
(N=2744)		 Comparison group
(n=5,588)
Person with disability 1% 3%
Indigenous status 5% 5%
Visible minority 12% 14%
Immigration Status 4% 4%
Married/Common law 17% 17%
Separated / Divorced 1% 1%
Single 79% 79%

Table 1e: Province of residence for CF participants and their EAS-only comparison group
Province/Territory Participants
(N=2744)		 Comparison group
(n=5,588)
Newfoundland and Labrador 2% 1%
Prince Edward Island 1% 1%
Nova Scotia 4% 5%
New Brunswick 2% 3%
Quebec 26% 24%
Ontario 37% 37%
Manitoba 4% 4%
Saskatchewan 2% 2%
Alberta 9% 10%
British Columbia 12% 13%
The North 1% 1%

Table 1f: Labour market characteristics for CF participants and their EAS-only comparison group – 1-year before participation
Key labour market indicators in the year preceding the start of participation Participants
(N=2744)		 Comparison group
(n=5,588)
No earnings 15% 14%
Had earnings 85% 86%
$1 to $5k 23% 22%
$5k to $10k 25% 25%
$10k to $20k 31% 30%
$20k to $40k 18% 18%
$40k to $60k 3% 4%
Over $60k 1% 1%
Received Employment Insurance 10% 11%
Received Social Assistance 2% 3%

4.2 Outcome trends

Figure 1 below shows the average trends in earnings across the 1-year pre-program, in-program, and 5-year post-program period for Career Focus participants. The average annual employment earnings of Career Focus participants increased remarkably during and 1-year post participation, followed by steady increases until the end of the study time horizon.

Figure 1: Trends in average earnings over the 1-year pre-program, in-program, and 5–year post-program period for Career Focus stream

Infographique
Text description of figure 1
Period Average earnings
1 year pre $11,326
In program $20,112
1 year post $27,775
2 year post $29,277
3 year post $31,609
4 year post $33,502
5 year post $36,299

5. Incremental impact

5.1 Overall incremental impact results

The incremental impact analysis shows that on average, CF participants outperformed their comparison group on all labour market indicators (see Table 2 below). In particular, participants had higher average earnings and incidence of employment during the program, and over the 5 years following participation. Career Focus participants were also less dependent on government income supports, and received less Social Assistance income and EI benefits.

Table 2: Incremental impact for Career Focus (CF), 2010 to 2011 cohort (n=2,744)
Indicators Program start year
(0)		 1 yr post-program
(1)		 2 yr post-program
(2)		 3 yr post-program
(3)		 4 yr post-program
(4)		 5 yr post-program
(5)		 Annual average post-program
(1-5)		 Cumulative in & post-program
(0-5)
Total reported earnings ($) 7156*** 6391*** 4901*** 5248*** 5741*** 5926*** 5642*** 35366***
Employment Income ($)+ 7038*** 6270*** 4761*** 5043*** 5712*** 5888*** 5535*** 34715***
Incidence of employment (pp) 7.2*** 5.78*** 2.94** 3.21** 3** 2.82* 3.86*** N/A
EI benefits ($) -459*** 134* 89 78 -53 -50 34 -270
SA benefits ($) -82* -148*** -106** -98** -138*** -112** -126*** -710***
Dependence on income support (pp) -4.48*** -1.51* -1.54* -1.05 -1.77** -2.5*** -1.89*** N/A

The main incremental impact findings for specific sociodemographic subgroups are as follows, using the same time horizon as for the overall analyses:

5.2 Variability in incremental impacts results by industry of placement

As mentioned previously, there could be potential biases in the estimates of Career Focus impacts, due to the absence of information for field of study. Thus, we conducted a sensitivity analysis to examine whether the heterogeneity of the incremental impacts by the field of study. First, we developed a proxy for individuals who (possibly) studied in STEM fields of study by using industry codes from income tax data. Subsequently, we conducted the net impact analysis according to industry of work (that is, STEM and non-STEM), and by level of education (that is, college and university).

Table 3 below provides the net impacts on five-year post-program average earnings for the CF participants working in STEM and non-STEM industries. The results suggest that the net impact on earnings for participants who worked in STEM industries is substantially higher than the impacts for participants in non-STEM industries.Footnote 5 We can therefore interpret the results for these two groups as constituting a lower and upper bounds of the net impact on earnings for participants.

Moreover, as shown in table 4, the analysis points to incremental impacts on income of Career Focus ranging from approximately $2,500 to about $10,000 per year. These estimates are generally consistent with the results found in a recent paper by the C.D. Howe Institute (Wyonch, 2020) on the impact of co-op programs, by field of study.

Table 3: Incremental impact on earnings by in-program industry of work
Main categories Participant Average over 5 years Min over 5 years post Max over 5 years post
STEM industries 861 (31%) $9,581*** $8,931*** $10,141***
Non-STEM industries 1,883 (69%) $3,235*** $2,488*** $4,098***
Career Focus (Overall) 2,744 $5,535*** $6,391*** $4,901***

5.3 Interaction between field and level of education

Using an approach similar to the one undertaken by the C.D. Howe Institute (Wyonch, 2020), we estimated the net incremental impact on 5-year post average earnings by of field of study and highest level of education. Note that unlike the C.D. Howe Institute’s approach, the current analysis does not have information on actual field of study. Instead, it relies on a proxy under the premise that the industry of work during the in-program period most likely corresponds to the participants’ actual field of study. It is recognized that these may not be identical in some cases. The findings presented in Table 4 provide additional nuances around the previous results, showing that participants working in non-STEM industries (about 70% of the CF population) achieved the same benefits from the program regardless of whether they were in university or college. However, participants with a university education who worked in STEM-related industries experienced appreciably higher benefits from the program compare with their college counterparts.

These results are generally consistent with those found in the C.D. Howe study. Some additional limitations of this approach include: (1) the limited sample size for the population working in STEM industries; and (2) lack of information on the full educational trajectory for individuals. For instance, some university graduates may have also subsequently attended college, in order to gain more hands-on experience in their fields.

Table 4: Incremental impacts of Career Focus at the education-industry intersection
Main categories Participant Average over 5 years Min observed impact
over 1-5 yrs		 Max observed impact
over 1-5 yrs
Non-STEM & University 1,020 (37%) $3,569*** $2,788*** $4,113***
Non-STEM & College 639 (23%) $3,760*** $2,800*** $4,449***
STEM & University 462 (17%) $13,290*** $12,133 $14,509***
STEM & College 295 (11%) $5,681*** $4,458*** $7,793***
Career Focus - Overall 2,744 (100%) $5,535*** $4,901*** $6,391***

6. Cost benefit analysis

To inform the economic value of a given labour market program, we need methodological frameworks to quantify the various types of costs and benefits involvedFootnote 6. In particular, cost-benefit analysis (CBA) is a key analytical tool used to highlight the distribution of program costs and benefits from a participants’ and governments’ perspective.

This section presents the benefits and costs associated with participation in the Career Focus stream of the YES program. The costs and benefits of Career Focus accumulate to individual participants, governments and society over the short- and long-term (see Appendix A for further details). The impacts of the program begin with participation in Career Focus, which results in participants improving their employability skills and leading to higher earnings’ profiles relative to the earnings’ profiles of non-participants. The cost-benefit analysis compared the financial costs incurred by the Government in delivering the program with the benefits generated by participation (for example, increases in employment earnings, increased tax revenues, decreases in social assistance use).

In addition, given that the program participants were youth, a longer time horizon was required for fully assessing economic returns while meeting the Treasury Board’s guide on Canadian Cost-Benefit Analysis. Therefore, the CBA is based on an observed 5-year period and an additional 5-year projection. It is estimated by computing the difference between the discounted observed and projected net incremental benefits and costs from individual and government perspectives.

As described in more detail below, the CBA focuses on the following three indicators, each with respect to individual-specific benefits (for example, increases in gross employment earnings), government-specific benefits (for example, increases in tax revenues), and societal (individual + government) benefits:

  1. the Net Present Value (NPV): calculated by subtracting the program cost from the discounted benefits
  2. Social Rate of Return: calculated as the total net benefits to individuals and government divided by absolute value of total program costs
  3. payback period: amount of time required for the discounted benefits to equal the discounted costs. If the discounted benefits were inferior to the discounted costs 5 years post-participation, the impacts over the following years were extrapolated depending on the projection scenario used

It is also important to highlight up front that a CBA only provides an approximation of the costs and benefits of program participation. This is due to the difficulties in attributing an exact dollar value to both the full range of benefits (Mason & Tereraho, 2007) stemming from participation and the costs associated with program delivery. However, we addressed this uncertainty through the use of sensitivity analyses on our guiding assumptions.

6.1 Methodology

6.1.1 Costs included in the analysis

Program costs. The program costs are those incurred by the government in delivering the Career Focus program from January 1, 2010 to December 31, 2011. These costs include the administration costs and the direct costs of the program (that is, wage-subsidies received by participating employers and organisations). Table 5 provides a summary of the estimated program costs for the Career Focus stream, including the cost per participant.

Table 5: Estimated program cost per participant for Career Focus (2010 to 2011)
Program name Program cost Administration cost Total cost Number of participants Cost per participant
Career Focus $20,801,000 $2,600,000 $23,401,000 2,744 $8,528
The Marginal Social Cost of Public Funds (MSCPF).

In addition, to estimate program costs, we take into account the costs incurred by society associated with raising additional revenues to fund government spending. This is also referred to the Marginal Social Cost of Public Funds (MSCPF). Based on expert advice received during past evaluation studies, the Marginal Social Cost of Public Funds is calculated as follows:

(program costs - impacts on sales taxes - impacts on income taxes - impacts on EI - impacts on social assistance)*20%

In other words, the MSCPF represents 20% of the program cost and subsequent impacts on sales taxes, income taxes as well as on EI and SA outlays. The value of the MSCPF will be calculated from the costs and benefits for the government, and will also be taken into account when calculating the NPV, cost-benefit ratio, and payback period for society. However, to provide a conservative estimate on Net Present Benefit from the governmental perspective, the MSCPF was only applied to direct program costs.

6.1.2 Benefits included in analysis

Employment earnings. Employment earnings benefits consist of incremental impacts on participants’ employment earnings during and after participation (estimated using propensity score matching). Fringe benefits. Fringe benefits (FB) from employment are benefits that supplement employment earnings. Fringe benefits include (but are not limited to) employer-paid health and life insurance, Canada Pension Plan (CPP)/Québec Pension Plan (QPP) contributions, and employer-sponsored pension plan contributions. In line with the methods used by the Ministère et de la Solidarité Sociale du Québec (2006) and the Social Research and Demonstration Corporation (2002), fringe benefits were assumed to comprise 15% of the incremental impact on employment earnings.

Formally, the incremental impact on fringe benefits is defined as:

∆FB = α ∆y where:

Federal and provincial income taxes. The differences between the employment earnings profiles of participants and non-participants also affect their respective amount of federal and provincial income taxes paid. Federal and provincial income taxes paid and the incremental impact on taxes paid were derived from the Canada Revenue Agency tax data. Similar to the impacts on employment earnings, propensity score matching was used to estimate the incremental impact on taxes paid by the participants.

Sales taxes. An increase in employment earnings also increases participants’ sales tax payments, as a result of additional purchasing power. In line with the methods used by the Ministère de l’Emploi et de la Solidarité Sociale du Québec (2006), the sales taxes paid will be simulated by assuming that part of the change in employment earnings is devoted to taxable consumer goods. The proportion of employment earnings spent on taxable goods is estimated at 51% from 2012 to 2016Footnote 7.

However, as not all goods are taxable, the average share of spending typically used to buy taxable goods and services was estimated using data from Statistics Canada’s Survey of Household Spending (Table 11-10-0223-01 Household spending by household income quintile, Canada, regions and provinces). This proportion is estimated at 51% for 2012 to 2016. The sales tax rate is estimated as the sum of the federal sales taxes rate and the weighted average of provincial sales tax rates. The national weighted average of the provincial tax rates between 2010 and 2017 is 7%, whereas the average federal tax rate is 5%. The national average of the propensity to consume between 2012 and 2016 is 0.956 (1-0.044).

Formally, the impact on sales taxes paid is expressed as follows:

∆STP = (tp + tf ) ∆y * (1 –PMS) *‬ λ

where:

Employment Insurance and Social Assistance benefits. Represent the differences in the amount of EI and SA benefits received between participants and their comparison group (EAS), after participation in the initiative. Thus, the reduction of SA and EI benefit use would represent an increase in benefits from the government perspective. Incremental impacts on SA and EI benefits were, similar to employment earnings, estimated using propensity score matching.

Employment Insurance premium. An increase in employment earnings changes the EI premium rate. To account for these fluctuations, the EI premium was identified from Canada Revenue Agency tax data. The incremental impacts on EI premiums were then estimated. As with the impacts on employment earnings, propensity score matching was used to estimate the incremental impacts on EI premiums paid by the program participants.

6.2 Approach for estimating and reporting cost-benefit results

6.2.1 Time frame and discount rate

Program costs were observed at the program start year, with benefits calculated over an 11-year period consisting of the in-program period, an observed 5-year post-program period, and an additional 5-year forecast period.

Because the program costs are incurred at the beginning of participation and benefits are experienced in the years after, all benefits were discounted by 5% starting with the year following the participation start year. The current choice of 5% is representative of the increase in interest on foregone government investmentFootnote 8.

6.2.2 Conceptual and accounting framework

An important aspect of the cost-benefit analysis is to determine who bears a particular cost or benefit (that is, participant, government, and society). What can be a benefit from one perspective could be a cost from another perspective. For example, a decrease in EI benefits received can be viewed as a reduction in income for the participant (that is, a cost), but as a benefit for the government. The costs and benefits assigned to each perspective are identified in Table 6.

The societal perspective combines the participant and government perspectives. For a given factor, a net gain to society occurs only when a gain to one group was not at the expense of another group. For example, higher employment earnings after participation represent a benefit for participants and are neither a benefit nor a cost to the government. Thus, the net result is a gain for society. A cost to society occurs when a factor is a cost from one perspective and not a gain from the other perspective. For example, program costs represent a cost to the government, but not to participants; thus, they are considered a cost to society. Factors that constitute a net gain from one perspective but a net loss from the other perspective are equal to 0 from the societal perspective. For example, EI benefit reductions may represent a cost to participants and a benefit to the government, and neither a cost nor a benefit for society.

Table 6a: Costs accounting framework from the perspectives of participants, government, and society
Costs factors included Participants Government Society Estimation methods
Program cost 0 - - Cost estimates are based on program expenditure and intervention file data
MSCPF 0 - - Estimates are measured as 20% of program costs

Table 6b: Benefits accounting framework from the perspectives of participants, government, and society
Benefits factors included Participants Government Society Estimation methods
Employment earnings + 0 + Observed period values are based on incremental impacts findings, whereas unobserved/ future period are based on fixed average observed periods’ incremental impacts. This includes during program earnings which is associated with the wage subsidy (depending on wage subsidy received by the employer).
Fringe benefits + 0 + Estimates are measured as 15% of employment earnings
Federal and provincial income taxes - + 0 Estimates are based on earnings, federal and provincial income tax rate for future years, and trend function
Federal and provincial sale taxes - + 0 Incremental impacts on net total reported earnings are multiplied by the propensity to consume (96%), the proportion of household spending on taxable goods and service (50%) and by the total average federal and provincial sales tax rate (12%)
EI -/+ -/+ 0 Observed period values are based on incremental impacts findings, whereas unobserved/ future period are based on fixed average observed periods’ incremental impacts
SA -/+ -/+ 0 Observed period values are based on incremental impacts findings, whereas unobserved/ future period are based on fixed average observed periods’ incremental impacts
EI Premium -/+ -/+ 0 Observed period values are based on incremental impacts findings, whereas unobserved/ future period are based on fixed average observed periods’ incremental impacts

6.3 Results

Results from the cost-benefit analysis are explained below and presented in Figure 2 and 3 for the following key indicators:

6.3.1 Net Present Value

Figure 2 (following page) presents the Net Present Value (NPV) of the costs and benefits for Career Focus over the 1-year in program period and 10-year post-participation period, based on the EAS comparison group.

As previously discussed, future Career Focus impacts such as earnings, federal and provincial income taxes, EI benefits, SA benefits, and EI premiums were projected based on the trend over the last 4 years of the observed post-participation period. The NPV indicated that from a societal perspective, the benefits of participation exceeded the costs by $46,293 during program participation and over the 10-year post participation. The societal benefits arose primarily from benefits to participants ($42,272), with a much smaller government benefits ($4,021).

6.3.2 Annualized social rate of return

The annualized social rate of return for Career Focus was 543% over an 11 year time horizon (1-year in-program plus a 10 year post-participation period), which equates to an 18% annual rate of return.

6.3.3 Payback period

As illustrated in Figure 2, from the government’s perspective, it would take 7.4 years (from the start of the program) for future benefits associated with the program to outweigh its initial costs. Whereas from a societal perspective, it would take only about 4 months (0.33 years). Note also that the societal payback period falls within the observed time frame (5-years post-participation), and therefore the projection methods applied in this study are irrelevant in this regard.

Figure 2: Illustrative example of how the long-term benefits of the CF program exceeds its initial cost from a Government Perspective

Figure 2: Illustrative example of how the long-term benefits of the CF program exceeds its initial cost from a Government Perspective
Text description of figure 2
Period Upfront cost
(plus 20% MSCF)		 Accumulated benefit Accumulated cost
(excluding up front cost)		 Accumulated cost
(including up front cost)
In program -$10,234 $1,392 $ - -$10,234
1 year post -$10,234 $2,991 -$134 -$10,368
2 years post -$10,234 $4,450 -$224 -$10,592
3 years post -$10,234 $6,021 -$303 -$10,895
4 years post -$10,234 $7,899 -$303 -$11,198
5 years post -$10,234 $9,966 -$303 -$11,501
6 years post -$10,234 $11,685 -$320 -$11,820 *
7 years post -$10,234 $13,403 -$336 -$12,157
8 years post -$10,234 $15,122 -$353 -$12,510
9 years post -$10,234 $16,840 -$370 -$12,880
10 years post -$10,234 $18,558 -$387 -$13,267

* break-even point


Figure 3: Costs and benefits per participant over 11 years, net present value

Figure 3: Costs and benefits per participant over 11 years, net present value
  • Source: Administrative Data - Labour Market Program Data Platform cohorts of Career Focus participants from January 1, 2010 to December 31, 2011.
  • 1. Cost includes the program direct cost per participant (estimated at $8,528) plus the marginal cost of public funds (estimated at $1,706; that is 20% of the program direct cost per participant).
  • 2. The social rate of return (543%) is the net benefit to the society ($46,293) divided by the program direct cost per participant. The annual average social rate of return of 18% per year is equal to 5.43 (543%) plus 1, raised to the power of 1 divided by the number of years covered by the cost-benefit analysis (that is, 11 years).
  • 3. Fringe benefit is defined as a proportion of compensation given to employees in addition to wages which approximately represents 15% of the employment earnings.
Text description of figure 3
Net present value for government
Components analysis Government
Cost ($) * -$10,234
EI & SA benefits ($)** $1,192
Sales tax paid ($)** $2,819
Income Tax Received $10,244
Net present value $4,021

Net present value for participants
Components analysis Government
Income Tax Paid -$10,244
Sales tax paid ($) -$2,819
EI & SA benefits ($) -$1,192
Fringe benefit ($)*** $7,373
Earnings*** ($) $49,154
Net present value $42,272

6.4 Challenges and limitations

The CBA was based on major quantifiable benefits and costs that are available in the administrative and Canada Revenue Agency tax data files. Although the calculations take into account the majority of the key costs and benefits, some are estimated, or were simply not included due to data limitations.

Participation costs: Participants generally incur some personal costs related to participation in Career Focus, such as travel, tuition, training or job search-related expenses. Unfortunately, given that the administrative data or other available information sources do not provide information on out-of-pocket costs paid by participants, it was not possible to include this information in the analysis.

Tax benefit: Moreover, the analysis does not take into account the Canada Child Benefit (CCB), formerly known as Canadian Child Tax Benefit (CCTB), because it depends on the number of children below 18 (information that is not available in the administrative databases that were used). However, such benefits are likely to be small or largely inapplicable in the case of Career Focus participants.

Indirect benefits: In addition, the CBA does not capture “intangible”, non-pecuniary, and indirect benefits stemming from finding employment. Intangible benefits can include improvement of participants’ mental and physical well-being associated with increased income and reduced duration of unemploymentFootnote 9. However, these hypotheses cannot yet be directly confirmed, as the microdata files required for this type of analysis are currently neither integrated nor accessible to ESDC.

Finally, the CBA does not consider the “displacement effect” (for example, Dahlberg & Forslund, 2005): participants may take away jobs that would otherwise be obtained by other unemployed individuals.

7. Sensitivity analysis

The CBA is based on a number of assumptions about the input parameters, which are considered to be plausible in the current context. However, there will always be uncertainty involved, and therefore we conducted a sensitivity analysis to determine the effect of a reasonable variation in the specific inputs. Table 10 below presents net present value and rate of return over 11 years under various scenarios from all perspectives.

7.1 Economic assumptions

Discount rate: Whereas the principal analysis relied on a discount rate of 5%, the sensitivity analysis examines the variation in the NPV under discount rates of 3%, and 8%. As well, we examine discount rates in combination with different MSCPF scenarios and time horizons for costs and benefits. These alternative rates are in line with the interim Treasury Board Secretariat of Canada (2007) guide for Cost-Benefit Analysis (the “TBS Guide”), which suggests a social discount rate of 3% and 8% percent depending on the circumstances.

As per table 10, cost-benefit outcome indicators show some inverse sensitivity to discount rate assumptions from all perspectives. For instance, both net present value increases when discount rate decreases from 5% to 3% (while keeping MSCPF fixed at 20%) since we are discounting at a lower rate (dividing the future cash flow by a smaller value).

The MSCPF: In the principal analysis, the MSCPF was calculated as 20% of the program cost. For testing a different MSCPF scenario, it is critical to take into account that when youth reach their full potential, Canadians, government and the Canadian economy benefit. Because youth represent one third of the Canadian population, investing in programs that help youth overcome barriers to employment, improves participation in this rapidly growing job market.

Due to the sensitive nature of MSCPF to taxes, we assumed a lower MSCPF of 10% for the sensitivity analysis. Table 10 shows that this change in MSCPF assumption only changes outcomes from a Government perspective, since MSCPF is only added to government costs. Decreasing the MSCPF by 10% (from 20% to 10%), decreases government costs, which positively affects the net present value (net present value being the difference between net benefit present value and net cost present value).

7.2 Field of study

As discussed in Section 5, the net incremental impacts on earnings were highly dependent on field of study. Therefore, we conducted a complementary CBA to determine the corresponding variation in the economic benefits arising from participation, for those who worked in STEM and non-STEM industries. Because industry was itself a sensitivity factor, we did not vary the other input parameters (that is, the original 5% discount rate and a MSCPF of 20% were used).

As discussed in section 5.2, the field of study greatly influences the net incremental impact on employment income. Participants who worked in STEM related industries experienced a much higher impact on income compared to those who worked in non-STEM industries. These impacts translated into a higher net present value and rate of return, as shown in Table 7. Relative to both the non-STEM industry scenario and the overall scenario for all Career Focus participants, the net present value and social rate of return improves substantially under the STEM scenario, and from all perspectives.

Table 7a: Sensitivity analysis for CBA
Scenario Discount rate MSCPF Participant Net Present Value
over 11 years		 Government Net Present Value
over 11 years		 Society Net Present Value
over 11 years		 Total social rate of return Annual social rate of return
Base 5% 20% $42,271 $4,021 $46,293 542.8% 18.4%
Scenario 1 5% 10% $42,271 $4,874 $47,146 552.8% 18.6%
Scenario 2 3% 20% $45,911 $5,411 $51,322 601.8% 19.4%
Scenario 3 3% 10% $45,911 $6,264 $52,175 611.8% 19.5%
Scenario 4 8% 20% $37,713 $2,284 $39,997 469% 17.1%
Scenario 5 8% 10% $37,713 $3,137 $40,850 479% 17.3%

Notes

  1. Timeframe is 10-years post participation plus 1-year in-program period.
  2. EAS- Comparison Group.

Table 7b: Sensitivity analysis for CBA – STEM and non-STEM
Scenario Discount rate MSCPF Participant Net Present Value
over 11 years		 Government Net Present Value
over 11 years		 Society Net Present Value
over 11 years		 Total social rate of return Annual social rate of return
STEM 5% 20% $69,789 $17,199 $86,988 1,020% 24.6%
non-STEM 5% 20% $25,908 -$1,036 $24,873 292% 13%

Notes

  1. Timeframe is 10-years post participation plus 1-year in-program period.
  2. EAS- Comparison Group.

8. Conclusion

Overall, the results indicate that participating in Career Focus leads to a substantial gain in earnings and employment. The analysis by subgroups (Annex C) shows that there was some heterogeneity among CF participants. Whereas all subgroups had very strong earnings and employment gains, these impacts were greater in some subgroups.

The main estimation method used to determine the incremental impacts was propensity score kernel matching combined with the difference-in-differences technique. The econometric model was based on rich information about individuals’ labour market experience, including socio-demographic characteristics as well as labour market history. Other personal and environmental factors, such as motivation, poor self and behavioural management, criminal history, drug use, and lack of social supports can also affect both participation and outcomes. However, these factors are not captured in the available data, and therefore may induce some bias in the results to the extent that they may not be equally distributed into the treatment and comparison groups.

Participants experienced considerable and sustained increases in earnings and employment relative to their comparison group. These findings are consistent with the previous literature and evaluation studies. Programs such as Career Focus can play a role in easing the transition of recent graduates into the labour market, with associated positive long-term impacts.

Moreover, cost-benefit analysis showed that over the 10-year post participation period, the Career Focus program resulted in a significant net positive return on investment from all perspectives. The sensitivity analyses, which used industry of work as a proxy for fields of study, showed large variation in results. However, changing the economic assumptions of the CBA produced only minor fluctuations in the results.

References

  1. Benach, J et al. (2010). A Micro-level Model of Employment Relations and Health Inequities. International Journal of Health Services. Volume 40 Number 2, 223-227.
  2. Bjerk, D. (2004). Youth criminal participation and household economic status. Department of Economics Working Paper. Hamilton, ON: McMaster University.
  3. Butler-Jones, D. (2008). The Chief Public Health Officer’s Report on the State of Public Health in Canada, 2008. Ottawa, ON: Public Health Agency of Canada.
  4. Caliendo, M., and S. Kopeinig (2005): “Some Practical Guidance for the Implementation of Propensity Score Matching,” Discussion Paper No. 1588, IZA
  5. Calmfors, L., & Lang, H. (1993). Macroeconomic effects of active labour market programmes- the basic theory. Stockholm, Sweden: Institute for International Economic Studies
  6. Canadian Institute for Health Information (2018). National Health Expenditure Trends, 1975 to 2018. Ottawa, ON: CIHI.
  7. Dahlberg, M., & Forslund, A. (2005). Direct displacement effects of labour market programs. Scandinavian Journal of Economics, 107, 475-94.
  8. Dahlby, B & Ferede, E. (2011). What Does It Cost Society to Raise a Dollar of Tax Revenue?, The Marginal Cost of Public Funds. C.D. Howe Institute, No. 324.
  9. Dhakal, S.P., Connell, J., & Burgess, J. (2018). Inclusion and work: addressing the global challenges for youth employment. Equality, Diversity and Inclusion: An International Journal, 37(2), 110-120. doi: 10.1108/EDI-12-2017-0290
  10. Duncan, G. (2005). Income and child well-being. Geary Lecture at the Economic and Social Research Institute. Dublin, Ireland.
  11. ESDC (2015), Summative evaluation of the Horizontal Youth Employment Strategy https://www.canada.ca/en/employment-social-development/corporate/reports/evaluations/2015-youth-employment.html
  12. Grün, C., Hauser, W., & Rhein, T. (2010). Is any job better than no job? Life satisfaction and reemployment. Journal of Labor Research, 31(3), 285-306.
  13. Haveman, R.H., & Weimer, D.L. (2015). Public Policy Induced Changes in Employment: Valuation Issues for Benefit-Cost Analysis. Journal of Benefit-Cost Analysis, 6(1), 112-153.
  14. Heckman, J. J. and J. A. Smith (1999): “The Pre-Program Earnings Dip and the Determinants of Participation in a Social Program: Implications for Simple Program Evaluation Strategies,” Economic Journal, 109(457), 313–348.
  15. Hoskins, B., Leonard, P., & Wilde, R.J. (2018). Negotiating uncertain economic times: Youth employment strategies in England. British Educational Research Journal, 44(1), 61–79. doi: 10.1002/berj.3318
  16. https://www.canada.ca/en/treasury-board-secretariat/corporate/reports/review-methodologies-determine-discount-rates.html#toc5
  17. International Journal of Health Services, Volume 40, Number 2, Pages 195–207, 2010 © 2010, Baywood Publishing Co., Inc.
  18. Jackson, P. R., & Warr, P. B. (1984). Unemployment and psychological ill-health: The moderating role of duration and age. Psychological Medicine, 14(3), 605-614.
  19. Jaison R. Abel & Richard Deitz & Yaquin Su, 2014. "Are recent college graduates finding good jobs?,"Current Issues in Economics and Finance, Federal Reserve Bank of New York, vol. 20.
  20. Kalleberg, A. L. (2020). Labor Market Uncertainties and Youth Labor Force Experiences: Lessons Learned. The ANNALS of the American Academy of Political and Social Science, 688(1), 258–270.
    https://doi.org/10.1177/0002716220913861
  21. Kluve, J. Youth Labor Market Interventions. IZA World of Labor 2014: 106 doi: 10.15185/izawol.106
  22. Kluve, J, Puerto, S et al. 2016, Interventions to Improve the Labour Market Outcomes of Youth: a systematic review of training, entrepreneurship promotion, employment services and subsidized employment interventions, Campbell systematic reviews, ILO, Geneva, viewed 27 Feb 2020.
    https://www.ilo.org/employment/Whatwedo/Publications/WCMS_508938/lang--en/index.htm
  23. Krahn, H. J., Howard, A. L., & Galambos, N. L. (2015). Exploring or Floundering? The Meaning of Employment and Educational Fluctuations in Emerging Adulthood. Youth & Society, 47(2), 245–266. https://doi.org/10.1177/0044118X12459061
  24. Lechner, M. (2019) "Modified Causal Forests for Estimating Heterogeneous Causal Effects," Economics Working Paper Series 1901, University of St. Gallen, School of Economics and Political Science.
  25. Lechner, M. and C. WUNSCH (2011) “Sensitivity of Matching-Based Program Evaluations of the Availability of Control Variables,” CESifo Working Paper No. 3381.
  26. Leung, A. (2004). The Cost of Pain and Suffering from Crime in Canada. Research and Statistics Division, Department of Justice Canada. Retrieved from:
    https://www.justice.gc.ca/eng/rp-pr/cj-jp/victim/rr05_4/rr05_4.pdf
  27. Mason, G., Buchel, A., Haddad, T., & Unruh, D. (2013). Assessing the value for money of active labour market programming for persons with disabilities. The Canadian Journal of Program Evaluation, 27(2), 21–38.
  28. Mason & Tereraho (2007). Value-for-money analysis of active labour market programs. The Canadian Journal of Program Evaluation. 22(1), 1–29.
  29. McKee-Ryan, F., Song, Z., Wanberg, C. R., & Kinicki, A. J. (2005). Psychological and physical well-being during unemployment: a meta-analytic study. Journal of Applied Psychology, 90(1), 53-76.
  30. Mendolia, S. (2014). The impact of husbands’ job loss on partners’ mental health. Review of Economics of the Household. 12(2), 277-94.
  31. Ministère et de la Solidarité Sociale Québec (2006). Étude sur le rendement de l’investissement relié à la participation aux mesures actives offertes aux individus par Emploi-Québec. Rapport d’évaluation.
  32. Oreopolos, P., Page, M., & Stevens, A. (2005). The intergenerational effect of worker displacement. National Bureau of Economic Research Working Paper, no. 1587.
  33. Price, R. (1992). Psychological impact of job loss on individuals and families. Current Directions in Psychological Science. 1(1), 9-10.
  34. Public Health Agency of Canada. 2016. The Direct Economic Burden of Socio-Economic Health Inequalities in Canada: An Analysis of Health Care Costs by Income Level.
    http://publications.gc.ca/collections/collection_2018/aspc-phac/HP35-38-2014-eng.pdf
  35. Refrigeri, L., & Aleandri, G. (2013). Educational policies and youth unemployment. Procedia - Social and Behavioral Sciences, 93, 1263–1268. doi: 10.1016/j.sbspro.2013.10.026
  36. Rosalie Wyonch, 2020. "Work-Ready Graduates: The Role of Co-op Programs in Labour Market Success," C.D. Howe Institute Commentary, C.D. Howe Institute, issue 562, January 2020.
  37. Rosenbaum, P. R. and D. B. Rubin (1983): “The Central Role of the Propensity Score in Observational Studies for Causal Effects,” Biometrika, 70(1), 41–50.
  38. Social Research and Demonstration Corporation (2002). Making Work Pay. Final Report on the Self-Sufficiency Project for Long-Term Welfare Recipients. Retrieved from:
    http://www.srdc.org/media/11007/SSP54.pdf
  39. Treasury Board Secretariat of Canada (2007). Canadian Cost-Benefit Analysis Guide: Regulatory Proposals. (Catalogue Number: BT58-5/2007). Retrieved from:
    https://www.tbs-sct.gc.ca/rtrap-parfa/analys/analys-eng.pdf
  40. Wyonch, Rosalie. Work-Ready Graduates: The Role of Co-op Programs in Labour Market Success. C.D. Howe Institute Commentary 562.
    https://www.cdhowe.org/sites/default/files/attachments/research_papers/mixed/Commentary%20562.pdf
  41. Yeung, W.-J. J., & Yang, Y. (2020). Labor Market Uncertainties for Youth and Young Adults: An International Perspective. The ANNALS of the American Academy of Political and Social Science, 688(1), 7–19.
    https://doi.org/10.1177/0002716220913487

Appendix A: Additional methodology details

Propensity score matching methods

Program impacts were measured by using a propensity-score based weighting approach to equate the participant and comparison groups on observed characteristics. Kernel matching was the primary method used to match participants to comparison cases based on their propensity scores. Kernel matching uses the entire control group and re-weights control group members each time they are compared with a new participant, based on propensity-score differentials. The weights are assigned according to a probability density function, called the “kernel” such that small differentials result in large weights, and vice versa.

Inverse Probability Weighting (IPW) and Nearest Neighbour matching approach were used as an alternative method for validation purposes. Note that all of these methods were combined with Difference-in–Differences (DID). DID estimation allows for removing bias from time-invariant unobserved characteristics, by assuming common time trends in the pre- and post-participation period in these unobservable characteristics.

Propensity score model

The first step involved in conducting a matching approach is to estimate the propensity scores of both program participants and comparison cases. Development of a propensity score is based on a logistic regression model to predict the likelihood of participating in CF, based on the background characteristics of CF participants and the comparison group members.

The propensity score model is based on a rich set of background variables (see Table 8 for a summary), including socio-demographic variables, labour market history of individuals (including EI benefits, incidence of employment, and employment earnings in the 5 years preceding program entry). All of the variables included in the estimation of the propensity score are considered to be influential for the participation decision, based on economic theory.

Table 8: Summary of Propensity Score variables*
Socio-Demographics variables Labour market characteristics
  1. Age
  2. Gender
  3. Indigenous status
  4. Person with disability
  5. Immigrant
  6. Visible minority
  7. Marital status
  8. Education level
  9. Province of residence
  10. Urban, rural indicator
  1. Reason for job separation
  2. Industry (NAICS)
  3. New entrant to labour force
  4. Past interventions taken (Employment Benefits or Support Measures)
  5. Past Earnings up to 5 years
  6. Past EI Benefit usage up to 5 years
  7. Past social assistance income up to 5 years
  8. Past tuition amounts up to 5 years

*Disaggregating these variables can potentially provide up to 75 more granular variables, for use in the propensity score model.

Machine learning to select a subset of EAS participants as comparison group

As mentioned, the comparison group for Career Focus participants consisted of individuals who shared the same eligibility criteria as CF participants, but had received only minor interventions via EAS as part of the LMDAs. EAS clients may have received a wide range of one-on-one interventions to support their return to work, including computer training, career advice and employment counselling, job preparation and job search supports, job-finding clubs, and short-term training such as first aid certification or food safety.

To identify a comparison group (that is, EAS) of individuals with similar socio-demographics and labour market characteristics as Career Focus participants, a machine learning approach called ‘K-Nearest Neighbor (KNN) discriminative analysis’ was used. K-Nearest Neighbor is a supervised machine learning method that is used to classify individuals with similar characteristics into different groups. The variables used as input to the KNN algorithms were the same set used in the propensity score model. The final set of comparison group members selected for the Career Focus stream includes 5,610 EAS participants.

Balancing test results before and after matching

To ensure that impact estimates are reliable, balancing property tests were carried out on each of the covariates used in the propensity score models before and after matching. Tests for equality of means (t-test) and the standardized differences test (Rosenbaum and Rubin, 1985) were used. The summary of the balancing test results are displayed in Table 3 below. Based on the values of Pseudo R2 and the Mean Bias, the results suggest that, before matching, there were some small differences between the participants and the comparison cases. However, after matching, all of the significant differences virtually disappeared. In addition, the test of standardized differences confirmed that there were no large differences in the covariates between participants and comparison cases in the original sample (that is, before matching), showing that KNN procedure was highly successful in the initial identification of the comparison group. Only a small number of the variables had standardized difference values greater than 20% which is the rule of thumb used by Rosenbaum and Rubin (1985). After matching, the results show a considerable reduction in the standardized difference values for many of the covariates. It should also be noted that three participants were outside of the common support region (that is, their propensity scores were extreme and thus non-overlapping with the rest of the cases), and therefore excluded from the analysis, as no comparable matches for them could be found in the comparison group.

Table 9: Mean Standardized Bias and Pseudo R²(Kernel Matching Approach)
Program # Participants
before matching		 # Participants
after Matching*		 Pseudo R2
before matching		 Pseudo R2
after matching		 Mean Bias
before matching		 Mean Bias
after matching		 Off-CS**
CF 2,744 2,741 0.13 0.01 6.8 1.9 3

Appendix B: Cost-benefit conceptual framework

Figure 4: Net Social Cost-Benefit

Figure 4: Net Social Cost-Benefit
Text description of figure 4

Figure 4 presents a conceptual framework for how costs and short and long-term benefits might accrue from participation in the Career Focus program for individual, government, and society.

For individual:

  • there are no short-term costs
  • the short-term benefits are:
    • gain work experience
    • obtain higher quality employment
  • the long-term benefits are:
    • greater lifetime earnings
    • improved social outcomes

For government:

  • the short-term costs are:
    • program costs
    • marginal social costs of public funds
  • the short and long-term benefits are:
    • increased tax revenue from income and sales tax
    • decreased reliance on employment insurance, social assistance and other related labour market programming
    • decreased expenditures related to government social programming (for example, health, justice)
    • unobserved social benefits and societal engagement

For society (that is both government and individuals):

  • the short-term costs are:
    • program costs
    • marginal social costs of public funds
  • the short-term benefits are:
    • gain work experience
    • obtain higher quality employment
  • the long-term benefits are:
    • greater lifetime earnings
    • decreased reliance on employment insurance, social assistance and other related labour market programming
    • decreased expenditures related to government social programming (for example, health, justice)
    • unobserved social benefits and societal engagement

This graphic presents a conceptual framework for how costs and benefits might accrue from participation in the Career Focus program. While not intended to be exhaustive, it includes elements that are not part of the net benefit analysis due to insufficient information on how to quantify the specific cost or benefit, such as the gains from potential improvements to health outcomes, but are reasonably expected to have an effect.

Appendix C: Subgroup incremental impact results

Table 10a: Male (n=1,188) Incremental impact for Career Focus participants using EAS as comparison group
Indicators Program start year
(0)		 1 yr post-program
(1)		 2 yr post-program
(2)		 3 yr post-program
(3)		 4 yr post-program
(4)		 5 yr post-program
(5)		 Cumulative post-program
(1-5)		 Annual average post-program
Employment Income ($) 8552*** 8784*** 7370*** 7259*** 9052*** 9956*** 42423*** 8484***
Total reported earnings ($) 8646*** 8928*** 7467*** 7478*** 9064*** 9996*** 42935*** 8587***
Incidence of employment (pp) 7.28*** 7.28*** 3.34** 4.16*** 3.84** 3.02* N/A 5.04***
EI benefits ($) -566*** 138 35 20 -19 -173 -27 -6
SA benefits ($) -135*** -138*** -121*** -134*** -140*** -115*** -647*** -130***
Dependence on income support (pp) -5.46*** -2.59*** -1.63** -1.72*** -1.59** -3.06*** N/A -2.22***

Table 10b: Female (n=1,540) Incremental impact for Career Focus participants using EAS as comparison group
Indicators Program start year
(0)		 1 yr post-program
(1)		 2 yr post-program
(2)		 3 yr post-program
(3)		 4 yr post-program
(4)		 5 yr post-program
(5)		 Cumulative post-program
(1-5)		 Annual average post-program
Employment Income ($) 5857*** 4656*** 2900*** 2972*** 3192*** 3056*** 16777*** 3355***
Total reported earnings ($) 5941*** 4678*** 2938*** 2992*** 2988*** 2840*** 16438*** 3287***
Incidence of employment (pp) 7.21*** 6.26*** 2.89** 1.73 2.39 2.71* N/A 3.29***
EI benefits ($) -310*** 91 116 128 -30 83 445 89
SA benefits ($) -141** -254*** -134* -130** -213*** -164*** -926** -186**
Dependence on income support (pp) -4.1*** -2.04* -1.48 -1.33 -2.08** -2.71** N/A -1.82**

Table 10c: Youth between 20 to 24 (n=1,193) Incremental impact for Career Focus participants using EAS as comparison group
Indicators Program start year
(0)		 1 yr post-program
(1)		 2 yr post-program
(2)		 3 yr post-program
(3)		 4 yr post-program
(4)		 5 yr post-program
(5)		 Cumulative post-program
(1-5)		 Annual average post-program
Employment Income ($) 6769*** 6337*** 4709*** 4611*** 6443*** 7295*** 29396*** 5879***
Total reported earnings ($) 6895*** 6543*** 4860*** 4605*** 6216*** 7066*** 29292*** 5858***
Incidence of employment (pp) 7.39*** 4.26*** 0.07 2.37* 2.46* 1.18 N/A 2.25*
EI benefits ($) -231*** 105 -50 -37 -112 -207* -402 -81
SA benefits ($) -127*** -188** -141*** -86** -142*** -107** -732*** -147***
Dependence on income support (pp) -4.54*** -1.12 -2.15*** -1.55** -2.04*** -3.17** N/A -2.24***

Table 10d: Youth between 25 to 30 (n=1,479) Incremental impact for Career Focus participants using EAS as comparison group
Indicators Program start year
(0)		 1 yr post-program
(1)		 2 yr post-program
(2)		 3 yr post-program
(3)		 4 yr post-program
(4)		 5 yr post-program
(5)		 Cumulative post-program
(1-5)		 Annual average post-program
Employment Income ($) 7253*** 6034*** 3805*** 3790*** 4033*** 4315*** 21978*** 4395***
Total reported earnings ($) 7324*** 5953*** 3794*** 4112*** 4168*** 4293*** 22322*** 4464***
Incidence of employment (pp) 8.04*** 6.42*** 4.56*** 3.13** 4.08*** 4.09*** N/A 4.86***
EI benefits ($) -651*** 120 238* 21 -92 167 482 96
SA benefits ($) -129*** -216*** -139*** -108** -162*** -137*** -812*** -163***
Dependence on income support (pp) -6.57*** -2.44*** -0.71 -1.06 -1.77** -1.4* N/A -1.68**

Page details

Date modified: