Language selection

Government of Canada / Gouvernement du Canada

Search

Technical report: Estimating the incremental impact of Canada Summer Jobs

Technical report prepared by the Evaluation Directorate – Strategic and Service Policy Branch
By: Jamil Sayeed, Anna Yu, Himavanth Vempati, Erik Sagmoen, Andy Handouyahia, and Essolaba Aouli

On this page

Alternate formats

Technical Report: Estimating the incremental impact of Canada Summer Jobs [PDF - 853 KB]

Large print, braille, MP3 (audio), e-text and DAISY formats are available on demand by ordering online or calling 1 800 O-Canada (1-800-622-6232). If you use a teletypewriter (TTY), call 1-800-926-9105.

Executive summary

This technical report presents the methodology and key findings from the impact evaluation of the Canada Summer Jobs (CSJ) component of the Youth Employment Skills Strategy (YESS) published in 2025 on Canada.ca. YESS is a horizontal initiative led by Employment and Social Development Canada (ESDC) and is delivered in collaboration with 11 other federal departments, agencies, and Crown corporations. While YESS is comprised of 2 streams – CSJ and Youth Employment Skills Strategy Program (YESSP) – this report focuses only on the CSJ component of YESS. The CSJ is delivered exclusively by ESDC.

CSJ is 1 of the 2 streams of the Strategy. It provides wage subsidies to employers in not-for-profit organizations, public sector entities, and private sector organizations with 50 or fewer full-time employees, aimed to create quality summer work experiences for young people aged 15 to 30 years.

This is the first time an incremental impact analysis of the CSJ program has been conducted using non-experimental methods. This study includes participants who started a CSJ program during the period of April 2019 to December 2020. It addresses the following questions:

  1. to what extent is the CSJ reaching the targeted population of youth?
  2. what are the short-term impacts for the CSJ participants?
  3. to what extent have the supports and services offered under the Strategy contributed to employment amongst youth?

Methodology

The approach used for the CSJ impact evaluation focused on examining participant profiles (socio-economic characteristics), comparing labour market outcomes, and estimating incremental impacts. The analysis uses rich longitudinal administrative data from Labour Market Program Data Platform (LMPDP) for youth who participated in the CSJ program between April 2019 to December 2020. The LMPDP is an integrated relational database that contains information from multiple data sources such as labour market programs, Employment Insurance records, and income tax data from the Canada Revenue Agency.

The comparison group contains individuals who meet the same eligibility criteria and share similar socio-demographic characteristics as CSJ participants but had received only minor interventions via Employment Assistance Services as part of the Labour Market Development Agreements (LMDA).

The main estimation method used to assess the incremental impact of participating in the CSJ program is propensity score kernel matching. For the purposes of constructing the comparison groups, participants are matched to non-participants using a broad set of characteristics, such as socio-demographic factors, province of residence, prior qualifications, and labour market history.

The incremental impacts are conducted for the overall CSJ population, by subgroups and intersectionality factors based on gender. This study measures the program impact on 5 labour market outcome indicators over a 2-year period following program participation. The 5 indicators are employment earnings, incidence of employment, Employment Insurance (EI) benefit received, social assistance received, and dependence on government income supports Footnote 1.

Key findings

On average, incremental impacts demonstrate that, in the short term, CSJ participation strengthens the labour market attachment of youth participants relative to similar non-participants through increases in incidence of employment and employment earnings (second year post-program).

In addition, CSJ participants decrease their dependence on government income supports (that is the combination of social assistance and Employment Insurance benefits). An examination of intersectionality among gender and an additional characteristic (such as age and disability) found that in most cases, participants increased their incidence of employment and earnings relative to the comparison group, while decreasing their dependence on government income support.

Overall, CSJ is a good policy tool to assist youth in integrating into the labour market, as it makes some contribution to facilitating the transition from school to employment.

These results were validated using alternative matching techniques such as Inverse Probability Weighting and Nearest Neighbour methods. In addition, propensity score kernel matching combined with the difference-in-differences (DID) and an alternative estimation strategy was used as part of the robustness check. This triangulation exercise confirmed that the results were not sensitive to the choice of estimator.

1. Introduction

This technical report presents the methodology and findings related to the incremental impact analysis conducted as part of the Horizontal Evaluation of the Youth Employment Skills Strategy in compliance with the Financial Administration Act and the Treasury Board’s Policy on Results. This report supports the evaluation assessment of whether and to what extent the program is meeting its objectives. The report also outlines challenges related to the units of analysis used, the comparison group selection, the proposed econometric methods, and key findings. The analysis focuses on ESDC funded CSJ participants who started an intervention between April 2019 and December 2020 and assesses their labour market outcomes following their participation up to calendar year 2022.

The report is organized as follows:

The methodologies and approaches used in this study are described in detail in order to allow replication of the studies in the future.

1.1 Program overview

Canada Summer Jobs is a distinct component of YESS which targets youth aged 15 to 30 for summer work placements with mentorship. CSJ consists of 1 intervention that provides wage subsidies to employers of small businesses, not-for-profit organizations, and the public sector. This supports skills development, addresses national and local priorities, and improves labour market access for youth, including those who face unique barriers. It represents close to 45% of the total YESS funding.

The eligibility criteria and conditionsFootnote 2 provide valuable insights into the selection process of program participants.

Main eligibility criteria:

Other eligibility conditions:

In 2020, public health restrictions and associated shutdowns during the COVID-19 pandemic negatively impacted youth employment. Given this, the Government of Canada’s response provided for temporary flexibilities under Canada Summer Jobs and for additional job placements under both components of YESS to support youth employment. These temporary flexibilities were introduced in fiscal years 2020 to 2021 and 2021 to 2022 to support employers through a period of economic recovery.

The temporary flexibilities were provided to employers to offer participants part time employment (for example, fewer than 30 hours per week) during summer. Previously, all Canada Summer Jobs-funded employment had to be full time (a minimum of 30 hours per week for at least 6 weeks). Employers could also offer part-time employment to youth who wanted to work throughout the academic year.

Following fiscal year 2021 to 2022, the temporary flexibilities offered within Canada Summer Jobs were ended due to low uptake of these flexibilities.

1.2 Previous evaluation findings

The 2020 evaluation of the Youth Employment Strategy included the Summer Work Experience (SWE), which is similar to the CSJ in the modernized YESS. This evaluation did not conduct an incremental impact analysis as it focused on descriptive statistics and outcome analysis (ESDC, 2020). Among main findings, the evaluation found that the students were paid above the minimum wage in their province or territory, and most students felt that it helped pay for their school expenses. The evaluations also found that work placement duration decreased slightly on average from 2016 to 2018. The evaluation recommended for the Department to explore how to better leverage administrative and participant data to inform policy and design of the program.

2. Data and methodology

2.1 Scope of the study

The main objective of the current study is to measure the incremental impacts from participating in CSJ. The program effect will be determined relative to what would have happened had these youth only received a low intensity intervention such as Employment Assistance Services (EAS) Footnote 3. The study will also include a statistical analysis of participants’ profiles and outcomes.

The reference period of this study covers April 2019 to December 2020. The choice of reference period is guided by the fact that the modernized Strategy is still in its early years of implementation and the desire to measure programs’ short-term impacts over 2 to 3 years following the start of participation (up to calendar year 2022, which is the most recent year for which employment earnings information is available at the time of this study).

This study will examine participant profiles, labour market experiences (outcomes), and the incremental impacts for participants of CSJ by addressing the following questions:

  1. to what extent is the Strategy reaching the targeted population of youth (for example, youth facing barriers who are furthest from the labour market)?
  2. what are the early outcomes and incremental impacts (6 to 15 months after participation) of the interventions on the YESSP participants under the modernized Strategy who began participation between April and December 2020?
  3. what are the early outcomes and incremental impacts (1 to 2 years after participation) for the CSJ participants who began participation between June 2019 and December 2020?

2.2 Data and unit of analysis

The incremental impact analysis and the statistical analysis of the profile and outcomes are carried out using administrative data. The databases (please see table 1 below) used as part of this study are: Common System for Grants and Contributions (CSGC), Employment Insurance (EI) data (EI Part I and II data), and personal income tax data from the Canada Revenue Agency (CRA).

It should be noted that the CSGC database only contains participants who participated in ESDC-YESS funded projects. All of the previously mentioned databases are linked via a participant’s masked Social Insurance Number (SIN) so that the same individual can be tracked across time and different administrative datasets.

Table 1: Databases available for the study
Source Data Type of information provided by these data Period covered
EI Databank (100% of Recipients) EI part I data Information on EI claims 1996 to 2022
EI Databank (100% of Recipients) EI part II data (EBSM) –EAS only Employment Assistance Services (EAS) interventions and participants characteristics for non-insured clients. 1996 to 2022
EI Databank (100% of Recipients) Record of Employment (ROE) data Employment information on 100% of participants 1996 to 2022
Common System for Grants and Contributions (CSGC) - 100% of CSJ participants Canada Summer Jobs (CSJ) This file contains information on job experiences provided by funding recipients. There is 1 record for each work experience by date created; therefore, an individual may have taken part in more than 1 work experience. 2018 to 2023
Canada Revenue Agency T1 Annual income of tax filers including annual amount of SA received 1990 to 2021
Canada Revenue Agency T4 Annual data on wage and salaries of employees as reported by employers 1999 to 2022

The unit of analysis is the youth who participated in the ESDC funded CSJ program during the reference period (April 2019 to December 2020).

2.3 Outcome indicators

Outcome trends and incremental impacts were measured for the following individual-level labour market indicators:

Employment Insurance benefits + Social assistance benefits Employment Insurance benefits ⁢+ Social assistance benefits + Total Earnings*

*where “total earnings” includes wages and salaries as well as any self-employment income.

2.4 Estimating the incremental impact of CSJ

Identification strategy

The methodology of this report is built on the same framework as the evaluation of the Labour Market Development Agreements Footnote 4.

The main objective of this study is to estimate the average impact of participating in CSJ by employing a quasi-experimental method. In order to identify the CSJ program impact, the outcomes of individuals who participated (treatment group) in CSJ are compared to those of similar individuals who are eligible to participate in CSJ but did not (comparison group). There is a challenge in choosing an appropriate comparison group for CSJ participants due to limitations in data availability for identifying eligible individuals who did not participate in the CSJ program.

This study does not attempt to estimate the impact of participation in CSJ versus non-participation. Instead, similar to Anderson et al. (2022), this study compares the estimated outcome from participating in CSJ with participating in a very limited employment support intervention. Through the Labour Market Development Agreements, ESDC provides funding to provinces and territories to deliver a wide range of interventions, including lighter touch interventions such as Employment Assistance Services (EAS).

From lessons learned from similar program evaluations (LMDAs, the previous Youth Employment Strategy, Opportunities Fund for Persons with Disabilities), EAS participants under the LMDAs provides for a suitable comparison group for CSJ. The advantage of using EAS as a comparison group is that it can avoid the temporal alignment issues, which arise when comparing outcomes of program participants to those who did not participate but from another point in time. In addition, similar to CSJ, EAS intervention data contains a rich set of socio-demographic variables and historical labour market outcome variables enabling the use of robust quasi-experimental methods (for instance, propensity score matching).

This study focuses on estimating the average impact of the program on participants. A common approach to estimating program impacts in quasi-experimental designs is to rely on the selection on observables assumption, also known as conditional independence assumption (CIA).

This identification strategy assumes that the issue of non-random selection into the program can be resolved by conditioning on a set of observed variables (for instance, socio-demographics such as age, gender, disability, education and labour market characteristics such as occupational group, industry codes). Nevertheless, some unobserved factors such as motivation and ability can persistently influence labour market outcomes, which contributes to the issue of selection bias. To address this issue, the model incorporates pre-program participation labour market indicators (for instance, prior use of EI and social assistance benefits, employment earnings, as well as incidence of employment in the 2 years before participation) in the impact analysis to control for unobserved factors correlated with program participation. By using a matching approach to balance both observed and unobserved characteristics, any differences in outcomes between CSJ participant and the comparison group can be attributed to program participation (CSJ), rather than the results of pre-existing differences between the 2 groups.

If there are very small differences in pre-treatment outcomes (such as employment income) between treated and control groups after matching, then propensity score kernel matching methodology (which relies on the CIA) can be used to estimate incremental impact. The summary of the standardized test (see Table 24) reveals that the overall mean bias between the treated and control groups after matching is very small. This indicates that the use of the pre-treatment outcome in the matching procedure substantially mitigates the influence of any potential unobservable confounders. Consequently, the common trend assumption has minimal influence on this analysis.

There may be some instances where some unobserved characteristics may not be fully captured by conditioning on the available pre-participation labour market outcomes. An alternative identification strategy known as difference-in-differences matching which is based on conditional Bias Stability Assumption (BSA) can address this issue. It assumes that the conditioning on observed variables is sufficient to resolve the selection problem if the time invariant unobserved factors can be removed by differencing. This identification strategy allows for the comparison of outcomes between a treatment group (participants in the CSJ program) and a comparison group (youth participants who received an EAS intervention under the LMDA), both before and after the intervention. By analyzing differences in outcomes over time between the 2 groups, we can estimate the incremental impact of the CSJ program, accounting for pre-existing differences between participants and non-participants. This study uses this estimation method of matching combined with difference-in-differences as part of the robustness check.

The following sections discuss the approaches for determining the counterfactual comparison group and implementing the propensity score matching combined with difference-in-differences methodology.

Selection of the comparison group

There is a challenge in building the comparison group for CSJ participants since the available administrative data does not allow to identify youth that are eligible to participate in CSJ but did not participate in any labour market programs. Following the advice of independent experts from previous YES and LMDA evaluations, the comparison group was created using youth who participated in limited EAS intervention during the reference periods.

EAS are among employment benefits and support measures delivered by provinces and territories under the LMDA Footnote 5 . EAS provides low intensity services that support individuals as they prepare to enter/re-enter the labour force. These services range from job search assistance for job-ready participants to the development of in-depth return-to-work action plans for clients facing multiple employment barriers.

An advantage of using EAS-only participants to construct the comparison group is that it solves the temporal alignment problem as both the CSJ participants and EAS participants are starting their interventions at a similar reference period. In addition, the number of participants in of LMDA-EAS is large, so that it makes it possible to obtain close matches to program participants in terms of their observable characteristics. As well, EAS interventions are low-intensity and of short duration which is well suited in this case. Thus, using EAS for comparison is an intuitive approach to reducing potential bias. However, since EAS is an external comparison group, differences may still remain between participants and comparison group after matching, due to unobserved heterogeneity.

To summarize, the comparison group for CSJ is composed of youth who shared the same eligibility criteria and socio demographic characteristics as CSJ participants but had received lighter-touch EAS interventions via LMDA (they also never participated in other Employment Benefit such as a skills training or wage subsidy) and were between 15 and 30 years of age.

2.5 Implementation of the matching estimator and difference-in-differences method

We used non-experimental propensity score based weighting approach to measure the program impacts. This approach aims to ensure that the participants and comparison group are balanced in terms of background factors (that is, sociodemographic and labour market history variables) prior to estimating the program impacts. We applied the kernel matching technique, which uses the entire control group and re-weights control group members each time they are compared with a new participant, based on propensity score differentials. As part of the robustness check, these methods were combined with difference-in-differences (DID) estimation method. The DID method allows for the participants and comparison groups to differ on time-invariant unobserved characteristics, by assuming common time trends in the pre- and post-participation period in these characteristics.

A brief description of the propensity score model and difference-in-differences method are provided below.

Random sampling

The total number of CSJ participants for the reference years 2019 and 2020 is 115,798. The comparison group used for the incremental impact analysis in this report contains 87,567 observations. A random sample of 45% of the total CSJ participants was used to conduct the incremental impact analysis. This ensures that when using kernel matching, the analysis remains representative and effective, considering that kernel matching is computationally intensive, especially with a large sample size. This also allows for a sufficiently high comparison group relative to the treatment group.

Table 2: Number of participants in CSJ and comparison group, reference period 2019 and 2020
  Participants Random sample of participants Comparison group
CSJ 115,798 52,172 87,567

Propensity score model

This report uses the logistic regression model to estimate propensity scores. The propensity score is the conditional probability of participating in the program given the pre-participation variables. Propensity score matching uses the distance between estimated propensity scores to find similar individuals. The participants are pooled with potential comparison cases, and logistic models are estimated to predict the likelihood of participating in the CSJ program based on the background characteristics of the participants and the comparison group members. Model estimation relies on a rich dataset informing the labour market experiences and socio-demographic characteristics of participants and comparison cases. These characteristics include age, gender, disability, education, as well as industry codes. The data also informs labour market experience prior to program participation (including EI benefits, the incidence of employment, and employment earnings) for more than 2 years.

The choice of variables for estimating propensity scores is based on past evaluations (previous YES, LMDA) and the economic theory of active labour market program (see for example: Heckman, Lalonde, and Smith, 1999; Heckman and Smith, 1999, or Lechner and Wunch, 2013).

Table 3: Example of variables used in the model
Variables Source Description
Age Program data Age at program start (3 age groups: 15 to 19, 20 to 24 and 25 to 30)
Gender Program data Binary variable for biological sex (male and female) at program start
Indigenous status Program data Binary variable for indigenous status at program start
Disability Program data Binary variable for disability status at program start
Immigrant T1 Binary variable for immigrant status at program start
Visible Minority Program data Binary variable for visible minority at program start
Education Program data Education level at program start (categorized in the following 2 main groups: secondary, post-secondary)
Province of residence Program data/T1 Province of residence at program start
Industry codes T4s Binary variables based on North American Industry Classification System (NAICS)
Past earnings up to 2 years T1/T4s Total earnings before program start (categorical variable)
Past EI benefit usage up to 2 years EI Part 1 Total EI benefit received before program start (categorical variable)
Past social assistance up to 2 years T1 Total social assistance receipts before program start (categorical variable)

Difference-in-differences (DID)

DID is 1 of the most common methods to assess the causal effect on key outcome indicators of participating in an intervention or program. The DID method is used when outcome variables for participants and the comparison group are observed for 2 or more time periods (that is, before and after an intervention). The DID method requires that, in the absence of the intervention, the expected difference in outcome indicators between the participants and the comparison group is constant over time. This refers to a common trend assumption or constant bias assumption (Lechner, 2010). It is conventionally measured by comparing the observable trends in the pre-intervention period. In this analysis, we restricted the base pre-program period used for the DID method to 2 years prior to participation. This adjustment ensures that we capture relevant information about youth who may not have yet entered the labour market, considering their younger age.

This report applies the 3 following identifying assumptions in the estimation strategy.

Conditional independence assumption

The conditional independence assumption requires that the common variables that affect participation assignment and intervention-specific outcomes be observable. An advantage of this analysis is that the available administrative data contains rich information about sociodemographic and labour market characteristics for both participants and the comparison group. This allows for the inclusion of the most relevant variables influencing the decision to participate in the interventions and the labour market outcomes in the propensity score model.

Common support (overlap) assumption

This assumption ensures that persons with the same covariate values have a positive probability of being both participants and non-participants (Heckman, LaLonde, and Smith, 1999). In other words, each participant has 1 or more ‘counterparts’ in the comparison group with the same covariate profile, such that appropriate counterfactuals can be constructed. The common support assumption can be verified empirically. The most straightforward way to do this is through a visual inspection of the density distribution of the propensity score in both groups.

Conditional bias stability assumption

The motivation for the conditional bias stability assumption comes from the concern that some relatively stable unobserved characteristics, such as ability, motivation, and/or attractiveness, may persistently affect labour market outcomes, but not fully capture conditioning on the available pre-program data. This assumption is necessary to conduct the DID method.

2.6 Choosing matching algorithm

This report uses the kernel matching algorithm to match participants and the comparison group with respect to their propensity scores. Kernel matching is a non-parametric technique that uses weighted averages of the outcomes of all individuals in the comparison group to construct the counterfactual. One advantage of this approach is that it reduces the variance of the estimated effects.

We used Nearest Neighbour (NN) matching and Inverse Probability Weighting (IPW) as alternative methods to validate the results. In NN matching, the individual from the comparison group is chosen as a matching partner for a treated individual that is closest in terms of propensity score. The nearest-neighbor method assigns a weight of 1 to the nearest nonparticipant and 0 to others. Several variants of NN matching are proposed, such as NN matching ‘with replacement' and ‘without replacement'. In the former case, an untreated individual can be used more than once as a match, whereas in the latter case it is considered only once.

2.7 Subgroup and intersectionality factors analysis

In addition to the overall impact, we performed an analysis of the different subgroups and intersectionality factors Footnote 6 . However, due to sample size of the participants and comparison group, not all of the subgroups and factors were analyzed.

We were able to conduct the subgroup analysis for males, females, age group 20 to 24, age group 25 to 30, persons with disabilities, high school incomplete, rural, urban and participants with an intention of not returning to school. For the intersectionality factors, we were able to analyze female age group 25 to 30, males with disabilities, females with disabilities, male age group 20 to 24 and male age group 25 to 30.

3. Results

3.1 Profile of CSJ participants

In this section, we discuss several key sociodemographic characteristics of the CSJ participants. For further details about the profile of participants, please see Table 4.

Among the general profile of the CSJ participants, the report focuses on participants’ gender, age, and education level. Almost 2 thirds (62.1%) of the CSJ participants were female. Approximately 87% of participants were 24 years of age and younger (with participants in the 15 to 19-year-old group representing 43.7% of the population and participants in the 20 to 24-year-old group representing 43.1%). 2 thirds of participants (66.4%) reported having attended post-secondary studies, with 6.5% having completed a college diploma and 18% having completed a university degree.

Participants self-reported being members of the following groups:

Almost 2 thirds (64%) of participants reported that they intended to return to school full-time following their CSJ participation.

Approximately 12% of participants reported that CSJ was their first work experience. 1 year prior to CSJ participation, 19% of participants earned $0,44% of the participants had an income between $1-$9,999 and around a quarter (24%) of participants had an income between $10,000-$19,999.

Table 4: Socio-Demographic and Labour Market Characteristics of CSJ Participants by number of counts started in April 1, 2019 to December 31, 2020 (n=115,798) *
Categories Proportion
Male 36%
Female 62%
Other gender 1%
Age 15 to 19 44%
Age 20 to 24 43%
Age 25 to 30 13%
Elementary <1%
Secondary incomplete 12%
Secondary complete 21%
Post secondary incomplete 12%
Post secondary complete 7%
University incomplete 30%
University complete 18%
Person with disability 3%
Racialized youth 20%
Recent immigrant 2%
Youth living in rural or remote area 20%
Inuit <1%
Metis 1%
Non-Status 3%
Registered off-reserve 1%
Registered on-reserve 1%

3.2 Overall incremental impact results of CSJ

Overall, participants strengthen their labour market attachment following their CSJ work placement, through incremental gains in employment earnings and incidence of employment relative to similar non-participants. They also experienced lower dependence on government income supports due to lower receipt of EI and SA benefits.

CSJ program provides wage subsidies to employers to provide summer work experiences for young people aged 15 to 30 years. Therefore, this wage subsidy program is unlike other training programs, where earnings usually decrease during the participation period, since participants spend most of their time in training rather than working. As shown in Table 5 below, CSJ participants experience a decrease of $91 on average (however, statistically insignificant) in their employment earnings during the program participation year compared to similar non-participants. Following program participation, there is a decrease in employment earnings (-$536) for CSJ participants during the first post-program year. One plausible reason for this lower level of earnings is that many participants (approximately 66%) returned to school following their CSJ placements. However, employment earnings increased in the second ($2,321) post-program year. This is an indication that many of the participants may have completed their schooling and entered the job market.

Similar to the findings for employment earnings, the incidence of employment of CSJ participants increased (9.3pp) during the program participation period. During each of the 2 post-program years, participants experienced increases in incidence of employment relative to the comparison group (3.75 pp and 10.87 pp respectively).

Relative to the comparison group, CSJ participants decreased their dependence on income supports throughout the 2-year post-program period. This decrease is due to lower receipt of both EI and SA benefits relative to the comparison group.

Table 5: Overall Incremental Impacts using Matching Estimator for CSJ participants across Canada, for 2019 and 2020 (52,172)
Indicators In-program 1st year post program 2nd year post program
Employment earnings ($) -91 -536*** 2321***
Incidence of Employment (percentage points) 9.3*** 3.75*** 10.87***
EI benefits ($) 96*** -831*** -1239***
SA benefits ($) -319*** -283*** -266***
Dependence on income support (percentage points) -3.27*** -6.98*** -7.93***

3.3 Subgroup analysis

This report conducts incremental impact analysis for the subgroups for which sufficiently large comparison groups are available. Subgroups include female, male, age between 19-24, age between 25-30, persons with disabilities, high school incomplete, participants living in rural or urban locations, participants who intended not to return to school and participants who intended to return to school.

Female participants

Female CSJ participants strengthened their labour market attachment following their CSJ participation through incremental gains in employment earnings and in their incidence of employment relative to similar non-participants. They also decreased their dependence on income supports due to their lower use of EI and SA benefits.

As shown in Table 6 below, the decrease in employment earnings of female participants for the first post-program year is not statistically significant. However, there is a statistically significant increase in employment earnings in the second year following program participation ($3,168).

Similar to the findings for employment earnings, female participants experienced an increase in each of the 2 post-program years (3.83 pp, and 10.34 pp respectively).

Female CSJ participants decreased their reliance on government income supports during the post-program participation periods (-8 pp in the first year and -9.38 pp in the second-year post-program). The observed decline is attributed to decreases in both SA and EI benefits usage relative to the comparison group.

Table 6: Incremental Impact using Matching Estimator for female CSJ participants across Canada, for 2019 and 2020 (n= 32,281)
Indicators In-program 1st year post program 2nd year post program
Employment earnings ($) 142 -91 3168***
Incidence of Employment (percentage points) 8.58*** 3.83*** 10.34***
EI benefits ($) 22 -1015*** -1446***
SA benefits ($) -332*** -290*** -289***
Dependence on income support (percentage points) -3.77*** -8*** -9.38***

Male participants

Male CSJ participants had short-term mixed results during the CSJ participation year and 1 year after participation. However, they strengthened their labour market attachment, relative to the comparison group, in the second year post-CSJ participation.

As shown in Table 7 below, male CSJ participants experienced a decrease by $371 in their employment earnings during the program participation year compared to similar non-participants. This decrease in employment earnings continues into the first post-program year (-$1,840). However, employment earnings increase in the second post-program year ($761).

Unlike the incremental impacts for employment earnings, male participants increased their incidence of employment during the participation year (10.72 pp), and in each of the 2 post-program years (3.95 pp, and 11.25 pp respectively).

CSJ participants decreased their reliance on government income supports both during the program year (-2.64 pp) and in the first 2 post-program years (-5.12 pp and -5.36 pp respectively). The observed decline is attributed to decreases in both EI and SA benefits usage during post-program years relative to the comparison group.

Table 7: Incremental Impact using Matching Estimator for male CSJ participants across Canada, for 2019 and 2020 (n=18,712)
Indicators In-program 1st year post program 2nd year post program
Employment earnings ($) -371** -1840*** 761*
Incidence of Employment (percentage points) 10.72*** 3.95*** 11.25***
EI benefits ($) 161*** -498*** -794***
SA benefits ($) -273*** -274*** -242***
Dependence on income support (percentage points) -2.64*** -5.12*** -5.36***

Participants 20 to 24 years of age

CSJ participants aged between 20-24 had incremental gains in employment earnings and in incidence of employment during the second year of their post-participation period relative to similar non-participants. They also had a lower dependence on income support due to receiving fewer EI and SA benefits.

As shown in Table 8 below, CSJ participants aged between 20-24 experienced an increase and followed by a decrease in their employment earnings during the program participation year and the first year of program participation respectively compared to similar non-participants. However, these changes are not statistically significant. Employment earnings increased in the second post-program year ($3,752).

Participants had an increase (6.13 pp) in incidence of employment during the program year and then a slight increase (1.46 pp) in the first post-program year. Incidence of employment continues to increase during the second post-program year (8.63 pp).

Participants decreased their dependence on income supports relative to the comparison group throughout the post-program period. This observed decline can be attributed to the decrease in both EI and SA benefits usage relative to the comparison group.

Table 8: Incremental Impact using Matching Estimator for CSJ participants aged 20 to 24 across Canada, for 2019 and 2020 (n=22,643)
Indicators In-program 1st year post program 2nd year post program
Employment earnings ($) 40 -456 3752***
Incidence of Employment (percentage points) 6.13*** 1.46** 8.63***
EI benefits ($) 172*** -1006*** -1345***
SA benefits ($) -303*** -244*** -222***
Dependence on income support (percentage points) -2.56*** -7.16*** -7.85***

Participants 25 to 30 years of age

CSJ participants increased their labour market attachment with incremental increases in employment earnings and incidence of employment, as well as a decrease in dependence on government income supports such as EI and SA benefits, relative to the comparison group.

As shown in Table 9 below, during the program period CSJ participants aged between 20-24 experienced an increase by $1387 in their employment earnings during the program participation year compared to similar non-participants. This increase ($1337, and $5397 respectively) in employment earnings continues throughout the post participation period.

Participants aged 25 to 30 years experienced an increase in their incidence of employment during the participation year (7.46 pp), and in each of the 2 post-program years (4.25 pp, and 11.05 pp respectively).

In addition, participants decreased their dependence on income supports during the program and post program periods (-$5.44, and -$4.78 and -$5.59 respectively). This observed decline can be attributed to the decrease in both EI and SA benefits usage relative to the comparison group.

Table 9: Incremental Impact using Matching Estimator for CSJ participants aged 25 to 30 across Canada, for 2019 and 2020 (n=6,756)
Indicators In-program 1st year post program 2nd year post program
Employment earnings ($) 1387*** 1337*** 5397***
Incidence of Employment (percentage points) 7.46*** 4.25*** 11.05***
EI benefits ($) -122 -351*** -817***
SA benefits ($) -440*** -334*** -236***
Dependence on income support (percentage points) -5.44*** -4.78*** -5.59***

Participants with disabilities

CSJ female participants with disabilities strengthened their labour market attachment following participation. Relative to non-participants, CSJ female participants experienced higher employment earnings during the post-program period and increased their incidence of employment (second year post-program only). Participants decreased their dependence on government income supports in the post-program due to receiving fewer Employment Insurance and social assistance benefits.

Table 10: Incremental Impact using Matching Estimator for female CSJ participants with disabilities across Canada, for 2019 and 2020 (n=925)
Indicators In-program 1st year post program 2nd year post program
Employment earnings ($) 1911*** 2245*** 3382***
Incidence of Employment (percentage points) 9.47*** -2.61 11.34***
EI benefits ($) 137 -1545*** -1663***
SA benefits ($) -785*** -740*** -590***
Dependence on income support (percentage points) -8.57*** -13.75*** -11.07***

Relative to the comparison group, male participants with disabilities experienced statistically significant increases in incidence of employment in the second-year post-program participation. Male participants also decreased their dependence on government income supports mainly due to their lower use of Employment Insurance.

Table 11: Incremental Impact using Matching Estimator for male CSJ participants with disabilities across Canada, for 2019 and 2020 (n=544)
Indicators In-program 1st year post program 2nd year post program
Employment earnings ($) 680 -900 582
Incidence of Employment (percentage points) 15.85*** 1.37 9.5***
EI benefits ($) 385** -235 -1799***
SA benefits ($) -412 -60 -519
Dependence on income support (percentage points) -4.24* -4.26 -9.97**

High school incomplete

Following short-term mixed results during the first post-CSJ participation year, participants increased their subsequent labour market attachment with increases in employment earnings and incidence of employment and reduced dependence on government income supports relative to the comparison group.

As shown in Table 12 below, despite an initial decrease in employment earnings in the first post-program year (-$591), participants who did not complete high school experienced an increase in their second post-program year ($1126).

Participants also experienced increases in their incidence of employment during the participation year (18.09 pp), and in each of the 2 post-program years (7.05 pp, and 17.63 pp respectively).

In addition, participants decreased their dependence on income supports during in-program and post-program periods (-$5.98, -$11.91 and -$10.5 respectively) due to the decline in the usage of EI and SA benefits.

Table 12: Incremental Impact using Matching Estimator for CSJ participants who did not finish high school across Canada, for 2019 and 2020 (n=6,097)
Indicators In-program 1st year post program 2nd year post program
Employment earnings ($) 692*** -591** 1126***
Incidence of Employment (percentage points) 18.09*** 7.05*** 17.63***
EI benefits ($) -9 -1242*** -1193***
SA benefits ($) -504*** -441*** -419***
Dependence on income support (percentage points) -5.98*** -11.91*** -10.5***

Rural participants

Participants experienced negative impact on employment earnings during program participation and first year post-program participation. However, there was an increase in their employment earnings in the second post-program year. During the program year and post-program years, they increased their incidence of employment, and decreased their dependence on government incomes supports, due to receiving fewer EI and SA benefits.

As shown in Table 13 below, despite an initial decrease in employment earnings in the first post-program year (-$2832), participants who lived in rural areas experienced an increase in their second post-program year ($692).

Participants experienced an increase in their incidence of employment during the participation year (5.88pp), and in each of the 2 post-program years (4.8 pp, and 9.18 pp respectively).

In addition, participants decreased their dependence on income supports during the in-program and post-program periods (-$5.74, -$6.4 and -$5.79 respectively) due to the decline in their usage of EI and SA benefits relative to the comparison group.

Table 13: Incremental Impact using Matching Estimator for CSJ participants who did live in rural areas across Canada, for 2019 and 2020 (n=9,107)
Indicators In-program 1st year post program 2nd year post program
Employment earnings ($) -1157*** -2832*** 692
Incidence of Employment (percentage points) 6.96*** 4.8*** 9.18***
EI benefits ($) 49 -518*** -621***
SA benefits ($) -489*** -374*** -354***
Dependence on income support (percentage points) -5.74*** -6.4*** -5.79***

Urban participants

Following short-term mixed results during the first post-CSJ participation year, participants increased their labour market attachment with increases in employment earnings and incidence of employment and reduced dependence on government income supports relative to the comparison group.

As shown in Table 14 below, despite an initial decrease in employment earnings in the first post-program year (-$388), participants who lived in urban areas experienced an increase in their second post-program year ($2743).

Participants experienced an increase in their incidence of employment during the participation year (9.63pp), and in each of the 2 post-program years (3.86 pp, and 10.91 pp respectively).

In addition, participants decreased their dependence on income supports during in program and post program periods (-$2.86, -$7.23 and -$8.28 respectively) due to the decline in the usage of EI and SA benefits relative to the comparison group.

Table 14: Incremental Impact using Matching Estimator for CSJ participants who did live in urban areas across Canada, for 2019 and 2020 (n=43,065)
Indicators In-program 1st year post program 2nd year post program
Employment earnings ($) 41 -388* 2743***
Incidence of Employment (percentage points) 9.63*** 3.86*** 10.91***
EI benefits ($) 96*** -888*** -1313***
SA benefits ($) -290*** -277*** -252***
Dependence on income support (percentage points) -2.86*** -7.23*** -8.28***

Participants who did not return to school

This section focuses on CSJ youth participants who did not return to school full-time following their CSJ participation, which represents about 28% of participants during the 2019 and 2020 reference years.

Overall, these participants are more likely to be female (61%), between the ages of 20 to 24 (44%), and have at least attended post-secondary studies (68.2%) with 21% having completed college and 26% having completed a university degree. Compared to the overall CSJ participant profile, youth who did not intend to return to school were slightly more likely to have attended post-secondary studies (1.8 pp), and more likely to have obtained a college diploma (1.8 pp) and university degree (8 pp).

Overall, incremental impacts show that participants who did not intend to return to school strengthened their labour market attachment following CSJ participation with increases in employment earnings and incidence of employment and decreases in dependence on incomes supports due to lower receipt of EI and SA benefits usage.

As shown in Table 15 (below), during the program period CSJ participants who did not intend to return to school experienced an increase by $1529 in their employment earnings during the program participation year compared to similar non-participants. This increase ($2634, and $4594 respectively) in employment earnings continues throughout the post participation periods.

Participants also experienced an increase in their incidence of employment during the participation year (11.27 pp), and in each of the 2 post-program years (9.35 pp, and 15.97 pp respectively).

In addition, participants decreased their dependence on income supports during in-program and post-program periods (-$3.21, -$7.04 and -$4.37 respectively) due to the decline in the usage of EI and SA benefits relative to the comparison group.

Table 15: Incremental Impact for CSJ participants who did not return to school (n=14,467)
Indicators In-program 1st year post program 2nd year post program
Employment earnings ($) 1529*** 2634*** 4594***
Incidence of Employment (percentage points) 11.27*** 9.35*** 15.97***
EI benefits ($) 136*** -544*** -1321***
SA benefits ($) -371*** -328*** -267***
Dependence on income support (percentage points) -3.21*** -7.04*** -4.37***

Participants who intended to return to school

Overall, the incremental impact findings show that participants who did not intend to return to school strengthened their labour market attachment following CSJ participation with increases in employment earnings and incidence of employment and decreases in dependence on income supports due to receiving fewer EI and SA benefits.

As shown in Table 16 (below), during the program period CSJ participants who intended to return to school experienced a decrease in their employment earnings during the program participation and first year post-participation periods compared to similar non-participants. However, there was an increase in employment earnings during the second post-participation year ($1592).

Participants also experienced increase in their incidence of employment during the participation year (9.77 pp), and in each of the 2 post-program years (3.14 pp, and 9.69 pp respectively).

In addition, participants decreased their dependence on income supports during in-program and post-program periods (-$3.44, -$7.1 and -$8.27 respectively) due to the decline in the usage of EI benefit and SA benefits relative to the comparison group.

Table 16: Incremental Impact for CSJ participants who returned to school (n=33,360)
Indicators In-program 1st year post program 2nd year post program
Employment earnings ($) -640*** -1722*** 1592***
Incidence of Employment (percentage points) 9.77*** 3.14*** 9.69***
EI benefits ($) 34 -962*** -1259***
SA benefits ($) -305*** -279*** -258***
Dependence on income support (percentage points) -3.44*** -7.1*** -8.27***

Differential incremental impact due to return to school

Approximately 28% of CSJ participants did not return to school full-time following their CSJ participation. The incremental impact of CSJ may systematically vary for these participants compared to those who returned to school. To investigate this issue further, we compared the trend of incremental impacts on earnings of these 2 groups during the participation period and post-participation periods. Figure 1 shows that compared to the overall scenario, those who did not return to school have higher incremental impact on earnings and those who returned to school have lower impact. This result is intuitive as those who returned to school delay entering the labour market which impacted their post-participation earnings in the short-term. Therefore, for these youth the CSJ program impact on earnings for second and third years are more relevant than those of the first year.

Figure 1: Incremental impact on employment earnings for CSJ participants who returned to school versus who did not return to school
Figure 1: Incremental impact on employment earnings for CSJ participants who returned to school versus who did not return to school
  • Source: Common System for Grants and Contributions (CSGC), EI databank (EI Part I and II data), and the Canada Revenue Agency (CRA) tax data
Figure 1 – Text version

Chart: The incremental impact of CSJ varies depending on whether or not participants returned to school following program participation. We found that those who did not return to school have higher incremental impact on earnings and those who returned to school have lower impact.

Solid black line for total, solid red line for participants who intended to not return to school, dotted grey line for participants who intended to return to school..

4. Robustness Check

4.1 Alternative matching methods

In the baseline model, Kernel matching combined with DID method is used to estimate the incremental impact of the CSJ program. This approach is applied throughout the report. To assess the sensitivity of the estimates to the choice of matching method, alternative matching methods such as Nearest Neighbour and Inverse Probability Weighting are used. The estimates for all 3 methods are very similar, which demonstrates robust baseline findings across alternative matching methods. Please see annex C for details.

4.2 Alternative identification strategy

The incremental impact analysis in this study relies on the conditional independence assumption, which is sufficient to implement the matching method. When only matching method is used, it is assumed that inclusion of the pre-program labour market outcome variables (for instance, employment earnings, and incidence of employment) can control for unobserved factors such as ability and motivation in the impact analysis. However, if this assumption is violated and some unobserved characteristics continue to influence post-participation labour market outcomes, for in even after conditioning on the available pre-program data, then an alternative identification strategy needs to be used. To address this issue, this study uses matching with DID to estimate the incremental impact. This alternative identification strategy requires an additional assumption known as conditional bias stability. Table 17 below shows the findings of overall incremental impacts using matching estimator with DID. Findings from this alternative methodology suggests that the estimates (especially for the second- and third-year post participation period) are much higher than that of the baseline. This implies that the baseline findings provide a conservative estimate of the incremental impact of the CSJ program.

Table 17: Overall Incremental Impacts using Matching Estimator with DID for CSJ participants across Canada, for 2019 and 2020 (n=52,172)
Indicators In-program 1st year post program 2nd year post program
Employment earnings ($) 51 -497*** 3498***
Incidence of Employment (percentage points) 9.47*** 3.92*** 11.23***
EI benefits ($) 84*** -883*** -1190***
SA benefits ($) -293*** -260*** -216***
Dependence on income support (percentage points) -3.2*** -6.95*** -7.81***

5. Strengths and Limitations

The primary strength of this study lies in its utilization of a rich, relational set of administrative data, which contain information on a large set of socioeconomic and labour characteristics to be accounted for during the analysis. Moreover, this study represents the first attempt to assess the CSJ component of YESS, by comparing the labour market outcomes of participants with those of comparable non-participants. Rigorous econometric techniques were employed to construct an appropriate comparison group closely matched to participants. Furthermore, robust statistical tests and 2 matching estimation methods were utilized to validate the reliability of the results.

It is noted that estimating the incremental impacts for CSJ over 2 post-program years may be the limit of what should be examined given the short length of the intervention (11 weeks on average) and the changes that youth are experiencing in their lives (including returning to school, completion of education credentials, or accessing other training programs). As a result, after 2 post-program years, the participants and non-participants may start to differ for reasons other than the effects of the program.

There are some limitations in this study. Firstly, there may be the possibility of pre-existing but unobserved differences between the participants and the comparison group of non-participants that were not measured in the matching process. These differences could have influenced the outcomes. For example, factors such as ability, low self-confidence, health, and motivation, among others were not directly measured, except to the extent that they were captured in prior income and labour market attachment patterns. In addition, the comparison group was not sufficiently large to conduct incremental impact analysis effectively for certain subgroup and intersectionality.

The reference period of this study is April 2019 to December 2020 which spans 2 taxation years (2019 and 2020). Moreover, for the year 2022, only T4 data is available (no T1 tax information). These led to the following two limitations:

6. Conclusion

This report presents the methodology and key findings from the evaluation of the Canada Summer Jobs stream of the Youth Employment Skills Strategy. This is the first time an incremental impact analysis of the CSJ has been conducted. The key findings from the incremental impacts suggest that CSJ participation strengthens the labour market attachment of youth participants relative to similar non-participants through increases in incidence of employment and employment earnings (second year post-program).

In addition, CSJ participants decrease their dependence on government income supports (that is the combination of social assistance and Employment Insurance benefits). An examination of intersectionality among gender and

additional characteristics (e.g., age, or disability) found that in most cases, participants had positive increases in earnings and employment relative to the comparison group, while decreasing their dependence on government income support.

Overall, CSJ is a good policy tool to assist youth in integrating into the labour market, as it makes some contribution to facilitating the transition from school to employment.

References

Andersson, F., Holzer, H. J., Lane, J. I., Rosenblum, D., & Smith, J. (2022). Does federally-funded job training work? Nonexperimental estimates of WIA training impacts using longitudinal data on workers and firms. Journal of Human Resources.

ESDC (2020), Horizontal Evaluation of the Youth Employment Strategy - Summer Work Experience stream. /content/canadasite/en/employment-social-development/corporate/reports/evaluations/horizontal-summer-work-experience.html

ESDC (2022), Horizontal Evaluation of the Labour Market Development Agreements, Horizontal Evaluation of the Labour Market Development Agreements - Canada.ca

Heckman, J. J., & Smith, J.A. (1999). The Pre-Programme Earnings Dip and the Determinants of Participation in a Social Program. Implications for Simple Programme Evaluation Strategies. The Economic Journal, 109(457), 313-348.

Heckman, J. J., Ichimura, H., Smith, J.A., & Todd, P. (1998). Characterizing Selection Bias Using Experimental Data. Econometrica, 66, 1017-1098.

Heckman, J. J., Lalonde, R.J., and Smith, J.A. (1999). “The Economics and Econometrics of Active Labour Market Programs”, Handbook of Labour Economics, ed. by O. Ashenfelter, and D. Card, vol. 3A, chap. 31, pp. 1865–2097.

Huber, M., Lechner, M., & Wunsch, C. (2013). The performance of estimators based on the propensity score. Journal of Econometrics, 175, 1-21.

Lechner, M. (2010). "The Estimation of Causal Effects by Difference-in-Difference Methods," University of St. Gallen Department of Economics working paper series 2010 2010-28, Department of Economics, University of St. Gallen, revised Oct 2011.

Rosenbaum, P. R. & Rubin, D.B. (1983). The Central Role of the Propensity Score in Observational Studies for Causal Effects. Biometrika, 70(1), 41–50.

Roy, A. D. (1951). Some Thoughts on the Distribution of Earnings. Oxford Economic Papers, 3, 135–145.

Rubin, D. B. (1974). Estimating Causal Effects to Treatments in Randomised and Nonrandomised Studies. Journal of Educational Psychology, 66, 688–701.

Annex A: Intersectionality analysis

We conduct the incremental impact analysis of CSJ participants from the GBA plus lens. The main purpose of the analysis is to assess how the impact of the program vary across different intersectionality factors which are based on gender and other subgroup characteristics (for example: gender and age). This report conducts incremental impact analysis for the intersectionality factors for which sufficiently large comparison groups are available, which are the following:

Male participants aged 20 to 24

As shown in Table 18, male youth aged 20 to 24 experience mixed results during the CSJ participation year and the first year after participation. However, they strengthen their labour market attachment, relative to the comparison group, in the second year post-CSJ participation. In addition, these participants decrease their reliance on income support after program participation due to the decline in EI benefit and SA benefit usage.

Table 18: Incremental Impact using Matching Estimator for CSJ male participants aged 20 to 24 years of age, for 2019 and 2020 (n= 7,125)
Indicators In-program 1st year post program 2nd year post program
Employment earnings ($) -894*** -2395*** 462
Incidence of Employment (percentage points) 7.31*** -0.51 7.36***
EI benefits ($) 226** -721*** -1015***
SA benefits ($) -301*** -252*** -240***
Dependence on income support (percentage points) -2.09*** -5.44*** -5.73***

Male participants aged 25 to 30

Relative to the comparison group, male youth aged 25 to 30 years experienced post-program gains in employment earnings and incidence of employment. They also decreased their dependence on government income supports through decreases in SA benefit and EI benefit usage.

Table 19: Incremental Impact using Matching Estimator for CSJ male participants aged 25 to 30 years of age, for 2019 and 2020 (n=2,375)
Indicators In-program 1st year post program 2nd year post program
Employment earnings ($) 1030** 903 5228***
Incidence of Employment (percentage points) 7.68*** 5.48*** 12.32***
EI benefits ($) 8 -324** -675***
SA benefits ($) -454*** -309*** -170**
Dependence on income support (percentage points) -4.6*** -4.09*** -4.86***

Female participants aged 25 to 30 years

Female participants aged 25 to 30 years had incremental gains in employment earnings and incidence of employment during in-program and post-program periods. They also decreased their dependence on government income supports through decreases in SA benefits and EI benefits usage.

Table 20: Incremental Impact using Matching Estimator for CSJ female participants aged 25 to 30 years of age, for 2019 and 2020 (n=4,381)
Indicators In-program 1st year post program 2nd year post program
Employment earnings ($) 1422*** 1484*** 5490***
Incidence of Employment (percentage points) 6.7*** 4.01*** 10.29***
EI benefits ($) -147 -387** -909***
SA benefits ($) -431*** -387*** -239***
Dependence on income support (percentage points) -5.58*** -6.17*** -6.18***

Annex B: Outcome trend analysis between participant and counterfactual

Table 21: Example of incremental impact calculation using outcome trends – CSJ (n=52,172)
Indicators Outcome 1 year Post program (Matched participant's outcome) Outcome 1 year Post program (Counterfactual outcome) Incremental impact
Employment earnings ($) $ 13,960.94 $ 14,496.73 - $ 536
Incidence of employment (pp) 83.18% 79.43% 3.75
EI benefits ($) $ 1,843.67 $2,674.18 -$ 831
SA benefits ($) $ 96.07 $ 378.93 -$ 283
Dependence on income support (pp) 10.32% 17.30% -6.98
Figure 2: Trend of employment earnings between participant and counterfactual – CSJ
Figure 2: Trend of employment earnings between participant and counterfactual – CSJ
  • Source: Common System for Grants and Contributions (CSGC), EI databank (EI Part I and II data), and the Canada Revenue Agency (CRA) tax data
Figure 2 – Text version

Chart: Employment earnings between participants and counterfactual is similar from 2 years before to program to 1 year after the program. In the second year after the program, CSJ participants have higher average earnings.

Dotted grey line for participants, solid black line for counterfactual.

Figure 3: Trend of incidence of employment between participant and counterfactual – CSJ
Figure 3: Trend of incidence of employment between participant and counterfactual – CSJ
  • Source: Common System for Grants and Contributions (CSGC), EI databank (EI Part I and II data), and the Canada Revenue Agency (CRA) tax data
Figure 3 – Text version

Chart: Incidence of employment between participants and counterfactual is similar in the 2 years before the program. During the program and in the 2 years after the program, CSJ participants have higher incidence of employment.

Dotted grey line for participants, solid black line for counterfactual.

Figure 4: Trend of EI benefit use between participant and counterfactual – CSJ
Figure 4: Trend of EI benefit use between participant and counterfactual – CSJ
  • Source: Common System for Grants and Contributions (CSGC), EI databank (EI Part I and II data), and the Canada Revenue Agency (CRA) tax data
Figure 4 – Text version

Chart: EI benefit use between participants and counterfactual is similar from 2 years before to program to during the program. For both groups, EI benefit use increases one year after the program and declines slightly in the following year but EI benefit use is less among CSJ participants than the counterfactual.

Dotted grey line for participants, solid black line for counterfactual.

Figure 5: Trend of social assistance use between participant and counterfactual – CSJ
Figure 5: Trend of social assistance use between participant and counterfactual – CSJ
  • Source: Common System for Grants and Contributions (CSGC), EI databank (EI Part I and II data), and the Canada Revenue Agency (CRA) tax data
Figure 5 – Text version

Chart: Social assistance use is elevated among the counterfactual compared to participants from 2 years before the program to 2 years after the program. While social assistance use is stable for participant for all years, counterfactual social assistance use rises greatly during the program and levels off in the years post program.

Dotted grey line for participants, solid black line for counterfactual.

Figure 6: Trend of dependence on income support between participant and counterfactual – CSJ
Figure 6: Trend of dependence on income support between participant and counterfactual – CSJ
  • Source: Common System for Grants and Contributions (CSGC), EI databank (EI Part I and II data), and the Canada Revenue Agency (CRA) tax data
Figure 6 – Text version

Chart: Dependence on income support between participants and counterfactual is similar in the first 2 years before the program. During the program and in the 2 years after the program, dependence increases for both groups but has a greater effect on counterfactual.

Dotted grey line for participants, solid black line for counterfactual.

Annex C: Robustness check

Table 22: Incremental impact results across different matching methods
Indicators Methods In-program 1st Year post-program 2nd Year post-program
Employment earnings ($) KM -91 -536*** 2321***
Employment earnings ($) NN 183 -89 2989***
Employment earnings ($) IPW 263 49 2791***
Incidence of Employment (percentage points) KM 9.3*** 3.75*** 10.87***
Incidence of Employment (percentage points) NN 9.5*** 4.99*** 12.24***
Incidence of Employment (percentage points) IPW 8.44*** 5.82*** 12.52***
EI benefits ($) KM 96*** -831*** -1239***
EI benefits ($) NN -36 -740*** -1159***
EI benefits ($) IPW 26 -710*** -1095***
SA benefits ($) KM -319*** -283*** -266***
SA benefits ($) NN -380*** -326*** -320***
SA benefits ($) IPW -359*** -301*** -259***
Dependence on income support (percentage points) KM -3.27*** -6.98*** -7.93***
Dependence on income support (percentage points) NN -4.34*** -7.27*** -7.87***
Dependence on income support (percentage points) IPW -4.84*** -6.96*** -7.67***

Annex D: Common support before and after KM matching

To ensure that impact estimates are reliable, balancing property tests were carried out on each of the covariates used in the propensity score models before and after matching. Tests for equality of means (t-test) and the standardized differences test (Rosenbaum and Rubin, 1985) were used. Before matching, there were some notable differences between the participants and the comparison cases for most of the covariates. However, after matching all of the significant differences have disappeared. According to the test of standardized differences, there are no large differences in the covariates between participants and comparison cases in the original sample (i.e. before matching). After matching, the results show a considerable reduction in the standardized difference values for many of the covariates. Table 23 below summarizes the results. More detailed results are available upon request.

Table 23: Mean Standardized Bias and Pseudo R² (Kernel Matching)
Program Participants (Before Kernel Matching) Participants (After Kernel Matching) Pseudo R2 (Before) Pseudo R2 (After) Mean Bias (Before) Mean Bias (After) Out of common support
CSJ 52,172 50,907 0.404 0.014 18.8 2.2 1,265*
Figure 7: Propensity Score Distribution Before and After Kernel Matching (KM) for all CSJ
Figure 7: Propensity Score Distribution Before and After Kernel Matching (KM) for all CSJ
  • Source: Common System for Grants and Contributions (CSGC), EI databank (EI Part I and II data), and the Canada Revenue Agency (CRA) tax data
Figure 7 – Text version

Left panel: Propensity scores distribution before matching (treated vs control)

Right panel: Propensity scores distribution after matching (treated vs control)

Solid blue line for treated, dotted red line for control.

Page details

2026-05-19