Summative evaluation of the Horizontal Youth Employment Strategy

Career Focus (CF) - Key incremental impact (program attribution) findings

• CF participants experienced a six-year cumulative gain of $40,488 in annual employment earnings versus the comparison group.
• CF participants used modest amounts of Employment Insurance (EI) benefits and did not rely heavily on Social Assistance (SA) benefits.
• The cost-benefit analysis shows that overall gains in employment earnings were 4.74 times the average cost ^{Footnote 1} per CF participant.

Skills Link (SL) - Key outcome and incremental impact (program attribution) findings

• Following their participation in SL, one-quarter of the participants returned to school.
• Eighty-eight percent of SL participants reported finding employment after the program.
• Participants experienced a steady increase in earnings from one year prior to their SL program start year to five years after (from $3,651 to $14,009).
• Approximately 24% of SL participants were in receipt of SA benefits after participating in SL, an increase of 4 percentage points from the year prior to the program start year.
• SL participants spent comparatively less time in programming (on average 3.6 months) than CF participants who spent on average 6.4 months.
• SL participants were more likely to be employed than the limited treatment comparison group members.
• In the five years after SL, participants’ average earnings were $875 less per year than the comparison group. The importance of educational attainment is clear in the findings; subgroup analysis found that those with post-secondary education earned on average $1400 more per year than their comparison group members.
• A cost-benefit analysis could only be conducted for participants who had some or had completed their university education. The analysis demonstrated that overall gains in employment earnings were equivalent to the average cost per SL participant.

Summer Work Experience (SWE) – Key SWE outcome findings

• SWE survey respondents reported that their work experience was related to the occupation for which they were studying or hoped to study, allowing them to make decisions regarding whether or not to continue in a given career path.
• Participation in SWE also allowed participants to generate income to fund their education. On average, SWE participants were able to save enough for one-third of their educational expenses for the following school year.

Please refer to Section 3. Findings, conclusions, and recommendations for the key recommendations issued by this evaluation.

Actions taken

ESDC takes numerous steps to ensure adherence to program eligibility criteria for its YES program streams. For example, ESDC ensures client eligibility for Career Focus by using a Participant Information Form (PIF) where information on participant’s profile and situation is collected and carefully reviewed against requirements specified in the Career Focus Operational Directives. Social Insurance Numbers are also requested and verifications are conducted to ensure that participants have not been in receipt of Employment Insurance or prior Career Focus funding. In addition, ESDC subjects each of its contribution agreements to activity, financial and results monitoring.

Actions proposed

Compliance with YES Terms and Conditions is important in order to maintain the integrity and quality of YES programs. Management commits to continue monitoring ESDC’s programs to ensure adherence to eligibility criteria. As the lead department for the YES, ESDC commits to meeting with all YES partner departments/agencies in the next 12 months to reinforce the importance of adhering to program eligibility criteria as set out in the Terms and Conditions of the YES.

Actions taken

ESDC’s SL programming has been progressively realigned since 2010 toward a greater emphasis on direct work experiences and advanced skills development interventions.

Actions proposed

As part of the broader review of the YES, Management commits to examine the different components of the SL program and how they can be further improved upon in order to achieve stronger results and greater cost-effectiveness. The process will be informed by international best practices and will include a review of the program’s design for all YES partner departments/agencies who deliver SL.

Actions taken

Management is pleased that this evaluation was able to conduct incremental impact analysis using ESDC’s data.

Actions proposed

ESDC will continue to collect information that can allow for incremental impact analysis as part of its data collection strategy. Within the next 12 months, Management will engage YES partner departments/agencies to explore options to improve data collection across all YES programs.

• 2. The Performance Measurement Strategy should be revised by ESDC to include a corresponding data collection strategy. Information on the content of YES programming should be detailed and robust and be reported uniformly in the Common System for Grants and Contributions and in the Data Collection System. The strategy should be shared with YES partner departments/agencies.

Actions taken

A Performance Measurement Strategy (PM) for ESDC delivered programs is currently being prepared in collaboration with the Evaluation Directorate (along with a corresponding data collection strategy). Management is also in the process of improving online services for ESDC’s YES recipients to report results electronically.

Actions proposed

Once the PM Strategy and corresponding data collection strategy have been finalized, Management will share strategies with YES partner departments/agencies and invite them to adapt their own data collection and reporting based on ESDC’s strategy. Management will also encourage YES partner departments/agencies to undertake routine program monitoring.

Over the next 12 months, Management will engage YES partner department/agencies to examine and implement changes where feasible, which will improve the overall quality of data collection for YES program administration, operations, analysis and evaluations.

Options in data collection for SL and CF programs will be explored and will include the possibility for tracking additional participant profiles and more detailed data on interventions as well as being able to link these data to participants’ outcomes.

Other improvements for all program streams could also include the examination of more effective means for capturing administrative data such as transitioning to electronically captured data.

Actions taken

ESDC has already begun to review and investigate/implement opportunities for enhancement in the application process. As part of the Gs and Cs Modernization, ESDC has introduced several standardized processes and tools to improve the application process, such as the Funding Portal, the standard Application for Funding form and the standard Contribution Agreement.

ESDC is also exploring alternative funding delivery models including longer term contribution agreements. Multi-year agreements are now permitted under the 2013 revised YES Terms and Conditions (up to five years). These improvements are important as they will allow for an increase in operational efficiencies as well as a decrease in program cost and administrative burden.

Actions proposed

ESDC will continue to monitor the application process with these alternative delivery models.

Actions taken

Every year through Skills Canada, ESDC funds trades-focused skills competitions which provide secondary and post-secondary students with valuable hands-on exposure to in-demand fields. Through these skills competitions, secondary and post-secondary students are also exposed to careers in science, technology, engineering, mathematics and in the skilled trades as well.

In addition, EAP 2013 announced that the GoC would “reallocate $19 million over two years to inform young people about fields of study that are relevant to existing and forecasted demand for labour in particular occupations”. Management is currently in the process of implementing this commitment with the YES.

Actions proposed

In 2013, the YES served over 55,000 Canadian youth. In the context of the YES review, there is an opportunity for Management to explore ways in which labour market and career-related information could be better integrated into YES programming for the benefit of all these youth. Over the next 12 months, Management will engage YES partner departments/agencies to explore potential means for distributing labour market and career information to YES participants.

3.2.1 Career Focus stream

Client outcome findings

Since completing their Career Focus (CF) internship, 97% of survey respondents reported that they had been employed at some point since their CF internship ended. Approximately one in five (22%) CF participants reported returning to school or attending training of at least three months in length following their participation in the CF program. The decision to return to school was identified by over half of survey respondents (59%) as being influenced by participating in CF. Overall, employers interviewed indicated that they were satisfied with their CF experience, while the majority of CF participants (88%) indicated that they were satisfied with the quality of their work experience.

Table 1 presents the average annual employment earnings, as well as Employment Insurance (EI) and Social Assistance (SA) benefits received by CF participants in the 2003-2006 cohort (n=2,994), one year prior to program participation, in the program start year, and up to five years after program participation.

Source: Administrative data – ESDC CF participants only

As demonstrated in Table 1, the average annual employment earnings of participants steadily increased by a total of $26,193 since the year before the program start year, from $10,121 to $36,314 in the fifth year after program participation. In addition, when EI benefits and participation are examined from the year prior to program start, to the fifth year after program participation, there was a small increase in the average amount of EI benefits received, by $204, as well as EI participation, by 1%. It is worth noting that EI participation gradually declined from a high of 13% in the year immediately after program participation, to 9% in the fifth year of program participation, suggesting that improvements in participants’ labour market outcomes were most likely to occur in the year following program participation. In addition, average SA benefit use remained limited, with only a slight increase in the fifth year after program participation, by $65, in comparison to the year prior to program participation. Overall, SA benefit use remained relatively steady in all years examined in Table 1.

3.2.2 Skills Link stream

Key SL outcome findings – Comparing pre- and post-program participation

Overall, the findings for the Skills Link (SL) program stream indicate that the program is serving its targeted age population. A large majority of SL participants surveyed (88%) reported finding employment after the program. A majority of participants (70%) who had found employment attributed it to their participation in the program. One-quarter (25%) of SL participants returned to school after participating in the SL program. The majority of those who had returned to school (79%) stated that SL had impacted their decision to return to school. Seventy-four percent of SL participants reported that they had obtained assistance with a job search either during or just prior to their SL program, and 43% obtained specific career or local labour market information. The evaluation results demonstrated that most SL participants (86%) were satisfied with the quality of the programming received.

Table 4 presents the average annual employment earnings, as well as EI and SA benefits received by SL participants in the 2003-2006 cohort (n=50,726), one year prior to program participation, in the program start year, and up to five years after program participation.

Source: Administrative data – ESDC SL participants only

As demonstrated in Table 4, the average annual earnings of SL participants steadily increased from $3,651 one year prior to program start, to $14,009 five years after program participation, for a total increase of $10,358 over this six year period. In addition, when EI benefits and participation are examined from the year prior to program start, to the fifth year after program participation, there was an increase in the amount of EI benefits received, by $533, as well as EI participation, by 9%, suggesting an improvement in labour market participation over time. In addition, the percentage of SL participants receiving SA rose from one in five (20%) one year prior to the program start year, to one in four (24%) five years after the program start year. The findings suggest that a proportion of the SL clients continue to experience difficulties with labour market attachment following their participation in SL programming.

3.2.3 Summer Work Experience stream

Key SWE outcome findings

Overall, Summer Work Experience (SWE) reached its targeted population. Almost all (99%) SWE participants were in the target age range of 15 to 30 years of age (inclusive) at the time of program participation. ^{Footnote 12}

The majority of SWE survey respondents indicated that they would not have found an equivalent work experience opportunity without the SWE program. They also reported that their work experience was related to the occupation for which they were studying or hoped to study, allowing them to make decisions regarding whether or not to continue in a given career path. Participation in SWE also allowed participants to generate income to fund their education. On average, SWE participants were able to save enough for one-third of their educational expenses for the following school year. Both participants and employers were satisfied with their SWE experience.

During their work placement, most SWE participants generally worked full-time hours over an average of 12 weeks; however placement length ranged from as short as two weeks to as long as 28 weeks. These variations are important to note as program officials interviewed indicated that the duration of the work experience is an important factor in facilitating acquisition of skills by participants.

SWE participants indicated that they learned technical (occupation specific), employability, and life skills during their SWE placement. These skills were most commonly acquired through mentoring and coaching. Numeracy skills and written communication skills were less frequently mentioned as being improved through SWE participation, thus suggesting room for improvement in acquiring these competencies. SWE programming is restricted to providing a wage subsidy over a fairly short period of time and is thus a relatively low-cost intervention. The average cost per SWE participant during the 2008-2011 reference period was approximately $3,000 per work placement.

Recommendation #3

In order to ensure high quality evaluations going forward, it is recommended that the following areas concerning the collection and quality of data be addressed:

• 1. The data required to conduct incremental data analyses (e.g. Social Insurance Numbers) should continue to be collected by ESDC and additional avenues for incremental impact analysis should be pursued, where feasible, by YES partner departments and agencies.
• 2. The Performance Measurement Strategy should be revised by ESDC to include a corresponding data collection strategy. Information on the content of YES programming should be detailed and robust and be reported uniformly in the Common System for Grants and Contributions and in the Data Collection System. The strategy should be shared with YES partner departments and agencies.

Employers and/or project sponsors of CF and SWE program streams raised the issue of timing of funding as an area of improvement. It was suggested that they be given earlier notification of successful funding applications as well as decisions made on funding for longer terms or for multi-year contracts. Earlier notification of approved funding would enable employers and/or project sponsors to adequately prepare for and advertise the program and positions in a timely manner.

Recommendation #4

It is recommended that ESDC review the application process and bring about improvements regarding the more timely release of funding as well as alternative funding delivery models.

The evaluation results showed that few participants in SWE and CF had accessed labour market information or career related information from any source, including through their participation in YES. The majority of the participants who did receive these employment services considered them useful in their search for a job. These findings would indicate that there is an opportunity to better incorporate such employment services in all three YES streams.

Recommendation #5

It is recommended that ESDC and YES partner departments/agencies work to improve the distribution of relevant labour market and career information to youth.