Evaluation of the Canada – Nova Scotia Labour Market Development Agreement

On this page

• Executive summary
• Management response
• 1. Introduction
• 1.1 Labour Market Development Agreement background
• 1.2 Methodology
• 1.3 Overview of the studies summarized in this report
• 2. Evaluation findings
• 2.1 Rationale and Labour Market context
• 2.2 Skills Development
• 2.3 Skills Development-Apprentices
• 2.4 START
• 2.5 Self-Employment
• 2.6 Job Creation Partnerships
• 2.7 Employment Assistance Services
• 3. Comparison of key findings by program type
• 4. Conclusion
• 5. Recommendations
• References
• Acronyms
• Appendix A - Methodology
• Appendix B - Detailed Results Skills Development
• Appendix C - Detailed Results Skills Development-Apprentices
• Appendix D – Detailed Results for START
• Appendix E - Detailed Results Self-Employment
• Appendix F - Detailed Results Job Creation Partnerships
• Appendix G - Detailed Results Employment Assistance Services
• Appendix H - List of Nine Studies Included in the Synthesis Report

Executive summary

Introduction

Employment and Social Development Canada (ESDC) worked jointly with Nova Scotia and 11 other Provinces and Territories (P/Ts) to undertake the 2012 to 2017 second cycle for the Labour Market Development Agreement (LMDA) evaluation. The first cycle of LMDA evaluation was carried out between 1998 and 2012 and involved conducting bilateral formative and summative evaluations in all P/Ts. Under the second cycle, the evaluation work consisted of conducting two to three studies per year on the Employment Benefits and Support Measures (EBSM) similar programming delivered under these agreements. The studies generated evaluation evidence on the effectiveness, efficiency and design/delivery of EBSMs for Canada, for Nova Scotia and for the 11 other P/Ts.

Under LMDAs, Canada transfers $2.14B in Employment Insurance (EI) Part II funds to P/Ts for the design and delivery of programs and services to help unemployed individuals, mainly those eligible for EI, to find and maintain employment.

Programs and services delivered by Nova Scotia have to correspond to the EBSM categories defined under the EI Act. The following is a short description of the five programs and services examined in the evaluation:

• Skills Development (including Apprenticeships) helps participants obtain employment skills by giving them financial assistance in order to attend classroom training.
• Targeted Wage Subsidies (START) help participants obtain on-the-job work experience by providing employers with a wage subsidy.
• Self-Employment provides financial assistance and business planning advice to participants to help them start their own business.
• Job Creation Partnerships provide participants with opportunities to gain work experience that will lead to ongoing employment. Employment opportunities are provided by projects that contribute to developing the community and the local economy.
• Employment Assistance Services such as counselling, job search skills, job placement services, provision of labour market information and case management.

Labour Market Partnerships are also available under the LMDA in Nova Scotia; however, this program will be evaluated at a later stage.

Table i provides an overview of the share of funding allocated to the five EBSMs examined under the second cycle of the LMDA evaluation in Nova Scotia and the average cost per participant.

Sources: EI Monitoring and Assessment Reports 2002 to 2003 to 2014 to 2015.

This report presents a summary of the findings from nine studies produced on the Nova Scotia LMDA interventions. Results are presented for active and former EI claimants, and for long-tenured workers^{Footnote 1}, youth (under 30 years old) and older workers (55 years old and over) when the number of participants was sufficient to conduct quantitative analyses. Active EI claimants were receiving EI benefits at the time of their EBSM participation. Former EI claimants received EI up to three years before starting their EBSM participation.

Key findings

Effectiveness and efficiency of EBSMs

Incremental impacts and cost-benefit analyses addressed EBSM effectiveness and efficiency. Overall, incremental impacts demonstrate that LMDA programs and services are improving the labour market attachment of active and former EI claimant participants in Nova Scotia. As well, the social benefits of participation exceeded the cost of investments for most interventions over time. Finally, providing Employment Assistance Services interventions earlier during an EI claim (first twelve weeks) produced larger impacts on earnings and employment and facilitated an earlier return to work (especially during the first four weeks). This demonstrated the importance of targeting early participation of EI active claimants.

Figure i presents the incremental impacts on the incidence of employment for active and former claimants by type of program. The estimates can be interpreted as a change in the probability of being employed following participation. For example, participation in Skills Development increases the probability of being employed by 4.9 percentage points for active EI claimants relative to unemployed non-participants.

Text description of Figure i

Figure i. Change in Probability of Being Employed in Participants Relative to Non-Participants

		Skills Development	START	Job Creation Partnerships	Employment Assistance Services
Percentage Points	Active Claimants	4.9	5.1	5.1	1.4
	Former Claimants	4.2	7.8	6.7

Figure ii presents the cumulative increase in incremental earnings for active and former claimants over the 5 years post-participation. For example, relative to unemployed non- participants, active claimants who participated in Skills Development earned a cumulative of $23,728 in incremental earnings over the 5 years post-program participation.

Text description of Figure ii

Figure ii. Increased Cumulative Earnings of Participants Relative to Non-Participants

		Skills Development	START	Job Creation Partnerships	Employment Assistance Services
Cumulative Gains in Earnings	Active Claimants	$23,728	$13,617	$11,976	$5,287
	Former Claimants	$8,131	$11,841	$11,312

Table ii presents the number of years required for the social benefits to exceed program cost. Social benefits to participation exceeded investment costs in a period ranging between two and 12 years after program participation.

Main challenges about the EBSM design and delivery

Key informant interviews with service providers and program managers as well as a document review and the questionnaires completed by Nova Scotia representatives revealed specific challenges related to program design and delivery in Nova Scotia.

Skills Development

The application process for Skills Development aims to ensure that prospective participants are choosing training that will meet the labour market demand.

According to service providers and managers interviewed as part of the evaluation, the main challenges related to Skills Development design and delivery included:

• An administrative burden associated with the complexity of the financial aspects of Skills Development application (applying for student loan, family versus individual income).
• Lack of support to address barriers such as learning disabilities and mental health issues.
• Level of financial support is insufficient because tuition fees are not covered 100% and the amount of living allowance is low.
• High number of caseloads per caseworker.

Skills Development-Apprentices

Existing literature showed that there is a fairly high non-completion rate among apprentices. It was not possible with the available data to generate a reliable estimation of the completion rate of Skills Development-Apprentices participants in Nova Scotia. According to key informants from Nova Scotia and other provinces and territories, the drop-out from the apprenticeship process was due to factors such as:

• Financial difficulties during training.
• Apprentices leaving the trade.
• Employers unwilling to release their apprentices for training.
• Lack of training opportunities in local communities and/or low demand for certain trades.
• Lack of or low level of essential skills.

START

While evaluation results have demonstrated the effectiveness of START, the number of new interventions decreased between 2005 to 2006 and 2011 to 2012. According to key informants, employers may not be inclined to use the START program because:

• They are unfamiliar with the on-line process and find it complex.
• They do not have time to spend learning the application process and maybe unable to reach someone for assistance with the on-line process when they run into difficulties.
• They lack of awareness about the program.
• They have negative perception about potential START participants.

Increased awareness and referral as well as enhanced flexibility are credited with the recent increase in the number of participants in START since 2012 to 2013.

Employment Assistance Services

Key informants interviewed in the evaluation confirmed the need to have labour market information to support the delivery of Employment Assistance Services. They also reported the need for service providers to foster relationships with employers to develop local labour market information.

Challenges related to Employment Assistance Services included:

• Lack of awareness about available services
• Low levels of computer skills can create challenges for clients with online applications
• Limited hours of operation for some service providers can limit accessibility
• Service providers’ difficulty in attracting and retaining qualified staff because of uncompetitive set of maximum pay rates and short contracts

Recommendations

A total of six recommendations emerge from the evaluation findings. They are as follows:

The study on the timing of Employment Assistance Services participation showed that receiving assistance early after starting an EI claim can lead to better labour market impacts. However, key informants reported a lack of awareness about programs and services.

Recommendation 1: Consideration should be given to providing Nova Scotia with timely access to data on new EI recipients for supporting targeting and increasing awareness.

Key informants reported that mental and physical disabilities, learning disabilities and lack of essential skills or education were common barriers to accessing and completing training.

Recommendation 2: Consideration should be given to remove barriers to accessing and completing training such as literacy/essential skills training and learning disability assessments. The measures would help individuals with multiple barriers to prepare for vocational training and to reintegrate the labour market. The measures should be reported separately from other Skills Development interventions given their unique objectives.

Key informants interviewed in the evaluation confirmed the necessity of having labour market information to support program delivery. They, however, pointed to the difficulty of accessing labour market information at the regional or local level.

Recommendation 3: Consideration should be given about enhancing the capacity of service providers to access or produce, when needed, relevant labour market information.

The evaluation was not able to produce a conclusive assessment of Self-Employment effectiveness and efficiency since the data used to assess impacts on earnings may not be the best source of information available to reflect the financial wellbeing of the participants. As well, little is known about the design and delivery of this program. Overall, it is not clear whether the participant’s success in improving their labour market attachment through self-employment is more closely associated with their business idea and their entrepreneurship skills than the assistance provided under Self-Employment.

Recommendation 4: Consideration should be given to examine in more detail the design and delivery of Self-Employment and whether the performance indicators for this program are appropriate.

Job Creation Partnerships was found to be particularly effective at improving earnings and incidence of employment. However, the evaluation has not yet examined the design and delivery of this program. Therefore, a lot remains unknown about how this program operates and the factors that contribute to its effectiveness.

Recommendation 5: Future evaluation work should examine the design and delivery of the Job Creation Partnerships to better understand how this program operates in Nova Scotia.

Overall, the LMDA evaluation was able to produce a sound assessment of EBSM effectiveness and efficiency because the team had access to rich data on EI claimants, EBSM participation data and Canada Revenue Agency taxation files. However, some data gaps limited the evaluation’s ability to assess how EBSMs operate.

Recommendation 6: Improvements in the data collection is recommended to address key program and policy questions of interest to the federal and provincial/territorial governments. Specifically:

• Collect data on whether participants are members of designated groups including Indigenous peoples, persons with disabilities and recent immigrants.
• Collect data on the type of training funded under Skills Development and the type of assistance provided under Employment Assistance Services. Nova Scotia, ESDC and other provinces and territories should work together to define common categories for both EBSMs.

Management response

The Department of Labour and Advanced Education of Nova Scotia worked jointly with ESDC and eleven other provinces and territories to undertake the 2012 to 2017 second cycle for the LMDA evaluation. Nova Scotia was honoured to co-chair the work of the LMDA Evaluation Steering Committee and would like to thank ESDC and all participating provinces and territories for their work and dedication.

The Department of Labour and Advanced Education accepts the evaluation findings and conclusions, and has identified the following actions in relation to its recommendations.

Recommendation 1: Consideration should be given to providing Nova Scotia with timely access to data on new EI recipients for supporting targeting and increasing awareness.

Response: The evaluation clearly indicates that those who receive labour market programs and services early in their EI claim period have better labour market outcomes. Nova Scotia agrees that timely access to data on new EI recipients for supporting targeting and awareness should be considered by the Government of Canada.

Recommendation 2: Consideration should be given to remove barriers to accessing and completing training such as literacy/essential skills training and learning disability assessments. The measures would help individuals with multiple barriers to prepare for vocational training and to reintegrate the labour market. The measures should be reported separately from other Skills Development interventions given their unique objectives.

Response: Nova Scotia agrees that the identification and removal of barriers that hinder access to, and the completion of, adult basic education, a high school credential, and/or essential skills training should be a priority. Nova Scotia also agrees there is a need for learning disability assessments.

Recommendation 3: Consideration should be given about enhancing the capacity of service providers to access or produce, when needed, relevant labour market information.

Response: Nova Scotia agrees that the capacity of its service providers to access and produce labour market information should be enhanced. The transformation of Nova Scotia’s labour market program and service delivery system into Nova Scotia Works has made this a priority. Accordingly, Labour and Advanced Education initiated a Local Labour Market Information Pilot Project. The pilot will focus on leveraging regional networks to improve the collection and sharing of information; starting with the recently-established network of Nova Scotia Works Centres Employer Engagement Specialists. The pilot will be conducted in two phases. Phase one includes designing and piloting the data collection and validation process and phase two includes an assessment of the process/tools, governance model, and recommendations for moving forward.

Recommendation 4: Consideration should be given to examine in more detail the design and delivery of Self-Employment and whether the performance indicators for this program are appropriate.

Response: Given the challenges inherent to measuring entrepreneurship programs, Nova Scotia agrees that efforts should be made to ensure that support to entrepreneurs and their outcomes measured accurately.

Recommendation 5: Future evaluation work should examine the design and delivery of the Job Creation Partnerships to better understand how this program operates in Nova Scotia.

Response: Nova Scotia agrees that future evaluation efforts should include the Job Creation Partnerships’ program.

Recommendation 6: Improvements in the data collection is recommended to address key program and policy questions of interest to the federal and provincial/territorial governments. Specifically:

Collect data on whether participants are members of designated groups including Indigenous peoples, persons with disabilities and recent immigrants.

Collect data on the type of training funded under Skills Development and the type of assistance provided under Employment Assistance Services. Nova Scotia, ESDC and other provinces and territories should work together to define common categories for both EBSMs.

Response: Nova Scotia agrees that collecting data on whether clients identify as members of a designated group including Indigenous peoples, persons with disabilities and recent immigrants as well as other targeted groups identified by Nova Scotia would be of value. Nova Scotia has taken steps to upgrade its Labour Market Programs Support System to ensure data integrity and effective business analysis. These steps will greatly assist Nova Scotia in collecting labour market data relating to targeted groups.

Nova Scotia agrees that common category definitions for training and assistance under the Skill Development benefit and the Employment Assistance Services measure would be beneficial.

1. Introduction

Employment and Social Development Canada (ESDC) worked jointly with Nova Scotia and 11 other Provinces and Territories (P/Ts) to undertake the 2012 to 2017 second cycle of the Labour Market Development Agreement (LMDA) evaluations. The first cycle of LMDA evaluation was carried out between 1998 and 2012 and involved the conduct of bilateral formative and summative evaluations in all P/Ts. Under the second cycle, the evaluation work consisted of conducting two to three studies per year on the Employment Benefits and Support Measures (EBSM) similar programming delivered under these agreements. The studies generated evaluation evidence on the effectiveness, efficiency and design/delivery of EBSMs for Canada, Nova Scotia and for the 11 P/Ts that opted for a joint evaluation process with the Government of Canada.

This report presents a summary of the findings from nine studies conducted for Nova Scotia. The report is organised as follows:

• Introduction with an overview of the studies summarized in this report including their scope, methodology, and contextual information on the LMDAs.
• Findings section with a discussion around the rationale for investing in labour market programming.
• Conclusions and lessons learned.
• Recommendations that emerge from the evaluation findings.

1.1 Labour Market Development Agreement background

LMDAs are bilateral agreements between Canada and each P/T, and were established under Part II of the 1996 Employment Insurance (EI) Act. As part of these agreements, Canada transfers $2.14B annually to P/Ts (including $190M in administration funds) to design and deliver programs and services to assist individuals prepare for, obtain and maintain employment. Specifically, Nova Scotia receives approximately $79M in EBSM funding plus $10.1M in administration costs each year.

On April 24, 1997, the Canada-Nova Scotia LMDA was signed as an agreement on a Framework for Strategic Partnerships. Under this LMDA, Canada and Nova Scotia maintained separate responsibility for the delivery of their own labour market programs and services while agreeing to work closely in identifying areas of common interest. On June 13, 2008, a new LMDA was signed transferring responsibility to Nova Scotia for the design and delivery of programs and services classified under two categories: 1) Employment Benefits and 2) Support Measures.

Employment Benefits

Employment Benefits are offered to unemployed individuals who 1) are actively on EI (i.e., active claimants); 2) ended their benefit period within three years before participating (former claimants); or 3) established a claim for maternity or parental benefits within the past five years and are returning to the labour force for the first time (former claimants).^{Footnote 2} Employment Benefits include the following categories:

• Skills Development helps participants obtain employment skills by giving them financial assistance to enable them to select, arrange and pay for classroom training.
• Targeted Wage Subsidies (START) help participants obtain on-the-job work experience by providing employers with financial assistance to help pay the participants’ wages.
• Self-Employment provides financial assistance and business planning advice to EI-eligible participants to help them start their own business. This financial assistance is intended to cover personal living expenses and other expenses during the initial stages of the business.
• Job Creation Partnerships provides participants with opportunities to gain work experience that will lead to ongoing employment. Employment opportunities are provided by projects that contribute to developing the community and the local economy.
• Targeted Earnings Supplements encourage unemployed persons to accept employment by offering them financial incentives. This program was not covered by the evaluation.

Support Measures

Support Measures are available to all unemployed individuals including those not eligible to receive EI and include:

• Employment Assistance Services such as individual counselling, action planning, help with job search skills, job-finding clubs, job placement services, the provision of labour market information, as well as case management and follow-up.
• Labour Market Partnerships provide funding to help employers, employee and employer associations, and communities improve their capacity to deal with human resource requirements and implement labour force adjustments. These partnerships involve developing plans and strategies, and implementing labour force adjustment measures. This measure was not covered by the evaluation.
• Research and Innovation supports activities that identify better ways of helping people prepare for or keep employment and be productive participants in the labour force. Funds are provided to eligible recipients to enable them to carry out demonstration projects and research for this purpose. This support measure was not covered by the evaluation.

Table 1 provides an overview of the share of funding allocated to the five programs and services examined under the second cycle for LMDA evaluation and the average cost per participant. The average cost per participant was calculated based on the 2002 to 2005 data from the EI Monitoring and Assessment Reports. The 2002 to 2005 period corresponds to the cohort of participants selected for incremental impacts and cost-benefit analysis in the LMDA evaluation.

Sources: EI Monitoring and Assessment Reports 2002 to 2003 to 2014 to 2015.

1.2 Methodology

This section presents key aspects of the quantitative analyses carried out as part of the LMDA studies, while a more detailed description of the methodology is provided in Appendix A.

All quantitative analyses were based on administrative data from the EI Part I (EI claim data) and Part II (EBSM participation data collected by Nova Scotia and transferred to Canada) linked to the T1 and T4 taxation files from the Canada Revenue Agency. Incremental impact analyses and the cost-benefit analyses were based on up to 100% of participants in the reference period selected.

Incremental impacts analysis

Five studies assessed program effectiveness by estimating incremental impacts from EBSM participation on participants’ labour market experience (for example, earnings from employment/self-employment, incidence of employment, use of EI or social assistance and dependence on income support) after participation. The role of the incremental impact analysis is to isolate the effects of participation from other factors such as the economic cycles. To achieve this, the incremental impact analysis compared the labour market experience of participants, before and after their participation, with that of non-participants (see the Example of Incremental Impact Calculation in Figure 1).

The matching of participants and comparison group members used up to 75 socio-demographic and labour market variables observed over five years before participation. Two different comparison groups were used to measure impacts for active and former EI claimants. For active claimants, the incremental impacts were measured relative to a comparison group of active claimants who were eligible to participate in EBSMs, but did not, during the reference period.

Former claimants can be underemployed and unable to requalify for EI, out of the labour force for various reasons or on social assistance. Based on previous evaluation methodologies, on expert advice and given the difficulty in generating a suitable comparison for former claimants using administrative data alone, the comparison group for former claimants was created using individuals who participated in Employment Assistance Services only during the reference period. This is a conservative approach given the fact that participation in Employment Assistance Services can lead to limited effects on labour market outcomes. In other words, the experience of former claimants who received Employment Benefits (i.e., Skills Development, START, Self-Employment and Job Creation Partnerships) was compared to the experience of former claimants who received low intensity employment services (i.e. Employment Assistance Services only). Due to this difference in measurement, incremental impacts estimated for active claimants should not be directly compared to those of former claimant participants^{Footnote 3}.

Text description of Figure 1

Figure 1 Example of Incremental Impact

Participants
Average Annual Earnings	Before participation	After participation	Change in earnings
	$30,000	$38,000	$8,000
Comparison Group
Average Annual Earnings	Before participation	After participation	Change in earnings
	$31,000	$36,000	$5,000
Incremental Impact (Change due to program participation): + $3,000 (in other words, $8,000 - $5,000)

Factors accounted for in the Cost-Benefit analysis

Program efficiency was assessed through a cost-benefit analysis which compared the cost of participating in the program for the participants and the cost of delivering the program for the government to the benefits generated by the program. Overall, this analysis provided insight on the extent to which the program is efficient for society (that is, for both the participants and the government). The costs and benefits accounted for in the calculations were as follows (see detailed definitions in Appendix A):

• Program costs include program and administration costs paid by the government.
• Marginal social costs of public funds represent the loss incurred by society when raising additional revenues such as taxes to fund government spending.
• Employment earnings consist of incremental impacts on participants’ earnings during and after participation. The calculation accounts for the participant’s forgone earnings during participation (that is, opportunity cost). Employment earnings were also increased by 15% to account for fringe benefits such as employer-paid health, life insurance and pension contributions.

Strengths and Limitations of the Studies

One of the key strength of the studies is that all quantitative analyses were based on administrative data rather than survey responses. Compared to survey data, administrative data are not subject to recall errors or response bias.

The propensity score models used to match participants and non-participants for the incremental impact analyses are judged to be robust in part because they were based on five years of pre-participation data and on a vast array of variables including socio-demographic characteristics, location, skills level related to last occupation and indicators of labour market attachment. Sensitivity analysis and the use of alternative estimation methods have increased confidence in the incremental impact estimates. However, one limitation with the propensity score matching techniques is that no one can be fully sure the impacts were not influenced by factors not captured in the data.

The cost-benefit analysis accounted for all quantifiable costs and benefits that are directly attributable to the EBSMs and could be estimated with the available administrative data. The analysis did not account for non-quantifiable benefits such as improvements in participant’s wellbeing or for the multiplier effect of increased spending on the economy.

In some studies that relied on the use of qualitative data collection methods, the number of key informants was relatively small. Reponses provided by key informants reflect their own experience in their own region and may not be fully representative of the entire province.

1.3 Overview of the Studies Summarized in This Report

Findings presented in this report were drawn from nine separate studies carried out in Nova Scotia. These studies examined issues related to EBSM effectiveness, efficiency, design/delivery and used a mix of qualitative and quantitative methods. Each study examined evaluation issues in relation to active and former EI claimants.

Table H1 in Appendix H presents an overview of these studies, including the type of evidence generated, the methods used, the reference period and the length of the post-program period over which program effects were observed.

2. Evaluation findings

2.1 Rationale and labour market context

LMDA investments align with provincial government priorities

Active labour market programs are fairly similar across the Organization for Economic Co-operation and Development countries and consist of skills training in a classroom setting, work experience with employers (often subsidized) or in the public/non-profit sector, return-to-employment assistance and self-employment assistance. In Nova Scotia, the Department of Labour and Advanced Education administers a range of active labour market programming targeted to various groups of individuals.

In their Speech from the Throne (September 14, 2014), the Nova Scotia government committed to “streamline processes and direct our resources to the programs that are successfully getting more people into the workforce” by increasing “opportunities for ongoing skills development and knowledge growth”.

Furthermore, the Government of Nova Scotia, as part of their recent budget, promised to work together with business “to address unemployment and help more skilled people find work in Nova Scotia” by “improving services so all unemployed Nova Scotians have support when looking for work.” As well, the government is “investing in education, youth and job training because that’s how we can help grow our economy and create jobs.”^{Footnote 4}

Overall, impacts reported in the LMDA evaluation and discussed in this report demonstrated that LMDA funded programs and services delivered in Nova Scotia are generally helping participants improve their labour market experience after participation. As such, evaluation evidence suggests that LMDA funded programming contributes to achieving government commitments and the strategic priorities of the Department of Labour and Advanced Education to educate, invest and grow the labour market^{Footnote 5}.

2.2 Skills Development

2.2.1 Program description

Based on a document review and ten key informant interviews completed in the summer of 2015

In Nova Scotia, Skills Development provides support for individuals needing financial assistance to take full-time training necessary for sustainable employment. The program provides financial supports including tuition and student fees, books and supplies, basic living allowance, living away from home allowance, child/dependent care, disability supports, transportation and medical/dental insurance fees. Participants are expected to contribute to their skills training and may need to apply for a student loan.

The program supports skills training such as adult basic education, college courses, occupation specific training and university-based training. Training conducted by or in consultation with employers is also supported by Skills Development. Training is generally conducted in a classroom setting and has a duration of 2 years or less.

Table 2 presents the proportion of expenditures and interventions by type of training supported. The majority (91.4%) of program expenditures in 2013 to 2014 were used to support college, occupational and university training while 8.6% of funding supported adult basic education/essential skills. Overall, 88% of interventions represented college, occupational and university training while 12% represented adult basic education/essential skills training.

Source: Information submitted by the Department of Labour and Advanced Education

Table 3 presents the list of top occupations supported in 2012 to 2013 to 2014 to 2015. The most common occupations supported by Skills Development include continuing care assistants (10%) followed by office assistants (7%), heavy equipment operators (7%) and transport truck drivers (6%).

Source: Questionnaire completed by the Department of Labour and Advanced Education

2.2.2 Program delivery

Based on a document review and ten key informant interviews completed in the summer of 2015

The Skills Development application process begins with the unemployed individual being evaluated by caseworkers to determine their needs and identify any barriers to employment. A return to work action plan is developed to identify the appropriate interventions to assist the individual in returning to employment. The individual’s eligibility and suitability for the program, skill level, work experience, family and financial situation, and education levels are assessed by the caseworker.

Skills Development candidates are required to conduct labour market research on potential occupations and training programs for at least 3 months to demonstrate that there is sufficient labour market demand for their chosen training. The caseworker also provides labour market information to participants, validates occupational goals and ensures there is a strong and sound case for the training. The application is then submitted to Employment Nova Scotia, a branch of the Department of Labour and Advanced Education, for approval. Generally, the participant’s prepare their application and the caseworker recommends the application for departmental approval. The application process can take between 6 to 8 weeks.

Caseworkers monitor the participant’s training at least once every 30 days. Monitoring can be increased if the caseworker determines the participant has encountered challenges.

2.2.3 Targeting labour market demand

Overall, the Skills Development application aims to ensure that candidates are choosing a trade that will meet labour market demand. Candidates are expected to conduct labour market research (including job search activities and interviews with employers) as part of the application process. Key informants reported that Skills Development candidates must demonstrate that there is a demand for the type of occupations associated with their chosen training. Labour market information is shared with candidates by their caseworkers. While training is not targeted toward specific occupations, according to the questionnaire, training for occupations in low demand areas may not be supported or recommended for approval.

2.2.4 Profile of Skills Development participants

As shown in Table B1 in Appendix B, the majority of active claimants who started Skills Development participation in 2002 to 2005 were male (56%) but only 48% were male in 2006 to 2008. The majority of participants in both cohorts were under 35 years old (65% and 57% respectively). Participants in both cohorts most frequently held jobs requiring secondary or occupational training prior to participation (37% and 44% respectively). Active claimants who participated in Skills Development in 2002 to 2005 had slightly lower employment earnings in the year before participation ($15,690) compared to those who participated in 2006 to 2008 ($16,147).

Former claimants who started Skills Development participation in 2002 to 2005 and in 2006 to 2008 were predominantly female (53% and 61% respectively) and were under 35 years old (57% and 58% respectively). Secondary or occupational training was the most frequent skill level required by the claimant’s last job prior to their participation in Skills Development (41% in 2002 to 2005 and 46% in 2006 to 2008). Participants in the 2002 to 2005 cohort had slightly higher earnings in the year prior to participation ($8,090) than the 2006 to 2008 cohort ($7,823).

Barriers faced by participants

Based on a document review and ten key informant interviews completed in the summer of 2015

Key informants were asked to identify the main barriers to employment or to participation faced by Skills Development participants. By order of importance, the most frequently identified barriers were the following:

• Mental, physical or learning disabilities
• Lack of child care
• Lack of transportation
• Lack of skills or education
• Lack of work experience

2.2.5 Incremental impacts

Active claimants

As shown in Table B2 in Appendix B, active claimants who started Skills Development participation between 2002 and 2005 had incremental gains in earnings and incidence of employment in the five years following participation. As shown in Figure 2, earnings continuously increased over time with gains ranging between $1,856 in the first year after participation to $6,165 in the fifth year. Increases in incidence of employment ranged between 3.6 and 5.7 percentage points. As well, EI use decreased over the entire post-program period with annual averages ranging between $255 and $504, and social assistance use decreased by annual averages ranging between $101 and $143. The annual decrease in dependence on income support ranged between 1.9 and 3.4 percentage points.

The incremental impacts for the 2006 to 2008 participants followed a similar pattern compared to the impacts for the 2002 to 2005 cohort. Employment earnings and the incidence of employment presented gains in all three post-program years. Gains in earnings ranged between $3,427 in the first year after participation to $7,728 in the third year and the increase in incidence of employment ranged between 6.3 percentage points in the first year and 9.7 percentage points in the third year after participation in Skills Development. As well, the use of EI decreased by annual averages ranging between $440 and $1,051 and the use of social assistance decreased between $59 and $112 annually. The level of dependence on income support decreased between 2.6 and 5.2 percentage points annually.

Text description of Figure 2

Figure 2. Increased Earnings of Active and Former Skills Development Participants Relative to Non-Participants

		Year 1 Post-Program	Year 2 Post-Program	Year 3 Post-Program	Year 4 Post-Program	Year 5 Post-Program
Gains in Earnings	Active Claimants	$1,856	$4,260	$5,448	$5,999	$6,165
	Former Claimants	$444*	$1,592	$1,708	$2,400	$2,168

* The incremental impact is not statistically significant at the 95% confidence level.

Overall, active claimants increased their labour market attachment through increases in earnings, incidence of employment and a decrease in the level of dependence on government income support (use of EI and social assistance).

Results for sub-groups of active claimants were as follows:

• Youth (under 30 years old) who started participation in 2002 to 2005 improved their earnings and incidence of employment in the post-program period, and decreased their dependence on government income support. EI use decreased in the first three years following participation and social assistance use decreased between year 1 and 4 post-program.
• Long-tenured workers who started participation in the 2007 to 2009 period increased their earnings and incidence of employment in years 2 and 3 post-program. They also decreased social assistance use in years 1 and 2 post-program. Estimates for EI use and dependence on government income support were not statistically significant.

Former claimants

As shown in Table B3 in Appendix B, former claimants who started Skills Development participation between 2002 and 2005 had incremental gains in earnings in four of the five post-program years. As shown in Figure 2, average annual gains in earnings ranged between $1,592 and $2,400. As well, average annual gains in the incidence of employment ranged between 3.4 and 4.9 percentage points in the five years following participation. While the use of EI increased by a total of $1,773, the use of social assistance decreased by a cumulative of $1,010 over the five post-program years.

Former claimants who started Skills Development participation between 2006 and 2008 had increases in employment earnings in the second and third year’s post-program of $1,211 and $1,635 respectively. The result for the first year post-program was not statistically significant. Increases in incidence of employment ranged between 3.4 and 5.6 percentage points annually. As well, participation in Skills Development reduced the use of social assistance during the post-program period by an annual average that varied between $233 and $320, and decreased the level of dependence on government income support by 5.9 and 4 percentage points in the first and second year post-program respectively. Incremental impacts on the use of EI were mixed with a decrease in year one post-program by $387 followed by an increase in year three by $405.

Overall, former claimants increased their use of EI following participation. This indicates the inability of some former claimants to maintain the employment secured in the short-term. It can also be argued that the increase in EI use is an indication of increase labour market attachment for this client group since they did experience increases in employment earnings and incidence of employment as well as a decrease in the use of social assistance. As a reminder, former claimants are participants for whom the EI benefit period ended up to three years pre-participation.

Youth who started participation between 2002 and 2005 increased their use of EI by a cumulative of $2,609 and decreased social assistance use by a total of $1,336 over the five years post-program participation. As well, their incidence of employment increased by 4.3 and 4.1 percentage points in the first and second year post-program participation. All other estimates were not statistically significant. The increase in EI use indicates the inability of some youth to maintain the employment secured in the post-program.

2.2.6. Cost-benefit results

As shown in Table B4 in Appendix B, for active claimants, the benefits of Skills Development for society were $9,935 higher than the costs six years following participation. It would take 4.3 years after participation for the benefits to recover the costs. For former claimants, the benefits were $8,182 lower than the costs six years after the end of Skills Development participation. These benefits would match the costs 9.9 years after participation.

2.2.7 Challenges and lessons learned about Skills Development design and delivery

Based on document review and ten key informant interviews completed in the summer of 2015

Key informants identified the following challenges in relation to the design and delivery of Skills Development (order according to the number of key informants from the highest to the lowest):

• The complexity of the financial aspects of Skills Development creates an administrative burden for participants. For example, funding calculations are based on family and not individual income.
• Lack of support to address barriers such as learning disabilities and mental health issues.
• The level of financial assistance may limit participation for some individuals because tuition fees are not covered 100% and the amount of living allowance is low.
• Lack of discussion with Skills Development candidates regarding budget planning.
• There is a gap in financial support when EI benefits are exhausted before the beginning of training.
• Large caseloads (150 to 170 cases per caseworker) create challenges in terms of program access.
• Applying for a student loan can delay the participant’s application and access to the required supports.
• The labour market program support system does not automatically notify participants of the status of their application and the type of financial supports approved.

Key informants identified the following lessons learned:

• Maintaining a strong relationship and good communications with the training providers.
• Have relevant and adequate labour market information on occupations in demand.
• Ensure that candidates understand the expectations associated with the program.
• Government staff must understand the unique challenges associated with serving persons with disabilities and it is important to use the right tools to assist those with learning disabilities.
• More time and resources are needed to assist participants with multiple barriers:
• Importance of conducting a comprehensive assessment of all their barriers and ensuring that they receive the appropriate level of specialized support.
• Simplifying the application process and following-up more regularly with them.

2.3 Skills Development-Apprentices

2.3.1 Program description

Based on document review completed in the summer of 2015^{Footnote 7}

The objective of Skills Development-Apprentices is to assist eligible individuals obtain the skills they need for employment ranging from basic to advanced skills through direct assistance to individuals. Program participants have generally chosen a career and are attached to the labour market.

In Nova Scotia, the apprenticeship program involves on-the-job learning and technical training in a classroom setting. Apprentices are generally employed and on block release at the time of their training. Block release training is also known as in-class technical training.

Program funding is provided to apprentices who are EI eligible to help them offset the apprenticeship costs. This funding is provided to apprentices while they are on training to cover the following types of expenses:

• Transportation/travel (maximum of $150 per week)
• Childcare and/or dependent care (maximum of $100 per week)
• Living away from home allowance (maximum of $175 per week)

Apprentices in Nova Scotia are expected to pay a tuition fee of approximately $90 per week.

The Department of Labour and Advanced Education provided a list of the trades that participants were most frequently trained for in 2013 to 2014. The top five trades identified are:

• Electrician (28%)
• Steamfitter/Pipefitter (10%)
• Automotive technician (9%)
• Carpenter (7%)
• Plumber (7%)

2.3.2 Profile of Skills Development -Apprentices participants

As shown in Table C1 in Appendix C, the vast majority of active claimants who started Skills Development-Apprentices in 2003 to 2005 and 2013 to 2014 were male (97% and 93% respectively). In 2003 to 2005, most participants were 34 years old and younger (82%) however, in 2013 to 2014, most participants were older and between 25 to 44 years of age (83%). As well, 90% and 92% respectively of these participants had employment requiring secondary or occupational training prior to participation. Active claimants who participated in Skills Development-Apprentices in 2003 to 2005 had lower employment earnings in the year before participation ($22,933) compared to those who participated in 2013 to 2014 ($28,870).

The profile of former claimants participating in Skills Development-Apprentices is not available due to the small number of participants.

2.3.3 Labour market outcomes

Table C2 in Appendix C presents the labour market outcomes for active EI claimants who started Skills Development-Apprentices participation during the 2003 to 2005 period.

As shown in Figure 3, the earnings of active claimants who started program participation between 2003 and 2005 grew from $13,425 and $52,285 between the fifth year pre-program and the seventh year after the participation start year. Their incidence of employment decreased from 100% to 94% over the participation start year and the following seven years.

Text description of Figure 3

Figure 3. Average Earnings for Active Claimant Participants in Skills Development-Apprentices

	5 yrs pre-program	4 yrs pre-program	3 yrs pre-program	2 yrs pre-program	1 yr pre-program	Program start year	1 yr post-program	2 yrs post-program	3 yrs post-program	4 yrs post-program	5 yrs post-program	6 yrs post-program	7 yrs post-program
Employment Earnings	$13,425	$15,711	$18,022	$20,517	$24,377	$25,002	$31,649	$37,683	$43,880	$48,092	$49,997	$52,674	$52,285

Outcome results for former claimants who began their participation in 2003 to 2005 were not analyzed due to an insufficient number of observations required to per form the analyses.

2.3.4 Challenges about Skills Development-Apprentices design and delivery

Based on a document review completed in the summer of 2015

Existing literature has shown that there is a fairly high non-completion rate among apprentices in Canada (40 to 50%)^{Footnote 8}. Furthermore, subject matter literature revealed that despite the growth in apprenticeship registrations in Canada, there has not been a corresponding increase in completions^{Footnote 9}. While available data do not provide reliable information on completion and non-completion rates of Skills Development-Apprentices participants, P/T representatives interviewed confirmed this trend. Perspective on Skills Development-Apprentices challenges and lessons learned are reflected in the feedback received from key informants across ten provinces and territories.

National key informants identified factors that could lead the apprentices to dropping out. These included:

• Financial difficulties during training (ten P/Ts)
• Apprentices leaving the trade (seven P/Ts)
• Employers unwilling to release their apprentices for training (seven P/Ts)
• Lack of training opportunities in local communities (seven P/Ts)
• Labour market fluctuations and/or low demand for certain trades (six P/Ts)
• Lack of or low level of essential skills (six P/Ts)

National key informants also highlighted lessons learned related to the design and delivery of Skills Development-Apprentices or apprenticeship in general. These included:

• Providing more financial supports for apprentices (six P/Ts)
• Providing essential skills training to individuals facing multiple barriers to employment prior to the technical training sessions (six P/Ts)
• Conducting needs assessments to identify all potential barriers to training at the beginning of the apprenticeship process (four P/Ts)

2.4 START

2.4.1 Program description

Based on document review and twelve key informant interviews completed in the summer of 2015

Nova Scotia’s START program provides financial incentives to small and medium sized employers needing employees and who are willing to provide training to new employees.

The financial assistance varies depending on the type of employment offered and the skill level of the employee. However, there is a maximum of $25,000 per agreement (job opportunity). There is no maximum duration for the START program however the questionnaire noted that twelve months is the average duration of the program.

The incentives paid to the employer are primarily for wages, mandatory employer-related costs and training costs. Other related costs associated with hiring new employees may also be covered by the subsidy and are negotiated into the agreement.

2.4.2 Program delivery

Based on document review and twelve key informant interviews completed in the summer of 2015

According to key informants, the START program is equally driven by the employer and the participant and the delivery process differs depending on who is applying to the program. An employer must apply online through the Employment Nova Scotia portal. Participants who wish to apply can do so through their case manager at a Careers Nova Scotia Centre.

Case managers are also available to provide assistance to employers with the application process when needed. The approval for the employer’s application may take approximately one week. Once the employer finds a participant, the employer develops and signs an agreement with the case manager.

Participants seeking to apply to the START program approach a case manager to discuss their participation. The case manager meets with the participant, conducts a needs assessment and provides a marketing letter. The participant must attend a service orientation session prior to receiving their marketing letter.

It is the participant’s responsibility to find an employer willing to hire them. However, it is not unusual for employers to contact case managers directly seeking assistance in locating suitable participants.^{Footnote 10}

In Nova Scotia, the START program directs participants towards certain occupations based on the participant’s needs, skill level, desire and goals. Key informants reported that most of the START participants were in the service, retail and administrative industries. Identified occupations include housekeeping, childcare, theatre, festivals and non-profit sector positions.

2.4.3 Profile of START participants

As indicated in Table D1, Appendix D, active claimants who began their START program in 2002 to 2005 and in 2006 to 2008 were predominately male (54% and 56% respectively). Sixty-three percent of the 2002 to 2005 participants were between 25 and 44 years of age compared to 51% of the 2006-2008 participants. Participants in the 2006 to 2008 period were older as 63% of them were 35 years of age and older compared to 54% for the 2002 to 2005 participants. Participants from both cohorts most frequently had jobs requiring secondary or occupational training before participation (35% for 2002 to 2005 participants and 39% for 2006 to 2008 participants). Employment earnings were higher for those who participated in the 2002 to 2005 period ($16,099) compared to the 2006 to 2008 period ($14,654).

Former claimants in both the 2002 to 2005 and 2006 to 2008 cohorts were mainly men (55% each) and were between 25 and 44 years old (60% each). Before participation, they most frequently had occupations requiring secondary or occupational training (33% in 2002 to 2005 and 40% in 2006 to 2008). Those who participated in the 2002 to 2005 period had higher earnings ($10,781) compared to the 2006 to 2008 participants ($8,210).

2.4.4 Incremental impacts

Incremental impact results for active and former claimants are presented in Tables D2 and D3 in Appendix D.

Active claimants

As shown in Figure 4, active claimants who participated in START between 2002 and 2005 had incremental gains in earnings that ranged between $2,179 and $3,380 in the five years post-program period. Increases in the incidence of employment ranged between 4.9 and 7.3 percentage points. EI use decreased a cumulative amount of $2,207 in the post-program period. The estimates for the use of social assistance and the dependence on income support were mainly not statistically significant.

Active claimants who participated in START between 2006 and 2008 had incremental gains in earnings that ranged between $2,642 and $3,889 in the three years post-program period. As well, the use of EI decreased by an annual average ranging between $834 and $1,374. The level of dependence on income support decreased between 7.3 and 9.6 percentage points annually and the use of social assistance decreased by annual averages ranging between $168 and $178.

Overall, active claimants improved their labour market attachment through increases in earnings, incidence of employment, and decrease in the use of EI.

Text description of Figure 4

Figure 4. Increased Earnings of Active and Former START Participants Relative to Non-Participants

		Year 1 Post-Program	Year 2 Post-Program	Year 3 Post-Program	Year 4 Post-Program	Year 5 Post-Program
Gains in Earnings	Active Claimants	$2,179	$3,380	$2,521	$3,234	$2,302*
	Former Claimants	$1,880	$2,437	$2,057*	$2,719	$2,748

*The incremental impacts are not statistically significant at the 95% confidence level.

Former claimants

As shown in Figure 4, former claimants who participated in START in the 2002 to 2005 period had incremental gains in earnings that ranged between $1,880 and $2,748 in the five years post-program period. The incidence of employment increased in all years following participation and ranged between 6.1 and 9.4 percentage points. Social assistance use decreased in the first three years following participation and ranged between $230 and $281. Estimates for the use of EI and the dependence on government income support were not statistically significant.

Former claimant incremental impacts were not produced for the 2006 to 2008 participants or for the other sub-groups as the number of participants were too small.

Overall, former claimants improved their labour market attachment through increases in earnings, incidence of employment, and decrease in the use of SA.

2.4.5 Cost-benefit results

As shown in Table D4 in Appendix D, the benefits of START for active claimants exceeded the costs within 3.3 years after participation from the society perspective. The total benefits six years after program end exceeded the costs by $6,186. Similarly, the benefits of START for former claimants exceeded the cost 1.5 years after participation. The benefits six years after the end of participation exceeded the costs by $10, 907.

2.4.6 Challenges and lessons learned about START design and delivery

Based on document review and twelve key informant interviews completed in the summer of 2015

Despite being effective at helping participants to find and maintain employment, there has been a steady decline in the number of new START interventions delivered per year. The number of new interventions decreased between 2005 to 2006 and 2011 to 2012.

Reasons for the decline in the use of START included:

• Employers are frustrated with the application process because it requires several steps to register to obtain user identification, in addition to delays in securing approvals
• Employers do not have time to spend learning the application process and maybe unable to reach someone for assistance with the on-line process when they run into difficulties
• Lack of communication and awareness of the START program
• Employers’ perception about potential START participants
• Difficulties with the language used in the application. It is very ‘project-based’ and it can be difficult for employers to understand
• Possible slowdown in particular industries whereby employers have either slowed their hiring or stopped altogether

However, there was an increase in the number of START interventions since 2012 to 2013. Reasons for the increase included:

• An increase in employer awareness over the past years
• More individuals are getting directed to START
• The subsidies are larger and eligibility criteria were expanded.

Best practices or the lessons learned in terms of the design and delivery of START included:

• Direct engagement between the START case managers and the employers could help alleviate the challenges regarding the program’s awareness and outreach
• Having a database of employers seeking START participants available on the labour market program support system to match START participants with available employers and potentially leading to increase retention after the subsidy ends
• Case managers could complete applications on behalf of employers to speed up the process
• Encouraging participants to identify all their needs during the initial needs assessment to ensure they receive all available supports to help them succeed as soon as possible
• Using job developers to engage employers 

2.5 Self-Employment

2.5.1 Program description

Self-Employment helps individuals create jobs for themselves by starting a business or otherwise becoming self-employed. It provides financial assistance and business planning advice to EI eligible participants to help them start their own business. This financial assistance is intended to cover personal living expenses and other expenses during the initial stages of the business.

2.5.2 Profile of participants

As shown in Table E1 in Appendix E, active claimants who began Self-Employment participation in 2002 to 2005 and 2006 to 2008 were mainly males (64% and 54% respectively). Over one third (37% and 35% respectively) of participants in both cohorts were between 35 and 44 years of age while 33% of the 2002 to 2005 participants were between 25 to 34 years old and 33% of the 2006 to 2008 participants were 45 years of age and older. Participants in both cohorts most frequently had occupations requiring college or apprenticeship training (40% and 42% respectively) in the last job they held prior to participation. Participants in the 2002 to 2005 cohort had higher earnings in the year before participation compared to the 2006 to 2008 cohort ($24,674 and $21,433 respectively).

The majority of former claimants who participated in Self-Employment in 2002 to 2005 were male (54%) and in 2006 to 2008 participants were mostly female (58%). Former claimants who participated in the program in 2002 to 2005 were between 25 and 44 years old (68%) and former participants in 2006 to 2008 were 35 years of age and older (69%). Former claimants who participated in Self-Employment in 2002 to 2005 and in 2006 to 2008 most frequently had occupations requiring college or apprenticeship training (36% each). Participants in the 2002 to 2005 cohort had higher earnings in the year before participation compared to the 2006 to 2008 cohort ($11,110 and $10,157 respectively).

2.5.3 Incremental impacts

Like other EBSMs, incremental impacts were estimated for Self-Employment participants in the 2002 to 2005 and 2006 to 2008 periods. Results showed large decreases in employment/self-employment earnings and decreases in the incidence of employment. As well, compared to similar non-participants, participants decreased their use of EI and social assistance and reduced their dependence on government income support.

Detailed estimates are presented in Tables E2 and E3 in Appendix E. However, they are not discussed in the report since they may not provide an accurate depiction of the financial well-being of participants in the post-program period. Impacts were examined using individual earnings reported in the T1 and T4 taxation files from Canada Revenue Agency, and measured relative to active claimants who did not participate in Self-Employment and may have been in any employment/unemployment situation following participation (for example, unemployed, paid employee or self-employed).

According to a study from Statistics Canada, self-employed individuals in Canada have a lower average annual income than paid employees ($46,200 versus $52,400 in 2009), but the average net worth of their households is 2.7 times greater than that of the paid employee households, which indicates that some self-employed individuals may leave funds within their business for reinvestment purposes^{Footnote 11}. Overall, this suggests that looking at individual earnings alone, without taking the net worth into consideration, may not provide a fair assessment of how well Self-Employment participants are doing financially after participation.

As well, currently, little is known about the design and delivery of this program. In particular, there is a lack of understanding around the role played by this program in helping future entrepreneurs to implement viable business plans and to develop their entrepreneurship skills. Overall, it is not clear whether participant’s success in improving their labour market attachment through self-employment is more closely associated with their business idea and their entrepreneurship skills than the assistance provided under Self-Employment.

2.6 Job Creation Partnerships

2.6.1 Program description

Job Creation Partnerships projects provide participants with opportunities to gain work experience. Participants continue to receive their EI Part I benefits or receive an allocation while they are employed by a project funded under the program. Activities of the project help develop the community and the local economy.

2.6.2 Profile of participants

As shown in Table F1 in Appendix F, active claimants who started Job Creation Partnerships participation in 2002 to 2005 and 2006 to 2008 were mostly female (54% and 52% respectively) and between 25 and 34 years of age (35% and 31%). As well, thirty-one percent of active participants in 2006 to 2008 were 45 years of age and older. Participants in both the 2002 to 2005 and the 2006 to 2008 cohorts most frequently had jobs requiring secondary or occupational training in the last job they held before participation (33% each). Those who participated in the 2002 to 2005 period had higher employment earnings ($13,378) compared to participants from the 2006 to 2008 cohort ($11,772).

A little more than half (51% each) of the former claimants who started Job Creation Partnerships participation in 2002 to 2005 and 2006 to 2008 were male. Thirty-two percent of participants in the 2002 to 2005 period were between 25 and 34 years old. Participants in the 2006 to 2008 cohort were older compared to the earlier group with 30% between 25 and 34 years old and 35% being 45 years and over. Participants in both cohorts most frequently had occupations requiring secondary school or occupational training (36% and 39% respectively) prior to participation. Earnings for the 2002 to 2005 participants were slightly higher ($6,963) compared to participants from the 2006 to 2008 cohort ($6,159).

2.6.3 Incremental impacts

Active claimants

As shown in Table F2 in Appendix F, active claimants who started Job Creation Partnerships participation in 2002 to 2005 had incremental gains in earnings in all post-program years. As shown in Figure 5, statistically significant gains at the 95% level ranged between $2,551 and $3,162 annually. There were also gains in the incidence of employment ranging between 3.7 and 6.5 percentage points. Their use of EI decreased in the first two years following participation by $684 and $616 respectively. Decreases in the dependence on income support were observed in the first two years following participation (3.4 and 4.7 percentage points respectively). Estimates regarding the use of social assistance were not statistically significant.

Text description of Figure 5

Figure 5. Increased Earnings of Active and Former Job Creation Partnerships Participants Relative to Non-Participants

		Year 1 Post-Program	Year 2 Post-Program	Year 3 Post-Program	Year 4 Post-Program	Year 5 Post-Program
Gains in Earnings	Active Claimants	$1,295*	$3,162	$2,634	$2,333*	$2,551
	Former Claimants	$921*	$2,066	$2,346	$3,092	$2,887

*The estimates are not statistically significant at the 95% confidence level.

Estimates for active claimants who started Job Creation Partnerships participated in 2006 to 2008 were generally not statistically significant.

Overall, active claimants who participated in Job Creation Partnerships improved their labour market attachment through increases in earnings, incidence of employment and short-term decrease in the use of EI.

Former claimants

As shown in Figure 5, former claimants who started Job Creation Partnerships participation in 2002 to 2005 had incremental gains in earnings that varied between $2,066 and $3,092 in four of the five post-program years (see Table F3 in Appendix F). There were also gains in the incidence of employment ranging between 6.1 and 7.5 percentage points. Their use of EI benefits increased a total of $2,520 in the post-program period. Social assistance use decreased by a total of $1,178 in the post-program period. Dependence on income support also decreased in the first year following participation by 5.1 percentage points but was not statistically significant for the remainder of the post-program period.

Results for former claimants who participated in Job Creation Partnerships in the 2006 to 2008 period were not often statistically significant. However, results showed increases in the use of EI ranging between $523 and $531 and decreases in social assistance use ranging from $254 and $277 in the first two years following participation.

Overall, former claimants increased their use of EI following participation. This indicates the inability of some former claimants to maintain the employment secured in the short-term. For the 2002 to 2005 cohort of participants, it can also be argued that the increase in EI use is an indication of increase labour market attachment for this client group since they did experience increases in employment earnings and incidence of employment as well as a decrease in the use of social assistance. As a reminder, former claimants are participants for whom the EI benefit period ended up to three years pre-participation.

2.6.4 Cost-benefit results

As shown in Table F4 in Appendix F, the benefits of Job Creation Partnerships were $11,063 below the costs six years after participation from the society perspective. The benefits would need to be maintained over almost 12 years after the end of participation to match the costs. For former claimants, six years after program end, the benefits of Job Creation Partnerships were $9,034 below costs. The benefits would need to continue for 11 years after the end of participation to match the costs.

Overall, when interpreting cost-benefit results for Job Creation Partnerships, it should be acknowledged that program funding helps to develop the community and the local economy and none of those benefits were accounted for in the calculations as they are difficult to quantify.

2.7 Employment Assistance Services

2.7.1 Program description

Based on a document review and eleven key informant interviews completed in the summer of 2013

The goal of Employment Assistance Services is to help Nova Scotians find sustainable employment and offer a range of resources, supports and services to respond to the career and employment needs of individuals. It bridges the gap between employers seeking workplace skills and human resources to meet their business needs and individuals seeking employment.

Key Employment Assistance Services and resources in Nova Scotia included:

• Employment resource centres for unassisted job searches, career planning, and labour market information
• Client assessment and development of a return-to-work action plan
• Case management and employment counselling
• Employment group services in the areas of job preparation, job search, career and occupational decision-making
• Workshops on résumé writing, interview skills, job search, networking, professional language and presentation skills, and professionalism in the workplace
• Job finding club, job coaching and transitional employment support
• Work exploration for individuals with difficulties maintaining employment
• Physical, social, intellectual and/or psychological diagnostic services
• Employment-related life skills
• Marketing clients to potential employers and job development
• Short orientation and training sessions
• Work Activity Program providing training and job exposure in multiple industries to individuals who have weak labour market attachment

2.7.2 Program delivery

Based on a document review and eleven key informant interviews completed in the summer of 2013

In Nova Scotia, Employment Assistance Services are delivered by third party service providers. Employment Nova Scotia through Career Nova Scotia Centre Service agreements with third-party delivery organizations provide funding to businesses, organizations, municipalities, band/tribal councils, public health and educational institutions to conduct employment services.

Service providers use labour market information to support the provision of Employment Assistance Services. Among other things, labour market information can be used as part of the program application and may help participants make sound career choices. It can also be used in job search or career development group sessions, workshops and interview seminars.

2.7.3 Profile of Employment Assistance Services participants

As shown in Table G1 in Appendix G, active claimants who participated in Employment Assistance Services were mainly male (52% for participants in 2002 to 2005 and 51% for participants in 2006 to 2008). Fifty-nine percent and 54% of participants in both cohorts were aged between 25 and 44 years old while 27% and 32% in both cohorts were 45 years and older. As well, they mainly held jobs requiring secondary or occupational training before participation (38% in each cohort). Earnings were slightly lower for the 2002 to 2005 cohort ($17,108) compared to the 2006 to 2008 cohort ($18,304).

Slightly more than half of the former claimants who started Employment Assistance Services in both the 2002 to 2005 and 2006 to 2008 cohorts were male (52% and 51% respectively). Fifty-eight percent of the 2002 to 2005 participants and 55% of the 2006 to 2008 participants were between 25 and 44 years of age. Participants in both cohorts most frequently had jobs requiring secondary or occupational training prior to participation (37% each). Those who participated in 2002 to 2005 had lower earnings in the year before participation ($8,671versus $10,387).

2.7.4 Incremental impacts

As shown in Table G2 in Appendix G, active claimants who started Employment Assistance Services participation during the 2002 to 2005 period had increases in their employment earnings and incidence of employment. Incremental gains in earnings increased annually between $738 and $1,792 over the five-year post-program period. As shown in Figure 6, incidence of employment increased between 1.2 and 2.0 percentage points. Participants decreased the use of EI by an annual average that varied between $209 and $530 during the post-program period. While they increased their use of social assistance in the first post-program year only, dependence on income support decreased in three of the five post-program years between 1.3 and 1.7 percentage points annually.

Text description of Figure 6

Figure 6. Incidence of Employment for Active Claimant Participants in Employment Assistance Services

	Year 1 Post-Program	Year 2 Post-Program	Year 3 Post-Program	Year 4 Post-Program	Year 5 Post-Program
Percentage Points	1.0*	1.0*	1.2	1.9	2.0

*The estimates are not statistically significant at 95% confidence level.

The 2006 to 2008 participants also had increases in earnings and incidence of employment. Incremental gains in earnings averaged $1,346 and $1,653 in years two and three post-program period respectively. As well, incidence of employment increased between 1.2 and 1.9 percentage points annually. Participants’ use of EI benefits decreased between $414 and $705 annually after participation. The use of social assistance benefits increased $40 in the first year post-program but the level of dependence on income support decreased by 3.1, 2.7 and 2.3 percentage points over the three year post-program period.

Overall, active claimants improved their labour market attachment through increases in earnings and incidence of employment, and a decrease in the use of EI.

The results varied based on the sub-groups examined:

• Youth (under 30 years old) who started Employment Assistance Services participation in 2002 to 2005 had no statistically significant impacts on earnings and incidence of employment. The use of EI benefits decreased by a cumulative of $1,792 over the entire five-year post-program period. In two of the 5 post-program years, there was a decline in the level of dependence on income support by 1.9 and 2.5 percentage points.
• Older workers (over 55 years old) who started Employment Assistance Services participation in 2002 to 2005 had increases in earnings and incidence of employment in years 3, 4 and 5 post-program period. Incremental gains in earnings varied between $3,060 and $5,083 while the increase in the incidence of employed varied between 6.8 and 10.2 percentage points. Participants also decreased the use of EI by a cumulative of $2,504 over the entire five-year post-program period.
• Most estimates for long-tenured workers who started Employment Assistance Services participation in 2007 to 2009 were not statistically significant.

Earlier participation in Employment Assistance Services improves participants’ Labour Market outcomes

The study on the effects related to the timing of participation showed that incremental impacts on earnings and employment were larger for individuals who received Employment Assistance Services early during their EI claim compared to non-participants and to individuals who remained on EI a longer time before participating in the program (see Figure 7 below and Table G3 in Appendix G). Specifically, individuals who started their participation within four weeks after the start of their EI claim increased their earnings $6,836 over the post-program period and were the only group to return to employment 2.2 weeks earlier than the comparison group (see Table G4 in Appendix G). Impacts on the incidence of employment were not statistically significant.

Participants who started Employment Assistance Services between 5 and 8 weeks and those who started between 9 and 12 weeks after the start of their EI claim had increases in their earnings totalling $11,249 and $14,093 respectively over the post-program period. Participants in these two groups were the only groups to have statistically significant increases in the incidence of employment in one or two years of the five-year post-program period. While participants who started Employment Assistance Services participation between 9 and 12 weeks had the largest gains in earnings, they returned to work 0.9 week after the comparison group.

Text description of Figure 7

Figure 7. Cumulative Incremental Impacts on Earnings Related to the Timing of Participation in Employment Assistance Services

	1 to 4 weeks	5 to 8 weeks	9 to 12 weeks	2nd quarter	3rd quarter	4th quarter
Cumulative Gains in Earnings	$6,836	$11,249	$14,093	$6,605	$2,360*	$805*

*These estimates are not statistically significant at the 95% confidence level.

2.7.5 Cost-benefit results

Six years after participation, the benefits of Employment Assistance Services from the society perspective exceeded the cost by $2,059 (as shown in Table G5 in Appendix G). It took 4.9 years after the end of participation to recover the costs.

2.7.6 Challenges and lessons learned about the design and delivery of Employment Assistance Services

Based on a document review and eleven key informant interviews completed in the summer of 2013

Barriers to Employment faced by Employment Assistance Services participants in general

Key informants identified a number of barriers to labour market participation that EAS participants are facing. Examples include lack of essential skills, lack of employment opportunities and a large proportion of seasonal employment positions in Nova Scotia, lower wages and benefits compared to other locations in Canada, transportation issues (especially in rural areas), access to childcare, lack of work experience (especially for younger workers), and a social stigma towards hiring visible minorities and persons with disabilities.

Challenges experienced in delivering Employment Assistance Services

Key informants highlighted a number of challenges with delivering Employment Assistance Services:

• Awareness: Unemployed individuals are unaware of what services are available.
• Accessibility: Transportation is a barrier since it is not an eligible cost. Limited hours of operation of service providers can limit accessibility. Low levels of computer skills can create challenges for clients (especially for online applications).
• Program delivery: It was identified that organizations have limited budgets to provide workshops leading to certification. In addition, there is difficulty attracting and retaining qualified staff to service providers because of uncompetitive set maximum pay rates and short contracts contributing to staff turnover.
• Partnerships and relationships with other organizations: It was suggested that there is a need to improve the relationship with ESDC and Service Canada. As well, service providers suggested that communication appears to operate on a “need to know” basis where they are rarely informed about program changes until after their adoption.
• Service providers were unable to access additional funding to support a position to connect with employers despite the need to foster relationships with employers to develop local labour market information.

Key informants highlighted a number of lessons learned and best practices. Examples included:

• Communication: Bi-annual career resource manager meetings are valuable for discussing issues, sharing best practices and feedback. It is important to develop more connections between government and service providers. Monthly meetings between service providers that serve clients with disabilities are beneficial for sharing materials, best practices and lessons learned for streamlining services.
• Having two-year contracts with service providers was seen as beneficial because they save time and money.
• Operational: The career exploration officer that uses labour market information and helps clients make career decisions was identified as beneficial.
• Focus on clients: Offering a “one-stop” shop approach to service delivery and co-locating, when possible, with other relevant service providers were seen as beneficial. In addition, referrals for mental health and addictions were noted to have noticeably helped clients.
• Adapt programs to community needs and changes: There is a need to ensure that the program adapts to changes in the labour market (i.e. business closures), socio-demographic changes (that is, aging population and outmigration of younger workers) and the clients served (i.e. large proportion of participants with disabilities). It was suggested that rural employment agencies require additional funding and support to serve clients.

Lessons learned and best practices in assisting participants with multiple barriers

Lessons learned and best practices in assisting participants with multiple barriers included:

• Maintaining a client-centred approach to service delivery and having in-house staff expertise to provide clients with services more tailored to their needs.
• Having an on-duty career counsellor who is able to answer questions from walk-ins.
• Developing positive relationships with other employment agencies and enhancing partnerships with community organizations and employers.
• Focusing on long-term goals to help clients with multiple barriers.
• Conducting a detailed needs assessment upfront.
• Offering job coaching to motivate clients with multiple barriers and support their progress.
• Conducting employer engagement and outreach to address stigmas in hiring individuals with multiple barriers.

3. Comparison of key findings by program type

This section provides an overview of the key findings from the incremental impact analysis for Skills Development, START, Job Creation Partnerships and Employment Assistance Services for both active and former EI claimant participants who started participation in the 2002 to 2005 period.

Overall, incremental impacts demonstrate that LMDA programs and services are improving the labour market attachment of participants in Nova Scotia. As well, social benefits of participation exceeded the cost of investments for all interventions over time. Finally, providing Employment Assistance Services interventions earlier during an EI claim (first twelve weeks) produced larger impacts on earnings and employment and facilitated earlier return to work (especially during the first 4 weeks). This demonstrates the importance of targeting early participation of EI active claimants.

Program participants have a higher probability of being employed than comparison group members

As shown in Figure 8, active EI claimants who participated in Skills Development, START, Job Creation Partnerships and Employment Assistance Services had higher probability of being employed (that is, increased their incidence of employment) compared to similar non-participants. As well, former EI claimants who participated in Skills Development and had a higher probability of being employed compared to former EI claimants who received low intensity interventions under Employment Assistance Services.

Text description of Figure 8

Figure 8. Change in Probability of Being Employed in Participants Relative to Non-Participants

		Skills Development	START	Job Creation Partnerships	Employment Assistance Services
Percentage Points	Active Claimants	4.9	5.1	5.1	1.4
	Former Claimants	4.2	7.8	6.7

Increased earnings for participants compared to comparison group members

As shown in Figure 9, active EI claimants who participated in Skills Development, START and Employment Assistance Services increased their employment earnings compared to similar non-participants. As well, former EI claimants who participated in Skills Development, START and Job Creation Partnerships increased their employment earnings compared to former EI claimants who received Employment Assistance Services exclusively.

Text description of Figure 9

Figure 9. Increased Cumulative Earnings of Participants Compared to Non-Participants

	Skills Development	START	Job Creation Partnerships	Employment Assistance Services
Active Claimants	$23,728	$13,617	$11,976	$5,287
Former Claimants	$8,131	$11,841	$11,312

The use of EI is reduced for most active claimants. While EI use increased for former claimants who participated in Skills Development and Job Creation Partnerships, it can be argued that this reflects an increase in labour market attachment since the incremental impacts on employment earnings and incidence of employment are positive and given the decrease in the use of social assistance.

As shown in Figure 10, active EI claimants who participated in Skills Development, START and Employment Assistance Services decreased their use of EI compared to similar non-participants. In the case of Job Creation Partnerships, the decrease in EI use was observed in the first two years post-participation, however, the cumulative impact on the use of EI is not statistically significant. Active claimants who participated in Skills Development decreased their use of social assistance, while the estimates were not statistically significant for other programs.

Text description of Figure 10

Figure 10. Change in the Cumulative Use of Employment Insurance and Social Assistance for Active Claimants Relative to Non-Participants

	Skills Development	START	Job Creation Partnerships	Employment Assistance Services
Employment Insurance	-$1,910	-$2,207	-$933*	-$1,834
Social Assistance	-$606	-$240*	-$330*	$169*

* The estimates are not statistically significant at the 95% confidence level.

As shown in Figure 11, former claimants who participated in Skills Development and Job Creation Partnerships between 2002 and 2005 increased their use of EI following participation. This indicates the inability of some former claimants to maintain the employment secured in the short-term. It can also be argued that the increase in EI use is an indication of increase labour market attachment for this client group since they did experience increases in employment earnings and incidence of employment as well as a decrease in the use of social assistance. As a reminder, former claimants are participants for whom the EI benefit period ended up to three years pre-participation.

Text description of Figure 11

Figure 11. Change in Cumulative Use of Employment Insurance and Social Assistance for Former Claimants Relative to Non-Participants

	Skills Development	START	Job Creation Partnerships
Employment Insurance	$1,773	$728*	$2,520
Social Assistance	-$1,010	-$1,061	-$1,178

* The estimate is not statistically significant at the 95% confidence level.

Social benefits of participation exceeded costs of investments for all interventions.

As shown in Table 4, social benefits to participation exceeded investment costs in a period ranging between less than two and 12 years after program participation.

4. Conclusions

Evaluation evidence presented and discussed in this report demonstrated that programs and services designed and delivered by Nova Scotia under the LMDA are generally helping participants improve their labour market experience after participation. As such, evaluation evidence suggests that LMDA funded programming contributes to achieving government commitments and the strategic priorities of the Department of Labour and Advanced Education to educate, invest and grow the labour market.

Overall, incremental impacts demonstrate that LMDA programs and services are improving the labour market attachment of participants in Nova Scotia. As well, the social benefits of participation exceeded the cost of investments for most interventions over time. Finally, providing Employment Assistance Services interventions earlier during an EI claim (first twelve weeks) produced larger impacts on earnings and employment and facilitated an earlier return to work (especially during the first four weeks). This demonstrated the importance of targeting early participation of EI active claimants.

Key informant interviews with service providers and program managers as well as the reviewed documents and the questionnaires completed by Nova Scotia representatives revealed specific challenges and lessons about program design and delivery:

Skills Development

The application process for Skills Development aims to ensure that prospective participants are choosing training that will meet the labour market demand.

According to service providers and managers interviewed as part of the evaluation, the main challenges related to Skills Development design and delivery included:

• An administrative burden associated with the complexity of the financial aspects of Skills Development application (applying for student loan, family versus individual income)
• Lack of support to address barriers such as learning disabilities and mental health issues
• Level of financial support is insufficient because tuition fees are not covered 100% and the amount of living allowance is low
• High number of caseloads per caseworker

Skills Development-Apprentices

Existing literature showed that there is a fairly high non-completion rate among apprentices. It was not possible with the available data to generate a reliable estimation of the completion rate of Skills Development-Apprentices participants in Nova Scotia. According to key informants from Nova Scotia and other provinces and territories, the drop-out from the apprenticeship process was due to factors such as:

• Financial difficulties during training
• Apprentices leaving the trade
• Employers unwilling to release their apprentices for training
• Lack of training opportunities in local communities and/or low demand for certain trades
• Lack of or low level of essential skills

START

While evaluation results have demonstrated the effectiveness of START, the number of new interventions decreased between 2005 to 2006 and 2011 to 2012. According to key informants, employers may not be inclined to use the START program because:

• They are unfamiliar with the on-line process and find it complex
• They do not have time to spend learning the application process and maybe unable to reach someone for assistance with the on-line process when they run into difficulties
• They lack of awareness about the program
• They have negative perception about potential START participants

Increased awareness and referral as well as enhanced flexibility are credited with the recent increase in the number of participants in START since 2012 to 2013.

Employment Assistance Services

Key informants interviewed in the evaluation confirmed the need to have labour market information to support the delivery of Employment Assistance Services. They also reported the need for service providers to foster relationships with employers to develop local labour market information.

Challenges related to Employment Assistance Services included:

• Lack of awareness about available services
• Low levels of computer skills can create challenges for clients with online applications
• Limited hours of operation for some service providers can limit accessibility
• Service providers’ difficulty in attracting and retaining qualified staff because of uncompetitive set of maximum pay rates and short contracts

5. Recommendations

Recommendations emerging from the evaluation findings presented in this report are as follows:

The study on the timing of Employment Assistance Services participation showed that receiving assistance early after starting an EI claim can lead to better labour market impacts. However, key informants reported a lack of awareness about programs and services.

Recommendation 1: Consideration should be given to providing Nova Scotia with timely access to data on new EI recipients for supporting targeting and increasing awareness.

Key informants reported that mental and physical disabilities, learning disabilities and lack of essential skills or education were common barriers to accessing and completing training.

Recommendation 2: Consideration should be given to remove barriers to accessing and completing training such as literacy/essential skills training and learning disability assessments. The measures will help individuals with multiple barriers to prepare for vocational training and to reintegrate the labour market. The measures should be reported separately from other Skills Development interventions given their unique objectives.

Key informants interviewed in the evaluation confirmed the necessity of having labour market information to support program delivery. They, however, pointed to the difficulty of accessing labour market information at the regional or local level.

Recommendation 3: Consideration should be given about enhancing the capacity of service providers to access or produce, when needed, relevant labour market information.

The evaluation was not able to produce a conclusive assessment of Self-Employment effectiveness and efficiency since the data used to assess impacts on earnings may not be the best source of information available to reflect the financial wellbeing of the participants. As well, little is known about the design and delivery of this program. Overall, it is not clear whether the participant’s success in improving their labour market attachment through self-employment is more closely associated with their business idea and their entrepreneurship skills than the assistance provided under Self-Employment.

Recommendation 4: Consideration should be given to examine in more detail the design and delivery of Self-Employment and whether the performance indicators for this program are appropriate.

Job Creation Partnerships was found to be particularly effective at improving earnings and incidence of employment. However, the evaluation has not yet examined the design and delivery of this program. Therefore, a lot remains unknown about how this program operates and the factors that contribute to its effectiveness.

Recommendation 5: Future evaluation work should examine the design and delivery of the Job Creation Partnerships to better understand how this program operates in Nova Scotia.

Overall, the LMDA evaluation was able to produce a sound assessment of EBSM effectiveness and efficiency because the team had access to rich data on EI claimants, EBSM participation data and Canada Revenue Agency taxation files. However, some data gaps limited the evaluation’s ability to assess how EBSMs operate.

Recommendation 6: Improvements in the data collection is recommended to address key program and policy questions of interest to the federal and provincial/territorial governments. Specifically:

• Collect data on whether participants are members of designated groups including Indigenous peoples, persons with disabilities and recent immigrants.
• Collect data on the type of training funded under Skills Development and the type of assistance provided under Employment Assistance Services. Nova Scotia, ESDC and other provinces and territories should work together to define common categories for both EBSMs.

References

Canada Employment Insurance Commission. Monitoring and Assessment Report. Employment and Social Development Canada, 2004 to 2015.

Coe, Patrick. “Apprenticeship program requirements and apprenticeship completion rates in Canada.” Journal of Vocational Education and Training. 2013. 65(4): 575 to 605.

Calmfors, Lars, and Per Skedinger. 1995. Does Active Labour Market Policy Increase Employment? – Theoretical Considerations and Some Empirical Evidence from Sweden. The Industrial Institute for Economic and Social Research, No. 429, Stockholm.

Employment and Social Development Canada. Analysis of Employment Benefits and Support Measures (EBSM) Profile, Outcomes and Incremental Impacts for the EI Claimants Category “Long-Tenured Workers” in Nova Scotia: Technical Report. Evaluation Directorate. January 28, 2016

Employment and Social Development Canada. Analysis of National Employment Benefits and Support Measures (EBSM) Profile, Outcomes and Incremental Impacts for 2006 to 2008 Participants: Nova Scotia Technical Report. Evaluation Directorate. February 25, 2015

Employment and Social Development Canada. Cost-Benefit Analysis of Employment Benefits and Support Measures Delivered in Nova Scotia. Evaluation Directorate. May 26, 2016

Employment and Social Development Canada. Effects of the Timing of Participation in Employment Assistance Services in Nova Scotia. Evaluation Directorate. December 19, 2014

Employment and Social Development Canada. 2016 to 2019 Integrated Business Plan. Author: May 2016 Employment and Social Development Canada. National Occupational Classification, 2006. Gatineau: Employment and Social Development Canada. Available at: National Occupational Classification - Resources & Tools

Employment and Social Development Canada. Profile, Outcomes and Net Impacts of Employment Benefits and Support Measures Participants in Nova Scotia. Evaluation Directorate. April 2014.

Employment and Social Development Canada. Study on Employment Assistance Services Nova Scotia Technical Report. Evaluation Directorate. September 23, 2014

Employment and Social Development Canada. Study on Skills Development – Apprenticeship (SD−A) in Nova Scotia: Technical Report. Evaluation Directorate. May 12, 2016

Employment and Social Development Canada. Study on Skills Development (SD) Employment Benefit Program in Nova Scotia: Technical Report. Evaluation Directorate. April 22, 2016

Employment and Social Development Canada. Study on START Program in Nova Scotia: Technical Report. Evaluation Directorate. March 15, 2016

Government of Nova Scotia. Second Session of the 62nd General Assembly: Speech from the Throne. Retrieved from Throne Speech / Proceedings / The Nova Scotia Legislature on August 4, 2016.

Government of Nova Scotia. Budget 2016 to 2017: Working Together for a Stronger Nova Scotia. Department of Finance and Treasury Board. April 9, 2015

Group ATN Consulting Inc. START Program Evaluation Report. Nova Scotia. 2014

Human Resources Development Canada, Formative Evaluation of Employment Benefits and Support Measures Under the Canada-Nova Scotia Labour Market Development Agreement, December 1999.

International Labour Organization (accessed May 19)

LaRochelle-Côté, Sébastien and Sharanjit Uppal, "The Financial Well-Being of the Self-Employed," Perspectives on Labour and Income, vol. 23, no. 4, Winter 2011

Government of Nova Scotia. Connecting Nova Scotians to Higher-Value Jobs, Three-Year Strategic Framework 2013 to 2016. Department of Labour and Advanced Education. 2016

O’Leary, Christopher, Randall Eberts and Kevin Holenbeck. 2012. What Works for Whom in Public Employment Policies? Review of Literature on Effectiveness of Active Labour Market Policies. W.E. Upjohn Institute for Employment Research.

Organization for Economic Co-operation and Development. 1996. The OECD Job Strategy Pushing Ahead with Strategy. OECD, Paris

Red Seal. 2014. Apprenticeship Completion, Certification and Outcomes. Ottawa: Red Seal

Statistics Canada. Coverage and eligibility of the unemployed for Employment Insurance benefits, by province. The Daily. 2009 to 2014

Statistics Canada. Employment Insurance Coverage Survey 2015-11-23 on July 11, 2016

Statistics Canada. Table 282-0002 - Labour force survey estimates (LFS), by sex and detailed age group, annual, CANSIM (database). (Accessed: July 11, 2016)

Statistics Canada. North American Industry Classification System 2012. Ottawa: Statistics Canada.

Acronyms

EBSM

Employment Benefits and Support Measures

EI

Employment Insurance

ESDC

Employment and Social Development Canada

LMDA

Labour Market Development Agreements

P/T

Provinces and Territories

Appendix A: Methodology

Qualitative Data

Qualitative data reported in the Skills Development (SD), SD-Apprentices (SD-A), START and Employment Assistance Services (EAS) studies were collected from key informant interviews with managers and service providers and a document/ literature review. As well, Nova Scotia government representatives completed questionnaires for the SD, SD-A and START studies. Table A1 provides the number of key informants interviewed.

Key informant interviews for the EAS study were conducted in 2013 while those for the SD, SD-A and START studies were conducted in 2015.

Quantitative methods

All quantitative analyses were conducted using linked administrative data from EI Part I (EI claim), EI Part II (EBSM participation data) and T1 and T4 taxation files on up to 100% of participants in Nova Scotia.

Incremental impacts

The incremental impact analysis compared the labour market experience of participants before and after their participation with that of a comparison group. The goal was to determine the direct effect of program participation on key labour market indicators (see Figure 1 in the introduction section).

For active claimants, incremental impacts were measured relative to a comparison group of active claimants who could have participated in the EBSMs but did not. Former claimants can be underemployed and unable to requalify for EI, out of the labour force for various reasons or on social assistance. Based on previous evaluation methodologies, on expert advice and given the difficulty in generating a suitable comparison for former claimants using administrative data alone, the comparison group for former claimants was created using individuals who participated in low-intensity Employment Assistance Services only during the reference period. This is a conservative approach given the fact that participation in Employment Assistance Services can lead to limited effects on labour market outcomes.

Participants and non-participants were matched based on a wide array of variables including age, sex, location, skill level required by the last major occupation held prior to participation, reason for separation from employment, industry in which they were previously employed as well as employment earnings and use of EI and social assistance for each of the five years before participation.

All analyses were conducted using a unit of analysis called the Action Plan Equivalent, which combines all EBSMs given to an individual within no more than six months of each other. For reporting purposes, incremental impacts were attributed to the longest intervention of the Action Plan Equivalent when SD, START, Job Creation Partnerships or Self-Employment was the longest interventions. Impacts for EAS were calculated for Action Plan Equivalents that contained only EAS with no Employment Benefits.

The incremental impact estimates were produced using non-experimental methods, namely propensity score matching, using the Kernel Matching method, along with Difference-in-Differences method to estimate program impacts. Alternative matching techniques (that is, Nearest Neighbour and Inverse Propensity Weighting) were also used for validation purposes.

Incremental impacts were measured for the following indicators:

• Employment/self-employment earnings represent the total earnings an individual had from paid employment and/or self-employment. (This information is available by calendar year and was obtained from T1 and T4 tax return records.)
• Incidence of employment/self-employment represents the incidence of having earnings from employment and/or self-employment.
• Amount of EI benefits received represent the average amount of EI benefits received.
• Weeks in receipt of EI benefits represent the average number of weeks during which EI benefits were received.
• Social assistance benefits represent the average amount of Social Assistance (SA) benefits received. (This information is available by calendar year and is obtained from T1 tax return records.)
• Dependence on income support represents the ratio of participant’s income that came from EI and SA benefits (that is, EI benefits + SA benefits / (EI benefits + SA benefits + earnings from employment/self/employment)).

Incremental impacts were estimated for different cohorts of participants:

• All active and all former claimants as well as youth (under 30 years old) and older workers (55 years old and over) who started their EBSM participation between April 1, 2002 and March 31, 2005.
• All active and all former claimants who stated their EBSM participation between January 1, 2007 and March 31, 2008.
• Active and former claimants who were long-tenured workers and who started their EBSM participation between January 1, 2007 and December 31, 2009. Long-tenured workers covered in this study are individuals who have established an EI regular or fishing benefits claimants and who had paid at least 30% of the annual maximum employee EI premiums in seven of the ten years preceding their EI claim and who had collected 35 or fewer weeks of EI regular or fishing benefits in the five years preceding their claim. This definition is similar to the EI claimant category long-tenured workers introduced under Connecting Canadians with Available Jobs.

Cost benefit analysis

The cost-benefit analysis compared how much it cost for individuals to participate in the programs and how much it costs the government to deliver these programs with the benefits both the participants and the government drew from these programs. The analysis was carried out from the society perspective which combines the costs and the benefits for both the participants and the government.

Costs and benefits included in the calculations were as follows:

• Program cost included the administration cost and the direct cost of the EBSMs. The cost for each EBSM was calculated at the Action Plan Equivalent level. The costs were determined based on the average composition of the APE.
• The Marginal Social Cost of Public Funds represented the loss incurred by society when raising additional revenues such as taxes to fund government spending. The value was estimated as 20% of the program cost, sales taxes, income taxes, impacts on EI and impacts on SA paid or collected by the government.
• Employment earnings consisted of incremental impacts on participants’ earnings during and after participation. The calculation accounts for the participant’s forgone earnings during participation (that is , opportunity cost). These are based on incremental impacts for the 2002-2005 participants.
• Fringe benefits included benefits such as employer-paid health and life insurance as well as pension contributions. The rate used to calculate the fringe benefits was 15% of the incremental impact on earnings.

The program effects on EI and SA use and the sale and income tax revenues were not included in the calculations since these costs and benefits cancel each other out from the social perspective by definition. For example, while EI and SA are benefits received by participants, they represent a cost for the government. However, as indicated above, these effects are accounted for in the calculation of the Marginal Social Cost of Public Funds.

When producing the results, to bring all costs and benefits to a common base and to account for inflation and interest on foregone government investment, the estimates for the second year of participation and up to the sixth year post-program were discounted by 5% per year. As well, when the benefits were still lower than the costs six years after program end, the payback period was calculated by assuming that the average benefit or cost measured over the fifth and six year post-program would persist over time (discounted at a 5% annual rate).

Strengths and limitations from the studies

Overall, the number of key informants interviewed was relatively small in some studies. The key informants’ responses were representative of their own experience and their own region but it is unclear if they were fully representative of the entire province and territory.

The matching process led to the creation of comparison groups closely matched to the LMDA participants in terms of their background characteristics. Results obtained with Kernel Matching were validated with the use of two other techniques (that is, Inverse Propensity Weighting and Nearest Neighbour), increasing the level of confidence in the results. However, readers should be aware that incremental impacts may be affected by factors not captured by the matching process. For example, the motivation to seek employment was not directly measured except to the extent it was captured in prior income and labour market attachment patterns.

Readers should also keep in mind that it is not possible to compare the results obtained for each claimant type since the results for active claimants represent the effects of the EBSMs relative to non-participation while the results for former claimants represents the Employment Benefits relative to a limited treatment (that is, EAS).

The definition of long-tenured workers differs from the definition used in the literature as it does not consider the number of years the worker remained employed with the same employer.

The cost-benefit analysis was limited in the sense that it only took into account the quantifiable benefits and costs that were directly linked to EBSM delivery and participation and that could be estimated using available administrative data and the EI Monitoring and Assessment Report. The analysis did not capture “intangible”, non-pecuniary and indirect benefits. It did not consider the multiplier effect that improving participant’s income may have on the economy and did not account for the effect of EI Part II investment on sustaining a service delivery infrastructure and creating jobs among the governmental program service providers. As well, this analysis did not consider the displacement effect where participants may take away jobs that would otherwise be filled by other unemployed individuals. Finally, this analysis did not consider the possible effect of EBSMs on increasing skill prices.

Appendix B: Detailed results Skills Development

Proportions may not add up to 100% due to rounding

¹Skill level corresponds to the type and/or amount of training or education typically required to work in the last occupation participants had before opening the last EI claim they had before participating in EBSMs:

-Managerial: Management occupations-University: Occupations usually requiring university education (for example, University degree at the bachelor's, master's or doctorate level)
-College or apprenticeship training: Occupations usually requiring college or vocational education or apprenticeship training such as 2 to 3 years of post-secondary education at a community college, institute of technology or CEGEP or 2 to 5 years of apprenticeship training or 3 to 4 years of secondary school and more than 2 years of on-the-job training, specialized training courses or specific work experience and/or occupations with supervisory responsibilities and occupations with significant health and safety responsibilities, such as firefighters, police officers and registered nursing assistants.
-Secondary or occupational training: Occupations usually requiring secondary school and/or occupation-specific training such as one to four years of secondary school education or up to 2 years of on-the-job training specialized training courses or specific work experience.
-On-the-job training: On-the-job training is usually provided for occupations (for example, short work demonstration or on-the-job training or no formal educational requirements).

²The average was calculated including participants who reported $0 earnings during that year. Earnings for 2013 to 2014 participants have been adjusted according to the Consumer Price Index, published by Statistics Canada, to the 2002 base year.

Significance level *** 1%; ** 5%; * 10%
EI: Employment Insurance. SA: Social Assistance

Significance level *** 1%; ** 5%; * 10%
EI: Employment Insurance. SA: Social Assistance

Appendix C: Detailed results Skills Development-Apprentices

Proportions may not add up to 100% due to rounding

¹Skill level corresponds to the type and/or amount of training or education typically required to work in the last occupation participants had before opening the last EI claim they had before participating in EBSMs:

-Managerial: Management occupations-University: Occupations usually requiring university education (for example, University degree at the bachelor's, master's or doctorate level)
-College or apprenticeship training: Occupations usually requiring college or vocational education or apprenticeship training such as 2 to 3 years of post-secondary education at a community college, institute of technology or CEGEP or 2 to 5 years of apprenticeship training or 3 to 4 years of secondary school and more than 2 years of on-the-job training, specialized training courses or specific work experience and/or occupations with supervisory responsibilities and occupations with significant health and safety responsibilities, such as firefighters, police officers and registered nursing assistants.
-Secondary or occupational training: Occupations usually requiring secondary school and/or occupation-specific training such as one to four years of secondary school education or up to 2 years of on-the-job training specialized training courses or specific work experience.
-On-the-job training: On-the-job training is usually provided for occupations (for example, short work demonstration or on-the-job training or no formal educational requirements).

²The average was calculated including participants who reported $0 earnings during that year. Earnings for 2013 to 2014 participants have been adjusted according to the Consumer Price Index, published by Statistics Canada, to the 2002 base year.

¹Earnings outcomes excluding individuals who reported no earnings in a given year.
EI: Employment Insurance. SA: Social Assistance

Appendix D: Detailed results for START

Proportions may not add up to 100% due to rounding

¹Skill level corresponds to the type and/or amount of training or education typically required to work in the last occupation participants had before opening the last EI claim they had before participating in EBSMs:

-Managerial: Management occupations-University: Occupations usually requiring university education (for example, University degree at the bachelor's, master's or doctorate level)
-College or apprenticeship training: Occupations usually requiring college or vocational education or apprenticeship training such as 2 to 3 years of post-secondary education at a community college, institute of technology or CEGEP or 2 to 5 years of apprenticeship training or 3 to 4 years of secondary school and more than 2 years of on-the-job training, specialized training courses or specific work experience and/or occupations with supervisory responsibilities and occupations with significant health and safety responsibilities, such as firefighters, police officers and registered nursing assistants.
-Secondary or occupational training: Occupations usually requiring secondary school and/or occupation-specific training such as one to four years of secondary school education or up to 2 years of on-the-job training specialized training courses or specific work experience.
-On-the-job training: On-the-job training is usually provided for occupations (for example, short work demonstration or on-the-job training or no formal educational requirements).

²The average was calculated including participants who reported $0 earnings during that year. Earnings for 2006 to 2008 participants have been adjusted by the Consumer Price Index published by Statistics Canada, using 2002 as the base year.

Significance level *** 1%; ** 5%; * 10%
EI: Employment Insurance. SA: Social Assistance

Significance level *** 1%; ** 5%; * 10%
EI: Employment Insurance. SA: Social Assistance

Appendix E: Detailed results self-employment

Proportions may not add up to 100% due to rounding

¹Skill level corresponds to the type and/or amount of training or education typically required to work in the last occupation participants had before opening the last EI claim they had before participating in EBSMs:

-Managerial: Management occupations-University: Occupations usually requiring university education (for example, University degree at the bachelor's, master's or doctorate level)
-College or apprenticeship training: Occupations usually requiring college or vocational education or apprenticeship training such as 2 to 3 years of post-secondary education at a community college, institute of technology or CEGEP or 2 to 5 years of apprenticeship training or 3 to 4 years of secondary school and more than 2 years of on-the-job training, specialized training courses or specific work experience and/or occupations with supervisory responsibilities and occupations with significant health and safety responsibilities, such as firefighters, police officers and registered nursing assistants.
-Secondary or occupational training: Occupations usually requiring secondary school and/or occupation-specific training such as one to four years of secondary school education or up to 2 years of on-the-job training specialized training courses or specific work experience.
-On-the-job training: On-the-job training is usually provided for occupations (for example, short work demonstration or on-the-job training or no formal educational requirements).

²The average was calculated including participants who reported $0 earnings during that year. Earnings for 2013 to 2014 participants have been adjusted according to the Consumer Price Index, published by Statistics Canada, to the 2002 base year.

Significance level *** 1%; ** 5%; * 10%
EI: Employment Insurance. SA: Social Assistance

Significance level *** 1%; ** 5%; * 10%
EI: Employment Insurance. SA: Social Assistance

Appendix F: Detailed results job creation partnerships

Proportions may not add up to 100% due to rounding

¹Skill level corresponds to the type and/or amount of training or education typically required to work in the last occupation participants had before opening the last EI claim they had before participating in EBSMs:

-Managerial: Management occupations-University: Occupations usually requiring university education (for example, University degree at the bachelor's, master's or doctorate level)
-College or apprenticeship training: Occupations usually requiring college or vocational education or apprenticeship training such as 2 to 3 years of post-secondary education at a community college, institute of technology or CEGEP or 2 to 5 years of apprenticeship training or 3 to 4 years of secondary school and more than 2 years of on-the-job training, specialized training courses or specific work experience and/or occupations with supervisory responsibilities and occupations with significant health and safety responsibilities, such as firefighters, police officers and registered nursing assistants.
-Secondary or occupational training: Occupations usually requiring secondary school and/or occupation-specific training such as one to four years of secondary school education or up to 2 years of on-the-job training specialized training courses or specific work experience.
-On-the-job training: On-the-job training is usually provided for occupations (for example, short work demonstration or on-the-job training or no formal educational requirements).

²The average was calculated including participants who reported $0 earnings during that year. Earnings for 2013−2014 participants have been adjusted according to the Consumer Price Index, published by Statistics Canada, to the 2002 base year.

Significance level *** 1%; ** 5%; * 10%
EI: Employment Insurance. SA: Social Assistance

Significance level *** 1%; ** 5%; * 10%
EI: Employment Insurance. SA: Social Assistance

Appendix G: Detailed results Employment Assistance Services

Proportions may not add up to 100% due to rounding

¹Skill level corresponds to the type and/or amount of training or education typically required to work in the last occupation participants had before opening the last EI claim they had before participating in EBSMs:

-Managerial: Management occupations-University: Occupations usually requiring university education (for example, University degree at the bachelor's, master's or doctorate level)
-College or apprenticeship training: Occupations usually requiring college or vocational education or apprenticeship training such as 2 to 3 years of post-secondary education at a community college, institute of technology or CEGEP or 2 to 5 years of apprenticeship training or 3 to 4 years of secondary school and more than 2 years of on-the-job training, specialized training courses or specific work experience and/or occupations with supervisory responsibilities and occupations with significant health and safety responsibilities, such as firefighters, police officers and registered nursing assistants.
-Secondary or occupational training: Occupations usually requiring secondary school and/or occupation-specific training such as one to four years of secondary school education or up to 2 years of on-the-job training specialized training courses or specific work experience.
-On-the-job training: On-the-job training is usually provided for occupations (for example, short work demonstration or on-the-job training or no formal educational requirements).

²Average earnings for all individuals included in the study. The average was calculated including participants who reported $0 earnings during that year. Earnings for 2006 to 2008 participants have been adjusted by the Consumer Price Index published by Statistics Canada, using 2002 as the base year.

Significance level *** 1%; ** 5%; * 10%
EI: Employment Insurance. SA: Social Assistance

* Significant at 10%; ** significant at 5%; *** significant at 1%
n= refers to the number of participants. It corresponds to 100% of participants. pp= percentage points
Note: for the estimations we have selected a 50% random sample among comparison group in each cohort due to their large number.

* Significant at 10%; ** significant at 5%; *** significant at 1% Note: The means of the standardized bias reduction after matching were calculated as suggested by Rosenbaum and Rubin (1985). The overall bias after matching lies between 3% and 15%, which is generally considered as acceptable in empirical research papers .

Appendix H: List of nine studies included in the synthesis report