2016 Evaluation of the Targeted Initiative for Older Workers

Introduction

This report presents the findings and conclusions of the 2016 Evaluation of the Targeted Initiative for Older Workers (TIOW) program. The main purpose of the evaluation is to inform policy work in 2016-2017 prior to the end of the current renewal period (April 2014 to March 2017). It principally examines two areas: 1) how the introduction of two new eligibility criteria in 2014 impacted provinces and territories (PTs) in program design and delivery; and 2) how the program aligns with older worker labour market programming best practices when assessed against the academic literature. It also looked at the program’s performance in terms of achieving outcomes of employability, paid employment and self-employment where data was available.

Program description

The program is a federal-provincial/territorial cost-shared initiative. It supports unemployed older workers, primarily between the ages 55 and 64, with their re-integration into employment. The program is offered in small communities of 250,000 or less experiencing high unemployment and/or significant downsizing and closures. As per Budget 2014, the program was renewed for a third consecutive time for a three-year period (April 2014 to March 2017) representing a federal investment of $75M. Since its renewal in 2014, the program’s terms and conditions were expanded to include two new community eligibility criteria: unfulfilled employer demand^{Footnote 1} and skills mismatches^{Footnote 2}.

The expected outcome of the initiative is to increase the employability of unemployed older workers and help them to re-integrate into employment. PTs design projects based on the parameters set out in bilateral agreements between the Federal Government and participating PTs. Projects are typically delivered by third-party service providers.

The terms and conditions specify that projects must provide employment assistance activities such as résumé writing, interview techniques, employment counselling and job search techniques. In addition, projects must also offer at least two employability improvement activities such as assessment activities, peer mentoring, skills training, wage subsidies, community based work experiences, preparation for self-employment, post project follow-up and mentoring. Projects involving activities other than work experience or subsidized employment must offer income support in the form of allowances to participants, where PT legislation allows. All projects must involve at least 25 hours per week of activity for participants. There is no minimum or maximum number of weeks of participation.

Context

A more rigorous evaluation of the program was completed in 2014. It employed a complex and costly non-experimental design and survey and concluded authoritatively on the program’s performance. Among other things, it found that:

• Program participants were more likely than non-program participants to find employment by 6 percentage points.
• No employment earnings differences were found between participants and the comparison group of older workers.

The 2016 Evaluation complements this earlier work. It is calibrated and uses an outcome based design; its scope and focus are defined by the needs of the program area and the Department, as set out in the TIOW Evaluation Strategy approved by the Departmental Evaluation Committee in November 2014.

The 2016 Evaluation uses an exploratory/investigative approach focusing on enhancing the collective knowledge base on effective labour market programming approaches for older workers and the new demand driven aspects of the program. The evaluation examined the new project eligibility criteria and sought to understand the best program design activities, delivery practices and lessons learned from programs similar to TIOW. It also looked at the program’s performance in terms of achieving outcomes of employability, paid employment and self-employment where data was available.

The data collection consisted of a review of program documents, administrative data captured by the program, key informant interviews and a review of national and international literature.

The original scope of this evaluation included all provinces and territories with the exception of Québec, which carries out its own evaluations, and Nunavut, as it did not participate in the program. This evaluation bases its reporting on the administrative data available between April 2014 and February 2016 from four PTs: Ontario, British Columbia, Nova Scotia and Yukon.^{Footnote 3} Administrative data is only available from four PTs because Information Sharing Agreements (ISAs) with other PTs were still being negotiated during the data-gathering phase of this evaluation.

Data quality issues also presented challenges in conducting an outcomes-based assessment. Incomplete participant information, for example, constrained the level of analysis that could be undertaken. Since the evidence does not reflect the unrepresented provinces, the findings and conclusions that follow present a partial story of the program’s overall performance.

Conclusions

A total of 287 projects (60 new projects and 227 renewed) were delivered across Canada (including Quebec) by Project Sponsors between April 2014 and March 2016. Approximately 83% (50 out of 60) of the new projects qualified for funding as a result of the new eligibility criteria allowing communities challenged by skills mismatches and/or an unfulfilled employer demand to participate in the initiative. Overall, the program is reaching its targeted clients in Ontario, British Columbia, Yukon and Nova Scotia where the majority of participants are in the primary age range (55 and 64 years of age).

Participant outcomes from British Columbia, Yukon and Nova Scotia suggest that the TIOW program in those PTs is progressing towards its intended outcome. Outcome findings indicate that overall, participants expressed positive satisfaction regarding the program, and that job seeking activities and employability activities were seen as helpful overall.

When compared to the academic literature, research showed that overall the TIOW program is effectively designed and delivered to assist older workers to re-integrate into employment. For example the program engages in the recruitment of participants which includes the identification, assessment and selection of participants. It also provides income supports tailored to the needs of older workers. It provides relevant employment assistance and employability improvement activities, and facilitates those activities through group-based approaches. The program is in effect closely modeled after similarly designed labour market programs for older workers that successfully produced higher rates of employment and employment earnings as well as job satisfaction.

In spite of the program having many best practices in place, there are a few areas where potential improvements could be made:

• Many of the program activities strengthened job seekers self-efficacy beliefs; however job goals were not consistently clarified nor set with participants. When goals are clearly defined, a focused search strategy generally produces more positive employment outcomes.
• The most successful programs for older workers provide some form of marketing of older workers to employers. Not all projects marketed participants to employers or established partnerships with potential employers. Linking participants to employers responds to their need for pre-screened, trained older workers.
• A high level of ongoing, positive peer support motivates participants which is positively co-related with job-search intensity. There was a lack of formal, ongoing support facilitated by projects following the end of participation in programming.
• Labour market programs for older workers should include the careful identification, screening and selection of its participants. As noted above, this is done to an extent. However, Project Sponsors do not consistently assess all potential participants; the processes and tools used for assessment vary from one jurisdiction to another; and some assessments are more robust than others.

Based on the findings and conclusions presented in this report, the evaluation proposes the following three recommendations:

Introduction

The Skills and Employment Branch (SEB) would like to thank the Evaluation Directorate and all those who participated in the 2016 Evaluation of the Targeted Initiative for Older Workers (TIOW). In particular, SEB acknowledges the contribution of provinces and territories (PTs), participants, including key informants, and program officials during the course of this Evaluation.

TIOW is cost shared between the federal and provincial/territorial governments and supports unemployed older workers (typically ages 55 to 64) living in small, vulnerable communities with their reintegration into employment. It is targeted to cities and towns with a population of 250,000 or less that are affected by high unemployment and significant downsizing or closures. As noted below, two additional community eligibility criteria were added in 2014: unfulfilled employer demand and skills mismatches.

Under TIOW, PTs are responsible for targeting specific communities and for designing and delivering projects which offer group-based programming and/or self-employment assistance. The federal government is responsible for establishing policy parameters, approving projects, overseeing overall implementation of agreements, managing allocations and conducting program evaluations.

TIOW was renewed in 2014 for an additional three years to support the Government’s commitment to making labour market programming more demand-driven. Accordingly, the program’s eligibility criteria were expanded to provide PTs with the added flexibility to implement TIOW projects in communities that are experiencing an unfulfilled employer demand or skills mismatches.

TIOW is intended to complement other types of labour market programming currently available such as, but not limited to, the Canada Job Fund, Labour Market Development Agreements and Labour Market Agreements for Persons with Disabilities in its orientation towards human capital development focusing on both employment assistance activities (such as résumé writing, interview techniques, counselling and job finding clubs, etc.) and employability supports (such as skills training, preparation for self-employment, basic skills upgrading work experience, direct marketing to employers etc.) for older workers.

In the Summer-Fall 2016 ESDC undertook a comprehensive review of its full suite of labour market transfer agreements (LMTAs) including broad-based consultations with PTs and stakeholders with the objective of ensuring that LMTAs are relevant, flexible and responsive to current and emerging labour market needs and priorities. This aligns with the Government of Canada’s commitment to rationalize and expand agreements that support skills training. Findings from this review supported the need for future labour market agreements to:

• reflect to the needs of individuals, workers, employers and under-represented groups;
• build on strong evidence for relevant performance measurement to better inform and serve Canadians and help them achieve meaningful employment outcomes; and
• foster capacity to develop innovative program approaches and the sharing of best practices within the skills training agreements.

ESDC is currently exploring policy options for moving forward that have been informed by the review. Findings from this evaluation will be particularly relevant in helping to identify best practices in labour market programming designed to reach those clients most in need.

As the population ages and older Canadians represent a larger share of the workforce, it is important to encourage their retention and re-entry in the labour market to support economic growth.

SEB agrees with the evaluation findings and provides the following Management Response.

Key findings

While the Evaluation looked at the program’s performance in terms of achieving outcomes where data was available, its main focus was to examine two areas:

1. how the introduction of two new eligibility criteria in 2014 impacted program delivery by PTs and
2. how the program aligns with older worker best practices when assessed against the academic literature.

The Evaluation found that 83% of new TIOW projects approved between April 1, 2014 and March 31, 2016 were designed to support communities meeting the new eligibility criteria. This suggests that these added flexibilities addressed local labour market needs. The Evaluation also indicated that TIOW programming uses many of the best practices identified in the literature and identified some gaps that could be addressed in the future when designing labour market programming for older workers. Evidence shows that the program is reaching a vulnerable population in need of employment services. In addition, the Evaluation found that participants were satisfied with the job seeking and employability activities they received through the program and that TIOW effectively assists unemployed older workers improve their employability and transition into employment. While data limitations present challenges for analysis, this Evaluation nonetheless provides valuable insights into the program’s effectiveness.

Recommendations and follow-up actions

The evaluation outlined three recommendations:

3.1 The 2014 TIOW summative evaluation

The TIOW Summative Evaluation conducted in 2014 employed a complex and costly non-experimental design and concluded authoritatively on the program’s performance. The evaluation methodology consisted of a survey of former program participants, an incremental impact analysis using a limited-treatment comparison group, a benchmark study to examine the extent of job loss of older Canadians living in the same geographic areas as the program participants, key informant interviews with employers, project representatives, ESDC program officials and PT representatives and a review of key documents.

The 2014 Evaluation concluded that 75% of survey respondents found paid employment following their participation in the program and the majority of former program survey respondents believed that their program participation improved their employability. Many factors may have contributed to this positive outcome. The incremental impact analysis was used to determine how much of those outcomes can be attributed to the program. It concluded that:

• Program participants were more likely than non-program participants to find employment by 6 percentage points.
• No employment earnings differences were found between program participants and the comparison group of older workers.

3.2 The 2016 TIOW evaluation

The objective of the 2016 Evaluation was to inform policy work in 2016-2017 prior to the end of the current renewal period on March 31, 2017. It was therefore calibrated, rather than employing the more comprehensive methodology used in the 2014 Summative Evaluation of the program. The Evaluation Directorate and the Employment Program Policy and Design Directorate agreed to a strategy that would enhance the collective knowledge base on effective labour market programming approaches for older workers.^{Footnote 7}

More specifically, it focused on examining best practices in program design and exploring the implementation of the new program eligibility criteria. Where data was available it also assessed the program’s performance in terms of achieving outcomes such as employability, paid employment and self-employment.

The data collection consisted of four methodologies that included:

• A review of program documents.
• A review of national and international literature.
• Key informant interviews with ESDC National Capital Region program administrators, PT program representatives, community based organization project sponsors, participants and employers from three provinces (Alberta, Manitoba and Nova Scotia).
• Administrative data captured from Participant Information Forms and Participant Feedback Forms available from four PTs (Yukon, British Columbia, Ontario and Nova Scotia), Project Approval Documents and Project Sponsor information.^{Footnote 8}

The analysis of the design and delivery of the program consisted of identifying older worker labour market programming best practices and lessons learned from the literature. They were assessed for their relevance and compared against the key design elements of the program through program documents and key informant interviews. Any design aspects that could potentially improve the program were identified and are described in other sections of this report.

Program employment outcomes relied mainly on the available administrative data and reporting was based on three PTs [British Columbia, Yukon and Nova Scotia]. As such, the interpretation of program outcomes cannot be assumed to represent the program’s overall performance. Finally, program documents and to a lesser extent views from respondents from three provinces [Alberta, Manitoba and Nova Scotia] were also used to report on outcomes. See Annex B for a more detailed description of the methodology used for this evaluation.

4.1 Alignment of the TIOW program with best practices

5.1 Program reach and participant profile

5.2 Outcomes

This section presents outcomes for British Columbia, Yukon and Nova Scotia. It is based on administrative data from 646 participants from these three PTs.^{Footnote 19} The outcomes of interest are employability, paid employment and self-employment. Supporting evidence was drawn from the program’s administrative data available at the time of evaluation, key informant interviews and program documents.

6.1 Performance Measurement Strategy

The program’s Performance Measurement Strategy was updated in 2013 and outcome statements in the logic model were re-examined in 2014 in light of the new eligibility criteria.^{Footnote 21} Overall program activities, outputs and general outcome statements remained unchanged.

The majority of the program performance data is gathered through three paper forms. Participant Information Forms (PIFs) are filled in by program participants prior to starting the program. Participant Project Feedback Forms (PEFs) are filled out immediately after the completion of the program. Follow-up Evaluation Forms are sent to former participants three to six months afterwards. Completed PIFs and PEFs are forwarded by Project Sponsors to PTs. Evaluation Forms are typically administered directly by PTs, where after conducting the analysis they may wish to do, PTs send the forms to ESDC. Program officials at ESDC manually enter data from the forms into a database. Most of the key informants from the PTs and Project Sponsors felt that capturing the PIF and PEF electronically would be more efficient.

6.2 Data quality

The program uses employability outcome measures that are similar to those found in the literature, but they may not provide the most accurate measures of employability. For example, program participants are asked to respond yes or no as to whether the TIOW project improved their confidence in finding a job. However, there are several measurement instruments developed and used by scholars which would more accurately measure employability outcomes.^{Footnote 22} For example, one such tool measures re-employment self-efficacy where respondents are asked to indicate their degree of confidence in being able to find paid employment on four prospect statements using a five-point Likert scale. Another measures job search self-efficacy and asks respondents to indicate their degree of confidence in being able to successfully perform six job search activities using a five-point Likert scale. Given it is the only program in the department to collect data directly from its participants before and after their participation, there is an opportunity to capture informative pre and post observations on behaviours and outcomes of interest.

The completeness of the compiled data was relatively good for demographic variables (age, gender, education, and region). However, outcome variables were poorly populated. For example, more than 50% of the records for the employment variable were missing. Other variables related to earnings (i.e. number of Employment Insurance collections, number of months on Social Assistance or on Employment Insurance) also have many missing values. Participant feedback variables (i.e. participant auto ranking of self-confidence) have approximately three quarters of the values missing, which limits the usefulness of some of the data. In addition, outcome variable data was only accessible for three PTs. In the end, this constrained the analysis evaluators could undertake with the administrative data.

6.3 Program costs

A cost-benefit analysis was not conducted for the TIOW program interventions and services since the evaluation design did not include an incremental impact analysis. Also, it was not possible to calculate the average cost for the overall program because participant data was not available nationally. However, it was possible to make a more limited calculation of the average cost per participant based on contributions made by the federal government for four jurisdictions.

In 2014-2015 and 2015-2016 Ontario, Nova Scotia, British Columbia and the Yukon claimed $15M from the federal government and 2,785 individuals participated in program in those PTs. The estimated average cost per program participant is $5,386 (see Table 4 below). The previous evaluation estimated the average cost at $7,000 per participant.

Source: *Program budget and administrative data for Ontario, British Columbia, Yukon and Nova Scotia however excludes Ontario for 2015-2016 since participant data was not available.

Since the analysis in Table 4 excludes Ontario for 2015-2016, the difference in the cost per client between 2014-15 and 2015-16 may be explained by varying project delivery costs within a PT.^{Footnote 23} For example, costs within a PT can be driven by the availability of existing labour market programming infrastructure that can be leveraged by a Project Sponsor in an eligible community. In particular, costs may be higher in more remote communities. In addition, participant allowances may be higher in some PTs or not offset by wages paid from an employer-based work experience or through EI.

Recommendation #1:

First, Project Sponsors provided many activities that strengthened job seekers self-efficacy beliefs. However they did not consistently clarify and set job goals for participants. When goals are clearly defined, a focused search strategy generally produces the most positive outcomes.

Secondly, while some Project Sponsors linked participants with employers, others did not. The marketing of participants to employers, which may include linking participants with employers, could help improve access to labour markets.

Finally, providing ongoing formal peer supports during the entire time participants are actively seeking employment is a critical design element. Project Sponsors created supportive environments for their participants and the participants acknowledged this value. Participants indicated forming informal groups to support each other following the end of their participation in programming. However, evaluators observed little by way of formal ongoing group sessions facilitated by Project Sponsors. Providing ongoing support is key given that almost half of participants were not employed following the end of their participation in the TIOW program.

Recommendation #2:

One of the design features that labour market programs for older workers should include is the careful identification, screening and selection of participants. This allows program coordinators to stream participants into targeted and tailored programming that meets their specific strengths and addresses their specific weaknesses. However, Project Sponsors do not consistently assess all potential participants. Where they do, the processes and tools used for assessment vary from one jurisdiction to another and some assessments are more robust than others.

The literature suggests that individuals with high levels of education tend to be more self-directed learners and may not respond favourably to TIOW programming. Individuals lacking in social support, that demonstrate symptoms of anxiety and possess lower levels of education benefit from a more structured and formal programming that offers assistance and ongoing support.

Consistently robust assessments of participants would assist in streaming older workers into labour market programming that is more responsive to their needs.

Recommendation #3:

The ongoing monitoring of participant demographic characteristics and outcomes between program evaluations is paramount to effectively managing any program. For example, deviations from recruiting the most suitable participants can be identified using data collected on an ongoing basis from participants. If deviations are detected, Project Sponsors can be alerted to ensure program implementation is in keeping with key policy requirements. However the ongoing monitoring of performance requires the systematic, timely and efficient collection of reliable data.

The TIOW program performance data is captured through paper based forms completed by participants immediately before and after program participation. The completed forms are expedited in batches by the PTs to ESDC program officials where they are manually entered into a database. The process is time consuming and prone to input error. The profile variables (age, gender, education, and region) were suitably captured, but the outcome variables were poorly populated, suffering from non-response errors. Furthermore, though the employability and employment outcomes measures captured by the program are relevant, there are more reliable measures which could capture more informative data for both performance measurement and evaluation purposes.

Also, the absence of Social Insurance Numbers (SINs) from participants limits the scope of future evaluations. The collection of the SINs from program participants would allow ESDC to conduct a more thorough analysis of program performance and incremental impact analysis by facilitating participant identification and linkage of the program’s administrative data to other administrative data (e.g. Canada Revenue Agency).

The current system as it exists limits ESDC’s ability to perform adequate monitoring and produces challenges in terms of analysis for the evaluation of the TIOW program.

TIOW evaluation questions

Introduction

The following presents the methodologies used for the Evaluation of the Targeted Initiative for Older Workers (TIOW) program. This Appendix describes the scope of the evaluation, data sources, data collection methods and constraints and limitations.

Scope and breadth

This evaluation is outcomes based and examined the period covering April 2014 to June 2016. It focused on the issues and questions identified in Appendix A. It includes all provinces and territories with the exception of Québec, which carries out its own evaluations, and Nunavut, as it did not participate in the program. However, due to the limited availability of administrative data, the scope related to reporting on program outcomes was restricted to British Columbia, Yukon and Nova Scotia.

Methodologies

The following methods were used to respond to the evaluation questions. All methods were carried out in-house by ESDC Evaluation Directorate staff.

• Document and literature review - This included previous evaluation technical and final reports, a review of national and international literature and program documentation provided by the program.
• Administrative data - Program performance data was generated from the TIOW program Access database. Data captured included information from the Participant Information Forms, Participant Feedback Forms, Participant Evaluation Forms and Project Sponsor information.
• Key informant interviews – Interviews were carried out by telephone or in-person. Respondents included representatives from ESDC National Capital Region program administration (n=5). It included randomly sampled respondents from three provinces (Alberta, Manitoba and Nova Scotia) that consisted of PT program representatives (n=3), project sponsors (n=8), participants (n=10) and employers (n=4) from those provinces. Interviews were carried out by ESDC evaluators between January and February 2016.

Analysis

The data analysis strategy included the triangulation of multiple lines of evidence. The table below describes the proportional and frequency terms used in the report to quantify the extent of agreement amongst interview respondents to specific questions and issues.

Constraints and limitations

In 2015, the program was putting in place Information Sharing Agreements between ESDC and the PTs in order to access participant data (PIF and PEF). By February 2016, data from four PTs (Ontario, British Columbia, Yukon and Nova Scotia) was available for evaluation purposes. Readers should be mindful when interpreting findings citing administrative data since it may not be representative of the other provinces and territories participating in TIOW. Furthermore, Social Insurance Numbers are not collected for TIOW program participants. Evaluators were unable to link the TIOW program administrative database (PIF and PEFs) to the Canada Revenue Agency and ESDC databases and apply incremental impact modelling.