2016 Evaluation of the Targeted Initiative for Older Workers
2016 Evaluation of the Targeted Initiative for Older Workers [PDF - 252.96 KB]
Large print, braille, MP3 (audio), e-text and DAISY formats are available on demand by ordering online or calling 1 800 O-Canada (1-800-622-6232). If you use a teletypewriter (TTY), call 1-800-926-9105.
On this page
- List of acronyms
- List of tables
- Executive summary
- Management response
- 1. Introduction
- 2. Program description
- 3. Evaluation strategy
- 4. Best practices in programming for unemployed older workers
- 5. Performance – Program reach, client profile and outcomes
- 6. Efficiency and Economy - Performance measurement and program costs
- 7. Conclusions
- Recommendations
- Appendix A – Evaluation matrix
- Appendix B – Methodology
- Appendix C – Government of Canada funding contribution
List of acronyms
- ISA
- Information Sharing Agreement
- PT
- Provinces and Territories
- SIN
- Social Insurance Numbers
- TIOW
- Targeted Initiative for Older Workers
List of tables
- Table 1: The TIOW program participant profile by gender, age and educational attainment
- Table 2: The TIOW program participant primary job loss experience by gender and age
- Table 3: Participant rating of the TIOW program employment assistance and employability improvement activities
- Table 4: TIOW program client and costs summary
Executive summary
Introduction
This report presents the findings and conclusions of the 2016 Evaluation of the Targeted Initiative for Older Workers (TIOW) program. The main purpose of the evaluation is to inform policy work in 2016-2017 prior to the end of the current renewal period (April 2014 to March 2017). It principally examines two areas: 1) how the introduction of two new eligibility criteria in 2014 impacted provinces and territories (PTs) in program design and delivery; and 2) how the program aligns with older worker labour market programming best practices when assessed against the academic literature. It also looked at the program’s performance in terms of achieving outcomes of employability, paid employment and self-employment where data was available.
Program description
The program is a federal-provincial/territorial cost-shared initiative. It supports unemployed older workers, primarily between the ages 55 and 64, with their re-integration into employment. The program is offered in small communities of 250,000 or less experiencing high unemployment and/or significant downsizing and closures. As per Budget 2014, the program was renewed for a third consecutive time for a three-year period (April 2014 to March 2017) representing a federal investment of $75M. Since its renewal in 2014, the program’s terms and conditions were expanded to include two new community eligibility criteria: unfulfilled employer demandFootnote 1 and skills mismatchesFootnote 2.
The expected outcome of the initiative is to increase the employability of unemployed older workers and help them to re-integrate into employment. PTs design projects based on the parameters set out in bilateral agreements between the Federal Government and participating PTs. Projects are typically delivered by third-party service providers.
The terms and conditions specify that projects must provide employment assistance activities such as résumé writing, interview techniques, employment counselling and job search techniques. In addition, projects must also offer at least two employability improvement activities such as assessment activities, peer mentoring, skills training, wage subsidies, community based work experiences, preparation for self-employment, post project follow-up and mentoring. Projects involving activities other than work experience or subsidized employment must offer income support in the form of allowances to participants, where PT legislation allows. All projects must involve at least 25 hours per week of activity for participants. There is no minimum or maximum number of weeks of participation.
Context
A more rigorous evaluation of the program was completed in 2014. It employed a complex and costly non-experimental design and survey and concluded authoritatively on the program’s performance. Among other things, it found that:
- Program participants were more likely than non-program participants to find employment by 6 percentage points.
- No employment earnings differences were found between participants and the comparison group of older workers.
The 2016 Evaluation complements this earlier work. It is calibrated and uses an outcome based design; its scope and focus are defined by the needs of the program area and the Department, as set out in the TIOW Evaluation Strategy approved by the Departmental Evaluation Committee in November 2014.
The 2016 Evaluation uses an exploratory/investigative approach focusing on enhancing the collective knowledge base on effective labour market programming approaches for older workers and the new demand driven aspects of the program. The evaluation examined the new project eligibility criteria and sought to understand the best program design activities, delivery practices and lessons learned from programs similar to TIOW. It also looked at the program’s performance in terms of achieving outcomes of employability, paid employment and self-employment where data was available.
The data collection consisted of a review of program documents, administrative data captured by the program, key informant interviews and a review of national and international literature.
The original scope of this evaluation included all provinces and territories with the exception of Québec, which carries out its own evaluations, and Nunavut, as it did not participate in the program. This evaluation bases its reporting on the administrative data available between April 2014 and February 2016 from four PTs: Ontario, British Columbia, Nova Scotia and Yukon.Footnote 3 Administrative data is only available from four PTs because Information Sharing Agreements (ISAs) with other PTs were still being negotiated during the data-gathering phase of this evaluation.
Data quality issues also presented challenges in conducting an outcomes-based assessment. Incomplete participant information, for example, constrained the level of analysis that could be undertaken. Since the evidence does not reflect the unrepresented provinces, the findings and conclusions that follow present a partial story of the program’s overall performance.
Conclusions
A total of 287 projects (60 new projects and 227 renewed) were delivered across Canada (including Quebec) by Project Sponsors between April 2014 and March 2016. Approximately 83% (50 out of 60) of the new projects qualified for funding as a result of the new eligibility criteria allowing communities challenged by skills mismatches and/or an unfulfilled employer demand to participate in the initiative. Overall, the program is reaching its targeted clients in Ontario, British Columbia, Yukon and Nova Scotia where the majority of participants are in the primary age range (55 and 64 years of age).
Participant outcomes from British Columbia, Yukon and Nova Scotia suggest that the TIOW program in those PTs is progressing towards its intended outcome. Outcome findings indicate that overall, participants expressed positive satisfaction regarding the program, and that job seeking activities and employability activities were seen as helpful overall.
When compared to the academic literature, research showed that overall the TIOW program is effectively designed and delivered to assist older workers to re-integrate into employment. For example the program engages in the recruitment of participants which includes the identification, assessment and selection of participants. It also provides income supports tailored to the needs of older workers. It provides relevant employment assistance and employability improvement activities, and facilitates those activities through group-based approaches. The program is in effect closely modeled after similarly designed labour market programs for older workers that successfully produced higher rates of employment and employment earnings as well as job satisfaction.
In spite of the program having many best practices in place, there are a few areas where potential improvements could be made:
- Many of the program activities strengthened job seekers self-efficacy beliefs; however job goals were not consistently clarified nor set with participants. When goals are clearly defined, a focused search strategy generally produces more positive employment outcomes.
- The most successful programs for older workers provide some form of marketing of older workers to employers. Not all projects marketed participants to employers or established partnerships with potential employers. Linking participants to employers responds to their need for pre-screened, trained older workers.
- A high level of ongoing, positive peer support motivates participants which is positively co-related with job-search intensity. There was a lack of formal, ongoing support facilitated by projects following the end of participation in programming.
- Labour market programs for older workers should include the careful identification, screening and selection of its participants. As noted above, this is done to an extent. However, Project Sponsors do not consistently assess all potential participants; the processes and tools used for assessment vary from one jurisdiction to another; and some assessments are more robust than others.
Based on the findings and conclusions presented in this report, the evaluation proposes the following three recommendations:
Recommendation #1:
It is recommended that ESDC work with the provinces and territories to develop guidelines for projects to include activities that:
- 1.1 Clarify and set job goals for participants.
- 1.2 Improve access to labour markets by linking participants with employers.
- 1.3 Provide facilitated ongoing group sessions for participants following the end of their formal programming.
Recommendation #2:
It is recommended that ESDC work with the provinces and territories to develop recruitment guidelines for projects through the careful identification and screening of older workers so that they may be streamed into the most appropriately designed labour market interventions. This may translate into providing less intense programming (i.e. less than 12 weeks) to more highly educated participants through the TIOW program or referring them elsewhere within a PT where other, more relevant labour market programming is available.
Recommendation #3:
It is recommended that ESDC improve the quality of the data collected from all Project Sponsors and participants for the continuous monitoring and evaluation of outcomes by:
- 3.1 Capturing more reliable employability and employment measures on the Participant Information Forms, Participant Feedback Forms and follow-up Evaluation Forms.
- 3.2 Examining the feasibility of implementing the electronic capture of its data collection forms from the provinces and territories.
- 3.3 Engaging key stakeholders to explore the feasibility of collecting participants’ Social Insurance Numbers in future years.
Management response
Introduction
The Skills and Employment Branch (SEB) would like to thank the Evaluation Directorate and all those who participated in the 2016 Evaluation of the Targeted Initiative for Older Workers (TIOW). In particular, SEB acknowledges the contribution of provinces and territories (PTs), participants, including key informants, and program officials during the course of this Evaluation.
TIOW is cost shared between the federal and provincial/territorial governments and supports unemployed older workers (typically ages 55 to 64) living in small, vulnerable communities with their reintegration into employment. It is targeted to cities and towns with a population of 250,000 or less that are affected by high unemployment and significant downsizing or closures. As noted below, two additional community eligibility criteria were added in 2014: unfulfilled employer demand and skills mismatches.
Under TIOW, PTs are responsible for targeting specific communities and for designing and delivering projects which offer group-based programming and/or self-employment assistance. The federal government is responsible for establishing policy parameters, approving projects, overseeing overall implementation of agreements, managing allocations and conducting program evaluations.
TIOW was renewed in 2014 for an additional three years to support the Government’s commitment to making labour market programming more demand-driven. Accordingly, the program’s eligibility criteria were expanded to provide PTs with the added flexibility to implement TIOW projects in communities that are experiencing an unfulfilled employer demand or skills mismatches.
TIOW is intended to complement other types of labour market programming currently available such as, but not limited to, the Canada Job Fund, Labour Market Development Agreements and Labour Market Agreements for Persons with Disabilities in its orientation towards human capital development focusing on both employment assistance activities (such as résumé writing, interview techniques, counselling and job finding clubs, etc.) and employability supports (such as skills training, preparation for self-employment, basic skills upgrading work experience, direct marketing to employers etc.) for older workers.
In the Summer-Fall 2016 ESDC undertook a comprehensive review of its full suite of labour market transfer agreements (LMTAs) including broad-based consultations with PTs and stakeholders with the objective of ensuring that LMTAs are relevant, flexible and responsive to current and emerging labour market needs and priorities. This aligns with the Government of Canada’s commitment to rationalize and expand agreements that support skills training. Findings from this review supported the need for future labour market agreements to:
- reflect to the needs of individuals, workers, employers and under-represented groups;
- build on strong evidence for relevant performance measurement to better inform and serve Canadians and help them achieve meaningful employment outcomes; and
- foster capacity to develop innovative program approaches and the sharing of best practices within the skills training agreements.
ESDC is currently exploring policy options for moving forward that have been informed by the review. Findings from this evaluation will be particularly relevant in helping to identify best practices in labour market programming designed to reach those clients most in need.
As the population ages and older Canadians represent a larger share of the workforce, it is important to encourage their retention and re-entry in the labour market to support economic growth.
SEB agrees with the evaluation findings and provides the following Management Response.
Key findings
While the Evaluation looked at the program’s performance in terms of achieving outcomes where data was available, its main focus was to examine two areas:
- how the introduction of two new eligibility criteria in 2014 impacted program delivery by PTs and
- how the program aligns with older worker best practices when assessed against the academic literature.
The Evaluation found that 83% of new TIOW projects approved between April 1, 2014 and March 31, 2016 were designed to support communities meeting the new eligibility criteria. This suggests that these added flexibilities addressed local labour market needs. The Evaluation also indicated that TIOW programming uses many of the best practices identified in the literature and identified some gaps that could be addressed in the future when designing labour market programming for older workers. Evidence shows that the program is reaching a vulnerable population in need of employment services. In addition, the Evaluation found that participants were satisfied with the job seeking and employability activities they received through the program and that TIOW effectively assists unemployed older workers improve their employability and transition into employment. While data limitations present challenges for analysis, this Evaluation nonetheless provides valuable insights into the program’s effectiveness.
Recommendations and follow-up actions
The evaluation outlined three recommendations:
Recommendation 1:
The Evaluation recommends that the program area work with the provinces and territories to develop guidelines for projects to include activities that: clarify and set job goals for participants, improve access to labour markets by linking participants with employers and provide facilitated ongoing group sessions for participants following the end of their formal programming.
Follow-up action 1:
A program guide developed by ESDC officials currently exists. In addition to outlining eligibility and other requirements of TIOW, as laid out in the terms and conditions of the initiative, the program guide highlights some best practices related to the delivery of group-based employment programming as well as specific tools and tips for project development. This document is evergreen and is regularly shared with PTs.
This guide will be strengthened by adding guidelines supporting goal setting, employer linkages and ongoing group sessions post intervention. It is important to note, however, that under current labour market transfer agreements, PTs have the flexibility to design and deliver programming that meets their local need as long as it is consistent with broader program parameters.
Administrating and delivering this cost-shared program has fostered strong Federal-Provincial/Territorial relationships. This multilateral engagement can further support a dialogue to address this recommendation. The regular sharing of best practices through FPT calls, the Best Practices Compendium and focused discussions at multilateral workshops is another aspect of the program that is transferable across other labour market programming as we look for opportunities to innovate.
Recommendation 2:
The Evaluation recommends that the program area work with the provinces and territories to develop recruitment guidelines for projects to support the careful identification and screening of older workers so that they may be streamed into the most appropriately designed labour market interventions.
Follow-up action 2:
Including such recruitment guidelines for projects could strengthen the program guide by providing PTs with additional tools and tips to improve project development. As noted above in response to Recommendation 1, it would be at the discretion of PTs to incorporate the guidelines into the design and delivery of their programming in response to their local labour market needs. The other multilateral engagements described above could also support a dialogue to address this recommendation. The program area will continue to look for opportunities to support innovation and the sharing of best practices as a means to ensure effective programming.
Recommendation 3:
The Evaluation recommends that ESDC improve the quality of the data collected from all project sponsors and participants for the continuous monitoring and evaluation of outcomes by: capturing more valid and reliable employability and employment measures on its Participant Information Form, Participant Feedback Form and follow-up Evaluation Form; examining the feasibility of implementing the electronic capture of its data collection forms from the provinces and territories; and, engaging key stakeholders to explore the feasibility of collecting participants’ Social Insurance Numbers (SINs) in future years.
Follow-up action 3:
The program area agrees that the quality of the data captured through the TIOW program could be enhanced. Similar recommendations were made in the 2014 Evaluation and a number of steps have been taken to address them. These include:
- Discussions with both internal and external stakeholders have been undertaken to explore alternate approaches to streamline the collection and dissemination of TIOW client data. In 2016-17, a TIOW database project proposal and business case that includes the electronic capture of program data was submitted as part of the annual Investment Management Process and was approved as recommended.
- The program area began negotiating Information Sharing Agreements (ISAs) with PTs in April 2014. At the same time, the program area was also negotiating separate ISAs under the Canada Job Fund Agreements. As a result, it has taken some time to establish and implement ISAs with all PTs. Work will continue to establish ISAs with the remaining jurisdictions to ensure that PTs can share client information with ESDC.
Performance measurement was an area of significant interest to stakeholders and provinces and territories during recent consultations on the future of ESDC’s suite of labour market transfer agreements. There is a need to improve data collection to better measure outcomes, support comparisons across the transfers and inform future policy development. There are also opportunities to reduce the administrative burden on both ESDC and our provincial and territorial partners.
The collection of Social Insurance Numbers (SIN) would greatly enhance performance measurement and ESDC is exploring the feasibility of collecting SINs under all labour market transfer agreements. PTs have been engaged on this subject and have expressed their support. Authority to collect SINs would need to be obtained from the Treasury Board Secretariat. Proper privacy and data-sharing protocols would need to be developed to ensure that SINs are used exclusively as a file identifier for the purposes of performance measurement and not to identify individuals.
1. Introduction
This report presents the findings and conclusions of the 2016 Evaluation of the Targeted Initiative for Older Workers (TIOW) program. It examines the period from April 1, 2014, when the program was last renewed, to June 2016, with the objective of informing policy work in 2016-2017 prior to the end of the current renewal period on March 31, 2017.
This Evaluation is calibrated; rather than employing the more comprehensive methodology used in the 2014 Summative Evaluation of the program, its scope and focus are defined by the needs of the program area and the Department, as set out in the TIOW Evaluation Strategy approved by the Departmental Evaluation Committee in November 2014. It aims to: (1) enhance collective knowledge on design, delivery and effectiveness and (2) explore implementation of new program eligibility criteria. It also looked at the program’s performance in terms of achieving outcomes of employability, paid employment and self-employment where data was available.
This Evaluation originally intended to cover all provinces and territories (PTs) with the exception of Québec, which carries out its own evaluations, and Nunavut, which does not participate in the program. However, in 2014, it was recommended that the Department put in place individual Information Sharing Agreements (ISA) with each PT. As a result, at the time of the data analysis there was limited availability of administrative data since ISAs were not in place with all PTs. Evaluators therefore reported the program’s reach and client profiles for British Columbia, Nova Scotia, Ontario and Yukon and reported on outcomes for British Columbia, Nova Scotia and Yukon, as Ontario does not provide program outcome data to ESDC. The evaluation was carried out entirely in-house by ESDC evaluation staff between March 2015 and June 2016.
This report provides a description of the program’s mandate, objectives, components and resources (Section 2); the evaluation strategy (Section 3); the evaluation’s key findings, lessons learned, conclusions and recommendations (Sections 4 to 7); the evaluation questions (Appendix A – Evaluation Matrix); the evaluation design and methodology (Appendix B - Methodology); and the Government of Canada Funding Contribution (Appendix C).
2. Program description
The TIOW program is a federal-provincial/territorial cost-shared initiative that has two main objectives. First, it supports unemployed older workers living in small, vulnerable communities by helping them to re-integrate into employment. Second, in labour markets with little likelihood of immediate employment, programming aims to increase the employability of unemployed older workers and extend their labour market participation while their communities undergo adjustment.
The Government of Canada introduced the TIOW program in 2006 to address challenges that older workers face with re-integrating into employment and started to establish bilateral Agreements with PTs in 2007. Budget 2008 extended the program until March 2012 and Budget 2011 extended it again until March 2014. Budget 2014 renewed the Initiative a third time for a three-year period (until March 2017) representing a federal investment of $75M (i.e. $25M/year in 2014-15, 2015-16, and 2016-17). The program is cost-shared between the federal government (up to a maximum of 70% of total program costs) and each participating PT (a minimum of 30% of total program costs).Footnote 4 See Appendix C – Government of Canada Funding Contribution for a breakdown by PT.
The program was initially designed to assist unemployed older workers in small communities of 250,000 or less that were experiencing high unemployment and/or significant downsizing and closures. With the renewal in 2014, the program’s terms and conditions were expanded to include two new eligibility criteria—that of unfulfilled employer demand and/or skills mismatches.Footnote 5 These were added in response to stakeholders’ desire to incorporate a more well-rounded approach focusing on both demand-driven and supply-side aspects.
Under TIOW, the federal government is responsible for establishing policy parameters, approving projects, overseeing overall implementation of agreements, managing allocations and conducting program evaluations. PTs are responsible for identifying specific communities for participation, as well as for designing and delivering projects that meet their local labour market situations and client needs. Projects are delivered by third-party service providers (Project Sponsors) and PTs are responsible for managing these contracts and conducting monitoring visits to ensure projects align with program objectives.
Once a PT identifies an eligible project, it submits a proposal to ESDC for assessment using a description and recommendation form. ESDC works with the PT to ensure that the proposal meets all of the necessary requirements and then submits it to the Minister of Employment, Workforce Development and Labour for approval.
In order to be eligible to participate in the program, older workers must meet all of the following criteria:
- Be unemployed.
- Be legally entitled to work in Canada.
- Lack skills needed for successful integration into new employment.
- Live in an eligible community.
- Normally be aged 55 – 64.Footnote 6
Under TIOW, all projects must provide employment assistance activities such as résumé writing, interview techniques, counselling and job search techniques. In addition, projects must offer at least two employability improvement activities such as:
- Assessment activities, which include (but are not limited to) prior learning assessment, personal portfolio development, essential skills assessment, vocational and interest testing, and high school equivalency assessment.
- Peer mentoring activities, which include (but are not limited to) counselling and tutoring.
- Skills training that allow participants to obtain the skills they need to become employed, ranging from basic to advanced skills.
- Wage subsidies to employers to help participant’s access available jobs.
- Preparation for self-employment where participants receive training, professional business support and mentoring to assist them with starting their own business.
- Community-based work experiences in communities with no other opportunities for older workers to gain practical hands-on work experience.
- Direct marketing to employers including the marketing of project participants by the sponsor or the provision of a job-worker matching service.
- Post-project follow-up mentoring and support including ongoing counselling, networking and individual support during a work placement or to project graduates who continue their job search.
- Other employability approaches that have demonstrated success (e.g. job shadowing).
Projects involving work experience or subsidized employment must provide income support in the form of wages to participants. Projects involving activities other than work experience or subsidized employment must provide income support in the form of allowances to participants unless PT legislation does not permit it. All projects must involve at least 25 hours per week of activity for participants. There is no minimum or maximum number of weeks of participation.
3. Evaluation strategy
3.1 The 2014 TIOW summative evaluation
The TIOW Summative Evaluation conducted in 2014 employed a complex and costly non-experimental design and concluded authoritatively on the program’s performance. The evaluation methodology consisted of a survey of former program participants, an incremental impact analysis using a limited-treatment comparison group, a benchmark study to examine the extent of job loss of older Canadians living in the same geographic areas as the program participants, key informant interviews with employers, project representatives, ESDC program officials and PT representatives and a review of key documents.
The 2014 Evaluation concluded that 75% of survey respondents found paid employment following their participation in the program and the majority of former program survey respondents believed that their program participation improved their employability. Many factors may have contributed to this positive outcome. The incremental impact analysis was used to determine how much of those outcomes can be attributed to the program. It concluded that:
- Program participants were more likely than non-program participants to find employment by 6 percentage points.
- No employment earnings differences were found between program participants and the comparison group of older workers.
3.2 The 2016 TIOW evaluation
The objective of the 2016 Evaluation was to inform policy work in 2016-2017 prior to the end of the current renewal period on March 31, 2017. It was therefore calibrated, rather than employing the more comprehensive methodology used in the 2014 Summative Evaluation of the program. The Evaluation Directorate and the Employment Program Policy and Design Directorate agreed to a strategy that would enhance the collective knowledge base on effective labour market programming approaches for older workers.Footnote 7
More specifically, it focused on examining best practices in program design and exploring the implementation of the new program eligibility criteria. Where data was available it also assessed the program’s performance in terms of achieving outcomes such as employability, paid employment and self-employment.
The data collection consisted of four methodologies that included:
- A review of program documents.
- A review of national and international literature.
- Key informant interviews with ESDC National Capital Region program administrators, PT program representatives, community based organization project sponsors, participants and employers from three provinces (Alberta, Manitoba and Nova Scotia).
- Administrative data captured from Participant Information Forms and Participant Feedback Forms available from four PTs (Yukon, British Columbia, Ontario and Nova Scotia), Project Approval Documents and Project Sponsor information.Footnote 8
The analysis of the design and delivery of the program consisted of identifying older worker labour market programming best practices and lessons learned from the literature. They were assessed for their relevance and compared against the key design elements of the program through program documents and key informant interviews. Any design aspects that could potentially improve the program were identified and are described in other sections of this report.
Program employment outcomes relied mainly on the available administrative data and reporting was based on three PTs [British Columbia, Yukon and Nova Scotia]. As such, the interpretation of program outcomes cannot be assumed to represent the program’s overall performance. Finally, program documents and to a lesser extent views from respondents from three provinces [Alberta, Manitoba and Nova Scotia] were also used to report on outcomes. See Annex B for a more detailed description of the methodology used for this evaluation.
4. Best practices in programming for unemployed older workers
The following section describes best practices in the design and delivery of labour market programming for unemployed older workers based on a review of national and international literature.Footnote 9 These best practices were compared to TIOW projects designed and delivered by PTs to determine the extent to which the program aligns with them. Lessons learned emerged in cases where the best practices were not observed in the program. This literature review examined the instructional components (e.g. employment assistance and employability improvement activities) and labour market interventions (e.g. income support) provided to older workers. Also included were specific program design elements that have been found to be particularly effective in helping older workers become re-employed, such as one-on-one and group support, peer mentoring and post-project follow-up. It also considers recruitment activities designed to identify, assess, select and stream participants into the most relevant labour market programs.
4.1 Alignment of the TIOW program with best practices
4.1.1 The TIOW program and what works
The program includes many design elements that are similar to other labour market programs that successfully reintegrate older workers back into the workforce. Furthermore, there are a number of other experimental studies of programs similar in design to the program that targeted other populations (e.g. general job seekers and people with physical disabilities, mental health problems, prison records and alcohol/drug problems). These studies found significantly higher employment rates for individuals who participated in those programs compared to individuals who received no labour market programming interventions.
- Some of the key program design best practices found in the TIOW program include the following:
- Employment assistance activities as well as direct marketing to employers;
- Training to help older workers obtain new job skills either through basic skills upgrading, specific occupational skills training or preparation for self-employment;
- Income support either in the form of allowances or wages;
- One-on-one and group support, peer mentoring and post project follow-up intended to help clients through the difficulties encountered in both training and the subsequent job search; and
- Participant recruitment and assessment activities.
4.1.2 The TIOW program and areas for potential improvement
This section presents several key design elements from the literature that could have a practical application when implementing TIOW. The evidence from the literature is firstly presented to support the best practice. It is then followed by some context as to the extent to which the best practice was observed in the field.
Goal establishment
The literature suggests that clarity around an individual’s job search goals can have a positive influence on job search effort and intensity. Footnote 10 When goals are clearly defined, a job search will be more focused and more likely to lead to the desired type of employment.
The establishment of goals for participants was not offered as a project activity by all Project Sponsors. Although some service providers interviewed stated that goal establishment was key, others did not put as much emphasis on it.
Marketing to employers
The literature suggests that the most successful programs for older workers involved the marketing of workers to employers.Footnote 11 More generally, partnerships between employers and community organizations delivering labour market programs to older workers should be well established. For example, job placement services improve access of older workers to employers and responding to the latter’s desire for pre-screened and trained older workers.
The marketing of participants to employers is not a mandatory activity for Project Sponsors. Interviews with Project Sponsor respondents suggest that many types of employer related activities were provided. For example, in some projects employers were invited to make presentations to groups of program participants. In other cases participants took part in job fairs. Finally some Project Sponsors promoted participants directly to potential employers.
However, some Project Sponsors did not establish partnerships with potential employers. This is a particularly important activity given that the literature points to older workers often being victims of age discrimination and negative stereotyping over their work performance. Linking participants with employers where feasible improves access to jobs, providing employers with sought after pre-screened and trained older workers.
Peer support
According to the literature an important job seeking activity for older workers (beyond resume writing, interview skills, counselling and job search techniques) is peer support.Footnote 12 Older job seekers who receive peer support tend to display high levels of activity and intensity in job searching. Moreover, supportive messages provided by peers such as unemployed friends are perceived to be more positive than those provided by family or employed friends. Programs using a peer support approach have shown greater job placement effectiveness than have those using traditional job referral and information strategies.
At the end of the formal TIOW program activities, some participants reported that they formed informal networks amongst themselves as a means of supporting each other. Some Project Sponsors reported offering former participants individual access to their facilities and services. However, no formal post program group support facilitated by Project Sponsors was observed with the exception of Prince Edward Island’s real-time interactive social media network platform known as a “Ning”.Footnote 13 Similar to Facebook, it allows current participants, past participants and program/provincial coordinators to share links, news, contact information, quotes, videos and most importantly, allows the participants to stay in touch during and after their program training ends.
Recruitment: identification, assessment and selection of participants
The literature suggests that the demographic characteristics of older adults can assist in the streaming of individuals into the most appropriately designed labour market programs.Footnote 14 For example, individuals lacking in social support, demonstrating anxiety and possessing lower levels of education would benefit from a more structured and formal programming that offers assistance and ongoing support. In contrast, individuals who have higher levels of education, are generally self-directed learners, and have considerable social support may not respond favourably to such programming and may prefer a more independent, investigative job search approach. The program administrative data shows that 54% reported “having at least some post-secondary education”.Footnote 15 The consistent application of standardized assessments of program participants may help ensure that they are streamed into other most relevant labour market programming offered within a province or territory.
Project Sponsors that included an initial assessment of participants generally did so on the basis of abilities, skills, experience, educational level, objectives, physical capabilities, motivation and personal or social problems. These criteria were regarded as critical by the Project Sponsors in terms of the success or failure of the project. The evidence gathered through interviews suggests that not all potential program candidates are assessed by the Project Sponsor. But in cases where program candidates are assessed, the key informants and Project Sponsors’ reporting documents suggest that the objectives of assessing and the types of assessments administered to participants vary. For example, some Project Sponsors screen for readiness and select those who are most likely to obtain employment. Some assessments are standardized (by jurisdiction for example) while some Project Sponsors create their own assessment tools. Some Project Sponsors do not assess potential participants and recruit on a first come first serve basis. As stated by a key informant, there are numerous routes of entry for participants to access programming.
5. Performance – Program reach, client profile and outcomes
This section presents findings on the TIOW program’s performance with respect to the achievement of its expected outcomes. Reporting on the number of projects delivered is based on the program’s administrative data from across Canada. However, reporting on participant characteristics rely on administrative data from Ontario, British Columbia, Yukon and NovaScotia in 2014-15 and British Columbia, Yukon and Nova Scotia in 2015-16.Footnote 16 Reporting on participant outcomes rely on administrative data from British Columbia, Yukon and Nova Scotia. It should be noted that the program performance data is limited in terms of its quality due to the significant number of missing responses.
5.1 Program reach and participant profile
5.1.1 Projects delivered
A total of 287 projects were delivered across Canada (including Quebec) by Project Sponsors between April 2014 and March 2016. Among them, 60 were new projects and 227 were renewed, generally un-amended projects.Footnote 17
5.1.2 New project eligibility criteria
The 2014 renewal of the program broadened eligibility criteria to include communities challenged by skills mismatches and/or an unfulfilled employer demand. For example, communities facing economic changes such as having their primary industry shift to another sector or industry (i.e. from coal mining to tourism) or anticipating changes to labour market needs (i.e. challenges in finding workers in the service sector) were eligible for funding. Of the 60 new projects 83% (50 out of 60) qualified under the new eligibility criteria.
5.1.3 Participant profile
Based on available data, Ontario, British Columbia, Yukon and Nova Scotia delivered 148 projects between April 1, 2014 and March 31, 2016, and reached 2,785 participants (2,336 participants in 2014-2015 in Ontario, British Columbia, Yukon and Nova Scotia and 449 participants in 2015-2016 in British Columbia, Yukon and Nova Scotia).
The majority of participants in Ontario, British Columbia, Yukon and Nova Scotia were in the primary target age range, with 72% between the ages of 55 and 64, 22% between 50 and 54 years of age and 6% of participants were 65 years of age or older. Overall, there were more female (54%) than male (46%) participants. Most of the participants (54%) had at least some post-secondary education (see Table 1). These findings are similar to those of the 2014 Summative Evaluation.
Percentage of participants | Number of participants | |
---|---|---|
Gender | ||
Male | 46% | 1,260 |
Female | 54% | 1,508 |
Total | 100% | 2,768 |
Age | ||
50-54 | 22% | 593 |
55-64 | 72% | 1,974 |
65 + | 6% | 176 |
Total | 100% | 2,743 |
Education | ||
High school or less | 46% | 1,254 |
At least some post-secondary education | 54% | 1,488 |
Total | 100% | 2,742 |
Source: Administrative Data for Ontario, British Columbia, Yukon and Nova Scotia, where N=2,785. The non-response rate (percentage of missing values) is less than 2% which is not significant. The gender, age and education totals vary from N=2,785 due to the non-response rate.
Table description of Table 1
Table 1 provides a distribution of the TIOW program participants by gender, age and educational attainment.
The table is divided into three subsections (gender, age range and education) as rows and includes three columns. The first column, from left to right, lists the gender (male and female) and the total response rate for the gender category; the age range (50 to 54 years; 55 to 64 years; and 65 years of age and older) and the total response rate for the age category; as well as the level of education (high school or less; and at least some post-secondary education) and the total response rate for the education category. The second column lists the percentage of participants per gender (male; female; and total response), per age category (50 to 54 years; 55 to 64 years; 65 years of age and older; and total response) and per education level (high school or less; at least some post-secondary education; and total response ) while the third column lists the number of participants per gender (male; female; and total response), per age category (50 to 54 years; 55 to 64 years; 65 years of age and older; and total response) and per education level (high school or less; at least some post-secondary education; and total response). The source of data and the non-response rate are inserted beneath the table.
5.1.4 Job separation
A majority of the participants (56%) in Ontario, British Columbia, Yukon and Nova Scotia experienced their primary job loss within two years prior to their participation in the program.Footnote 18 Participants aged 50-54 were more likely than other participants to experience a primary job loss within two years prior to their participation in the program. Women (60%) were more likely than men (53%) to experience a job loss within two years prior to their participation (see Table 2 below). There is also significant proportions of men (47%) and women (40%) who experienced their primary job loss more than two years prior to their participation in the program.
This speaks to the challenges unemployed older workers face in finding stable and good paying employment. The fact that the program reached a population unable to replace their primary job over an extended period of time suggests that it is fulfilling a need.
Primary job loss | |||||||
---|---|---|---|---|---|---|---|
Months | 0-6 | 7-12 | 13-24 | 25-36 | 37-47 | 48+ | Total |
Subgroup | % | % | % | % | % | % | n |
Gender (n=2,495) | |||||||
Male | 5% | 6% | 42% | 3% | 2% | 42% | 1,135 |
Female | 6% | 6% | 48% | 2% | 1% | 37% | 1,360 |
Age (n=2,491) | |||||||
50-54 | 5% | 4% | 49% | 3% | 1% | 38% | 538 |
55-64 | 5% | 6% | 44% | 2% | 2% | 41% | 1,794 |
65 + | 4% | 4% | 43% | 2% | 2% | 45% | 159 |
Source: Administrative Data for Ontario, British Columbia, Yukon and Nova Scotia where N=2,785. The non-response rate (percentage of missing values) is approximately 10%. The gender and age totals vary from N=2,785 due to the non-response rate.
Table description of Table 2
Table 2 provides a distribution of the TIOW program participant primary job loss experience (primary job loss time period) by gender and age.
The table is divided into two subgroups (gender and age) as rows and includes eight columns. From left to right, the first column lists the participant’s gender (male or female) and the participant’s age range (50-54 years, 55-64 years, and 65 years and older). The following six columns, left to right, show the percentage of participants under every job loss time period (0 to 6 months, 7 to 12 months, 13 to 24 months, 25 to 36 months, 37 to 47 months, 48 months and over) by subgroup (gender type and age categories). The final column lists the total number of respondents per gender type and per age category. The source of data and the non-response rate are inserted beneath the table.
5.2 Outcomes
This section presents outcomes for British Columbia, Yukon and Nova Scotia. It is based on administrative data from 646 participants from these three PTs.Footnote 19 The outcomes of interest are employability, paid employment and self-employment. Supporting evidence was drawn from the program’s administrative data available at the time of evaluation, key informant interviews and program documents.
5.2.1 Employability outcomes
Employment assistance and employability improvement activities were provided by all Project Sponsors.Footnote 20 Overall the majority of participants from British Columbia, Yukon and Nova Scotia expressed positive satisfaction regarding the employment assistance and employability improvement activities in which they participated as illustrated in Table 3.
Activities | Not helpful | Somewhat helpful | Very helpful |
---|---|---|---|
Employment assistance1 | |||
Resume writing (N=350) | 1% | 13% | 85% |
Job interview techniques (N=349) | 1% | 18% | 81% |
Employment counselling (N=334) | 1% | 18% | 80% |
Job search techniques (N=345) | 1% | 19% | 79% |
Employability improvement2 | |||
Prior learning assessment (N=298) | 6% | 24% | 70% |
Portfolio development (N=269) | 5% | 23% | 72% |
Other vocational or interest assessment (N=224) | 3% | 22% | 75% |
Monitoring (N=250) | 2% | 12% | 86% |
Short-term work placement with an employer (N=124) | 11% | 23% | 65% |
Assistance with starting a business (N=194) | 10% | 28% | 62% |
Work experience on a community project (N=101) | 8% | 28% | 64% |
Help obtaining new job (N=218) | 5% | 18% | 78% |
Source: Administrative Data for British Columbia, Yukon and Nova Scotia. Note that totals do not add to 100% due to rounding.
1Between 334 and 350 participants provided a rating for resume writing, job interview techniques, employment counselling and job search technique activities respectively.
2Between 101 and 298 participants provided a rating for employability improvement activities since fewer of these activities were offered by Project Sponsors.
Table description of Table 3
Table 3 presents the participant rating of the TIOW program employment assistance and employability improvement activities.
The table is divided into two subsections (employment assistance activities and employability improvement activities) as rows and includes four columns. From left to right, the first column lists the activities under employment assistance (resume writing, job interview techniques, employment counselling and job search techniques) and under employability improvement (prior learning assessment, portfolio development, other vocational or interest assessment, monitoring, short term work placement with an employer, assistant with starting a business, work experience on a community project and help obtaining new job). The next three columns, from left to right, are rating categories (not helpful, somewhat helpful, and very helpful). They list the percentage of participant’s rating for each employment assistance and employability improvement activity. The source of data and the non-response rate are inserted beneath the table.
Key informants interviews with participants and Project Sponsors indicated that technology training was much appreciated by participants. Many of the project activities included computer training that complemented certain job seeking activities. For example, by enhancing their computer skills (i.e. Microsoft Word, Excel, PowerPoint) participants were able to develop their own resumes and by learning to browse the internet, participants were able to look for work on various websites and use social media.
5.2.2 Employment/Self-employment outcomes
The Evaluation evidence indicates that the program is relatively successful in helping older workers find employment or become self-employed. According to the service providers interviewed, most of the participants since 2014 from Alberta, Manitoba and Nova Scotia found employment or became self-employment either at the end of or prior to completing their programming.
Employment data was also collected through Participant Project Feedback Forms filled out at the end of each project. A total of 323 responses were collected from projects in British Columbia, Yukon and Nova Scotia. Of those, 53% either found paid employment or became self-employed prior to completing or immediately following their participation in the program. In comparison, findings from the 2014 TIOW Summative Evaluation national survey indicated that 75% of participants reported finding paid employment when they were surveyed two years following their participation in the TIOW program. Many factors may have contributed to this positive outcome. It is worth noting that the 2014 evaluation carried out an incremental impact analysis that was used to determine how much of those outcomes can be attributed to the program. It concluded that TIOW program participants were more likely than non-participants to find employment by 6 percentage points.
According to key informants, many of the program participants who secured paid employment were working in sectors such as retail, sales and services, health and tourism. Participants were also hired in ‘big box stores’ such as Walmart and Sobeys or found secretarial work in local government, lawyer’s and doctor’s offices, hotels, community organizations, call centres and pharmacies. Some indicated finding full-time work and fewer indicated finding part-time work. Some participants secured employment with the same employer with which they completed their work placement. Key informants noted that factors such as high unemployment hindered participants’ ability to find and secure their employment of choice. Other factors such as seasonal employment (i.e. summer tourist destinations) did not create employment stability for those participants.
According to the document review more than half of the Project Sponsors anticipated providing self-employment activities to participants. Similarly, the information gathered during interviews demonstrates that certain jurisdictions have an interest in providing self-employment activities. The administrative data for British Columbia, Yukon and Nova Scotia indicates that approximately 30% or 194 of 646 participants took part in activities that provided them with assistance in starting a new business.
6. Efficiency and Economy - Performance measurement and program costs
This section examines how the TIOW program’s performance is measured and the extent to which the data captured is valid, reliable and collected efficiently. It also estimates the average cost per program participant.
6.1 Performance Measurement Strategy
The program’s Performance Measurement Strategy was updated in 2013 and outcome statements in the logic model were re-examined in 2014 in light of the new eligibility criteria.Footnote 21 Overall program activities, outputs and general outcome statements remained unchanged.
The majority of the program performance data is gathered through three paper forms. Participant Information Forms (PIFs) are filled in by program participants prior to starting the program. Participant Project Feedback Forms (PEFs) are filled out immediately after the completion of the program. Follow-up Evaluation Forms are sent to former participants three to six months afterwards. Completed PIFs and PEFs are forwarded by Project Sponsors to PTs. Evaluation Forms are typically administered directly by PTs, where after conducting the analysis they may wish to do, PTs send the forms to ESDC. Program officials at ESDC manually enter data from the forms into a database. Most of the key informants from the PTs and Project Sponsors felt that capturing the PIF and PEF electronically would be more efficient.
6.2 Data quality
The program uses employability outcome measures that are similar to those found in the literature, but they may not provide the most accurate measures of employability. For example, program participants are asked to respond yes or no as to whether the TIOW project improved their confidence in finding a job. However, there are several measurement instruments developed and used by scholars which would more accurately measure employability outcomes.Footnote 22 For example, one such tool measures re-employment self-efficacy where respondents are asked to indicate their degree of confidence in being able to find paid employment on four prospect statements using a five-point Likert scale. Another measures job search self-efficacy and asks respondents to indicate their degree of confidence in being able to successfully perform six job search activities using a five-point Likert scale. Given it is the only program in the department to collect data directly from its participants before and after their participation, there is an opportunity to capture informative pre and post observations on behaviours and outcomes of interest.
The completeness of the compiled data was relatively good for demographic variables (age, gender, education, and region). However, outcome variables were poorly populated. For example, more than 50% of the records for the employment variable were missing. Other variables related to earnings (i.e. number of Employment Insurance collections, number of months on Social Assistance or on Employment Insurance) also have many missing values. Participant feedback variables (i.e. participant auto ranking of self-confidence) have approximately three quarters of the values missing, which limits the usefulness of some of the data. In addition, outcome variable data was only accessible for three PTs. In the end, this constrained the analysis evaluators could undertake with the administrative data.
6.3 Program costs
A cost-benefit analysis was not conducted for the TIOW program interventions and services since the evaluation design did not include an incremental impact analysis. Also, it was not possible to calculate the average cost for the overall program because participant data was not available nationally. However, it was possible to make a more limited calculation of the average cost per participant based on contributions made by the federal government for four jurisdictions.
In 2014-2015 and 2015-2016 Ontario, Nova Scotia, British Columbia and the Yukon claimed $15M from the federal government and 2,785 individuals participated in program in those PTs. The estimated average cost per program participant is $5,386 (see Table 4 below). The previous evaluation estimated the average cost at $7,000 per participant.
Fiscal years | Actual expenditures | Total clients | Cost per client |
---|---|---|---|
2014-2015 | $10,690,867 | 2,336 | $4,577 |
2015-2016* | $4,308,234 | 449 | $9,595 (excludes Ontario) |
Total | $14,999,101 | 2,785 | $5,386 |
Source: *Program budget and administrative data for Ontario, British Columbia, Yukon and Nova Scotia however excludes Ontario for 2015-2016 since participant data was not available.
Table description of Table 4
Table 4 presents the TIOW program client and costs summary.
There are four columns. From left to right, the first column, lists two fiscal years (2014-2015 and 2015-2016) and total. The second column lists actual expenditures by fiscal year and for both years combined while the third column presents the total number of clients by fiscal year and for both years combined. The fourth column presents the cost per client by fiscal year and for both years combined. The source of data and the non-response rate are inserted beneath the table.
Since the analysis in Table 4 excludes Ontario for 2015-2016, the difference in the cost per client between 2014-15 and 2015-16 may be explained by varying project delivery costs within a PT.Footnote 23 For example, costs within a PT can be driven by the availability of existing labour market programming infrastructure that can be leveraged by a Project Sponsor in an eligible community. In particular, costs may be higher in more remote communities. In addition, participant allowances may be higher in some PTs or not offset by wages paid from an employer-based work experience or through EI.
7. Conclusions
The TIOW program was implemented as planned. Projects were delivered in all of the participating PTs and the majority of participants were in the primary age range of 55 and 64 years of age. The evaluation also confirmed that the two new community eligibility criteria added in 2014 were used by PTs to expand the reach of the program into small communities that may not have been eligible in the past.
Program outcomes relied mainly on the administrative data from three PTs (British Columbia, Yukon and Nova Scotia), program documents and to a lesser extent on views from key informants from three provinces (Alberta, Manitoba and Nova Scotia). Outcome findings for the three PTs indicate that overall, participants expressed positive satisfaction regarding the program’s job seeking activities and viewed employability activities as helpful overall. Together, this complementary evidence endorses the programs’ principle outcome: to effectively assist unemployed older workers to improve their employability and transition into employment.
The evaluation found that overall program interventions and activities align with evidence based approaches proposed in the literature. It highlights key features of the program that support PTs as they target, design, and deliver projects. It also identifies some gaps that could be addressed in the future when designing labour market programming for older workers.
Evaluation findings also point to lessons learned and include three recommendations to improve the design and delivery of the program. These recommendations propose improvements to services and interventions provided to participants, participant recruitment and assessment and data collection for performance measurement and evaluation purposes.
Recommendations
Overall the program interventions align with evidence-based approaches. The projects provide many types of relevant interventions such as occupational training, job search training and techniques, job development, assessment facilities and community outreach. However, not all Project Sponsors consistently offer interventions to improve job search behaviours.
Recommendation #1:
First, Project Sponsors provided many activities that strengthened job seekers self-efficacy beliefs. However they did not consistently clarify and set job goals for participants. When goals are clearly defined, a focused search strategy generally produces the most positive outcomes.
Secondly, while some Project Sponsors linked participants with employers, others did not. The marketing of participants to employers, which may include linking participants with employers, could help improve access to labour markets.
Finally, providing ongoing formal peer supports during the entire time participants are actively seeking employment is a critical design element. Project Sponsors created supportive environments for their participants and the participants acknowledged this value. Participants indicated forming informal groups to support each other following the end of their participation in programming. However, evaluators observed little by way of formal ongoing group sessions facilitated by Project Sponsors. Providing ongoing support is key given that almost half of participants were not employed following the end of their participation in the TIOW program.
It is recommended that ESDC work with the provinces and territories to develop guidelines for projects to include activities that:
- 1.1. Clarify and set job goals for participants.
- 1.2. Improve access to labour markets by linking participants with employers.
- 1.3. Provide facilitated ongoing group sessions for participants following the end of their formal programming.
Recommendation #2:
One of the design features that labour market programs for older workers should include is the careful identification, screening and selection of participants. This allows program coordinators to stream participants into targeted and tailored programming that meets their specific strengths and addresses their specific weaknesses. However, Project Sponsors do not consistently assess all potential participants. Where they do, the processes and tools used for assessment vary from one jurisdiction to another and some assessments are more robust than others.
The literature suggests that individuals with high levels of education tend to be more self-directed learners and may not respond favourably to TIOW programming. Individuals lacking in social support, that demonstrate symptoms of anxiety and possess lower levels of education benefit from a more structured and formal programming that offers assistance and ongoing support.
Consistently robust assessments of participants would assist in streaming older workers into labour market programming that is more responsive to their needs.
It is recommended that ESDC work with the provinces and territories to develop recruitment guidelines for projects through the careful identification and screening of older workers so that they may be streamed into the most appropriately designed labour market interventions. This may translate into providing less intense programming (i.e. less than 12 weeks) to more highly educated participants through the TIOW program or referring them elsewhere within a PT where other, more relevant labour market programming is available.
Recommendation #3:
The ongoing monitoring of participant demographic characteristics and outcomes between program evaluations is paramount to effectively managing any program. For example, deviations from recruiting the most suitable participants can be identified using data collected on an ongoing basis from participants. If deviations are detected, Project Sponsors can be alerted to ensure program implementation is in keeping with key policy requirements. However the ongoing monitoring of performance requires the systematic, timely and efficient collection of reliable data.
The TIOW program performance data is captured through paper based forms completed by participants immediately before and after program participation. The completed forms are expedited in batches by the PTs to ESDC program officials where they are manually entered into a database. The process is time consuming and prone to input error. The profile variables (age, gender, education, and region) were suitably captured, but the outcome variables were poorly populated, suffering from non-response errors. Furthermore, though the employability and employment outcomes measures captured by the program are relevant, there are more reliable measures which could capture more informative data for both performance measurement and evaluation purposes.
Also, the absence of Social Insurance Numbers (SINs) from participants limits the scope of future evaluations. The collection of the SINs from program participants would allow ESDC to conduct a more thorough analysis of program performance and incremental impact analysis by facilitating participant identification and linkage of the program’s administrative data to other administrative data (e.g. Canada Revenue Agency).
The current system as it exists limits ESDC’s ability to perform adequate monitoring and produces challenges in terms of analysis for the evaluation of the TIOW program.
It is recommended that ESDC improve the quality of the data collected from all Project Sponsors and participants for the continuous monitoring and evaluation of outcomes by:
- 3.1 Capturing more reliable employability and employment measures on the Participant Information Forms, Participant Feedback Forms and follow-up Evaluation Forms.
- 3.2 Examining the feasibility of implementing the electronic capture of its data collection forms from the provinces and territories.
- 3.3 Engaging key stakeholders to explore the feasibility of collecting participants’ Social Insurance Numbers in future years.
Appendix A – Evaluation matrix
Table A Evaluation matrix (in text for web version)
TIOW evaluation questions
Design and delivery
Q1. To what extent is the design of the program appropriate to achieve the expected results? Are program activities logically linked to the production of the expected outputs and results?
Lines of evidence
- Literature review
- Document review
- Key informant interview
Q2. How have provinces and territories implemented the program in communities experiencing skills mismatches and/or unfulfilled employer demand?
- How have they identified and engaged potential employers?
- What are the employment outcomes for program participants?
Lines of evidence
- Document review
- Key informant interview
Achievement of expected outcomes
Q3. To what extent have the expected outcomes of increased employability and/or re-integration into employment been achieved?
- Have participants increased their employability?
- Have participants obtained employment?
- Are there increased employment opportunities in vulnerable communities (i.e. as a result of new businesses being created by participants)?
Lines of evidence
- Document review
- Administrative data
- Key informant interview
Demonstration of efficiency and economy
Q4. Is the Performance Measurement Strategy generating valid and reliable performance data that supports ongoing performance monitoring and decision making?
- Are adequate data collection and reporting systems in place to support the summative evaluation?
Lines of evidence
- Document review
- Administrative data
- Key informant interview
Q5. What are best practices and lessons learned with respect to efficiently delivering the program and economically achieving outcomes?
Lines of evidence
- Literature review
- Document review
- Key informant interview
Appendix B – Methodology
Introduction
The following presents the methodologies used for the Evaluation of the Targeted Initiative for Older Workers (TIOW) program. This Appendix describes the scope of the evaluation, data sources, data collection methods and constraints and limitations.
Scope and breadth
This evaluation is outcomes based and examined the period covering April 2014 to June 2016. It focused on the issues and questions identified in Appendix A. It includes all provinces and territories with the exception of Québec, which carries out its own evaluations, and Nunavut, as it did not participate in the program. However, due to the limited availability of administrative data, the scope related to reporting on program outcomes was restricted to British Columbia, Yukon and Nova Scotia.
Methodologies
The following methods were used to respond to the evaluation questions. All methods were carried out in-house by ESDC Evaluation Directorate staff.
- Document and literature review - This included previous evaluation technical and final reports, a review of national and international literature and program documentation provided by the program.
- Administrative data - Program performance data was generated from the TIOW program Access database. Data captured included information from the Participant Information Forms, Participant Feedback Forms, Participant Evaluation Forms and Project Sponsor information.
- Key informant interviews – Interviews were carried out by telephone or in-person. Respondents included representatives from ESDC National Capital Region program administration (n=5). It included randomly sampled respondents from three provinces (Alberta, Manitoba and Nova Scotia) that consisted of PT program representatives (n=3), project sponsors (n=8), participants (n=10) and employers (n=4) from those provinces. Interviews were carried out by ESDC evaluators between January and February 2016.
Analysis
The data analysis strategy included the triangulation of multiple lines of evidence. The table below describes the proportional and frequency terms used in the report to quantify the extent of agreement amongst interview respondents to specific questions and issues.
Proportional and frequency terms | ||
---|---|---|
Proportion terms | Frequency terms | Percentage range |
All | Always | 100% |
Almost all | Almost always | 80-99% |
Many / Most | Often, usually | 50-79% |
Some | Sometimes | 20-49% |
Few | Seldom | 10-19% |
Almost none | Almost never | 1-9% |
None | Never | 0% |
Table description Proportional and frequency terms
Appendix B includes a table which presents the proportional and frequency terms as well as the percentage range used in the TIOW program evaluation report. There are three columns. From left to right, the first column lists the proportion terms (i.e. All), while the second column lists the frequency terms (i.e. Always) and finally the third column lists the percentage range (i.e.100%).
Constraints and limitations
In 2015, the program was putting in place Information Sharing Agreements between ESDC and the PTs in order to access participant data (PIF and PEF). By February 2016, data from four PTs (Ontario, British Columbia, Yukon and Nova Scotia) was available for evaluation purposes. Readers should be mindful when interpreting findings citing administrative data since it may not be representative of the other provinces and territories participating in TIOW. Furthermore, Social Insurance Numbers are not collected for TIOW program participants. Evaluators were unable to link the TIOW program administrative database (PIF and PEFs) to the Canada Revenue Agency and ESDC databases and apply incremental impact modelling.
Appendix C – Government of Canada funding contribution
Province/Territory | Allocation (2014-2015) |
Allocation (2015-2016) |
Allocation (2016-2017) |
Total |
---|---|---|---|---|
Newfoundland and Labrador | $853,792 | $853,792 | $853,792 | $2,561,375 |
Nova Scotia | $791,923 | $791,923 | $791,923 | $2,375,768 |
Prince Edward Island | $240,000 | $240,000 | $240,000 | $720,000 |
New Brunswick | $841,418 | $841,418 | $841,418 | $2,524,253 |
Québec | $6,545,736 | $6,545,736 | $6,545,736 | $19,637,207 |
Ontario | $8,339,936 | $8,339,936 | $8,339,936 | $25,019,807 |
Manitoba | $507,325 | $507,325 | $507,325 | $1,521,976 |
Saskatchewan | $334,092 | $334,092 | $334,092 | $1,002,277 |
Alberta | $1,917,938 | $1,917,938 | $1,917,938 | $5,753,813 |
British Columbia | $2,907,841 | $2,907,841 | $2,907,841 | $8,723,523 |
Yukon | $240,000 | $240,000 | $240,000 | $720,000 |
Northwest Territories | $240,000 | $240,000 | $240,000 | $720,000 |
Nunavut | $240,000 | $240,000 | $240,000 | $720,000 |
Total | $24,000,000 | $24,000,000 | $24,000,000 | $72,000,000 |
Of the $25,000,000 a year allocated for the TIOW program, $24,000,000 is allocated to PTs while the remaining $1,000,000 is retained by ESDC to support program management, operational, evaluation, and communication activities.
Table description Snapshot of the TIOW program Government of Canada funding contribution for fiscal years 2014-2015, 2015-2016, 2016-2017, by province and territory
Appendix C includes a table which presents a snapshot of the TIOW program Government of Canada funding contribution for fiscal years 2014-2015, 2015-2016, 2016-2017, by province and territory.
There are five columns. From left to right, the first column lists the provinces and territories while the second, third and fourth columns present the allocation amount for each fiscal year (2014-2015, 2015-2016 and 2016-2017) per province and territory. The final column lists the total amount of allocation by province and territory.
Page details
- Date modified: