Evaluation of the Literacy and Essential Skills Program

On this page

• List of abbreviations
• List of figures
• List of tables
• Introduction
• Key findings and recommendations
• Evaluation context
• Program background
• Program design and delivery
• Evaluation findings
• Conclusions and recommendations
• Management response and action plan
• Annex A: Evaluation questions and lines of evidence
• Annex B: Previous evaluation findings
• Annex C: Literacy and Essential Skills Program Logic Model
• Annex D: Skills for Success
• Annex E: Comparison of Skills for Success with the old skills framework
• Annex F: Methodology and limitations
• Annex G: Scale for qualitative result
• Annex H: Additional quantitative and quantitative findings
• Annex I: Learning journey maps

Alternate formats

Introduction

This report presents the findings of the evaluation of the Literacy and Essential Skills Program. The Literacy and Essential Skills Program (hereafter, "the Program") aims to help adult Canadians improve their essential skills levels to ultimately be successful in the labour market. The Program provides funding through the Adult Learning, Literacy, and Essential Skills Program and the National Essential Skills Initiative^{Footnote 1}, which are 2 grants and contributions program streams, collectively run under the Program. Funding is used to:

• develop new tools and conduct research, and
• pilot, replicate, and scale innovative practices to skills upgrading in varying contexts

Budget 2021 announced funding over 3 years, beginning in 2021 to 2022, leading to the launch of the new Skills for Success Program, the successor of the Literacy and Essential Skills Program.

Key findings and recommendations

Below are the key findings from this evaluation:

• overall, resources and support materials to develop the new Skills for Success model were appropriate
• recognition from program officials, Other Labour Market Programs^{Footnote 2}, partners, stakeholders, and experts that the Skills for Success model is sound, is designed to address the evolving needs of employers, and aligns with international essential skills approaches
• funding recipients are making use of Program resources, but may not be fully aware of the Skills for Success model and the tools available to them
• with support from the Program, Other Labour Market Programs have integrated essential skills into their training, supports, and services. Provinces and territories have largely adopted the Essential Skills Framework and developed new or existing essential skills programs, tools, and products
• participants in program-funded essential skills training interventions, including participants from under-represented groups^{Footnote 3}, are satisfied with the training they received. They reported that the training helped them overcome barriers in learning and employment, gain important skills, and increased their sense of wellness and wellbeing
• surveyed participants reported that the most common skills gained as a result of their training were social/interpersonal skills including team working skills (45%), communication skills (40%), and people skills (36%)
• some surveyed participants in program-funded essential skills interventions reported having found a new (31%) or better (16%) job 1-year after their participation in the essential skills training. A majority (57%) of surveyed participants reported retaining their job because of their training
• program-funded essential skills training interventions have successfully targeted multiple populations, including individuals from under-represented groups. However, stakeholders suggested tools and resources can be further adapted to address the needs of under-represented groups
• there is a lack of aggregate-level program data on participants in essential skills interventions, making it difficult to determine overall reach of essential skills training

Recommendations

The evaluation identified the following recommendations for the Program going forward.

Recommendation 1

Continue to strengthen performance management practices with funding recipients, including exploring a way to measure skill levels of training participants to provide regular and robust evidence to support policy analysis and program decision-making.

Rationale for recommendation 1

Evidence suggests that the Program made significant progress in collecting performance data from funding recipients and in providing resources, guidelines, and tools to funding recipients to report on results. Although there are limitations to the Program's role in this area given the funding instrument used to deliver the Program, there is a continued need to encourage funding recipients and Labour Market Programs to collect information on participants' skills acquisition and labour market outcomes, as well as participant demographics with an intersectional lens. This will facilitate the assessment of both participant and employer needs, and help determine the extent to which the Program reaches under-represented groups in the labour market. Experts stated that there is need for determining baseline proficiency levels, particularly for social-emotional skills, for a more objective measure of skills growth than self-assessment tools. This will allow for the implementation of reliable assessment tools pre- and post-training, which will help identify skills gaps and implement efficient, effective, and low-risk approaches to skills training.

Recommendation 2

Increase communication and engagement within Employment and Social Development Canada (ESDC) and with partners and stakeholders, particularly employers, to foster awareness of the Skills for Success Model and the tools to support the implementation of the new model.

Rationale for recommendation 2

The evaluation found that the Program made significant efforts to introduce and expand the application of the new model among partners and stakeholders. However, there is room for increased communication and engagement with partners and stakeholders, particularly employers, to foster awareness of the new model and its benefits. The need to develop and share tools specific to the needs of various sectors of the labour market was also emphasized by Other Labour Market Programs and partners, and Provincial and Territorial partners. Experts also expressed the need for more tools and resources that clearly outline the key similarities and differences between the former Essential Skills Framework and Skills for Success model to better align with the latter. Experts also indicated that more resources, guidelines, and examples are needed to emphasize the value of social-emotional skills, particularly "creativity and innovation" in the new model. It is important to note that the new model was not launched until May 2021; therefore, it is not expected that there would have been results related to this indicator for the current evaluation.

Recommendation 3

Continue to improve access to foundational and transferable skills training opportunities and to raising awareness of these opportunities particularly to under-represented groups.

Rationale for recommendation 3

Essential skills training, funded by the Program, has successfully targeted multiple populations, including individuals from under-represented groups. However, tools and resources can be further adapted to address the needs of under-represented peoples and those furthest from the labour market. Peoples from under-represented groups are less likely to find out about training opportunities. There is therefore a need to create more awareness of training programs and services among under-represented groups. In a survey of participants, many respondents including those from visible minority groups and participants with a disability, reported that the training was not easy to find. Funding recipients also identified the need for more support, resources, and information about assisting newcomers and the need for more networking opportunities to learn about best practices for addressing the needs of under-represented groups.

Evaluation context

Evaluation scope

This evaluation covers a 5 year period from April 2016 to March 2021 and its completion is in compliance with the Financial Administration Act and the Policy on Results. The evaluation questions, as well as the outcomes assessed and lines of evidence employed to address each question are presented in Annex A. The previous evaluation of this Program was completed in October 2017, and it covered the period between April 2012 and March 2016. The results from the previous evaluation are summarized in Annex B.

This evaluation is primarily summative, with a focus on the contributions of the Literacy and Essential Skills Program towards achieving expected outcomes. The Program's logic model is presented in Annex C. This includes the ultimate outcome related to improvements in essential skills and labour market outcomes of participants in essential skills training, to the extent possible.
The first evaluation question is formative, assessing the adequacy of the design of the new "Skills for Success" model (hereafter referred to as "the [new] model"). This new model aims to align with evolving labour market needs, which are characterized by new technology, globalization, and the changing nature of work.

Replacing the Essential Skills Framework, the Skills for Success model was implemented in 2020 and officially launched on May 18, 2021, along with a fully rebranded program called the Skills for Success Program. The new model defines nine key foundational and transferrable skills needed by Canadians to participate in work, in education and training, and in modern society more broadly. This question covers only 1 year of the implementation of the new model.

For an overview of the Skills for Success model, please consult Annex D. Annex E presents a comparison between the Essential Skills Framework and Skills for Success model.

Evaluation approach

The evaluation assesses the Program's contribution to achieving the expected outcomes, including the ultimate outcome related to improvements in essential skills and labour market outcomes of participants in essential skills training.

However, the Program's outcomes cannot be solely attributed to program activities. Because of intervening factors, it is not possible to assess the Program's impact (that is, the causal effects) on participants' essential skills levels and labour market outcomes. For this reason, this evaluation assesses the contribution of the Program towards achieving expected outcomes.

Multiple lines of evidence were used to respond to the evaluation questions. This evaluation employed a mixed-methods approach including several lines of evidence. A description of the methods and their associated limitations are provided in Annex F. It is important to note that pilot project performance results data were only available for the second half of the evaluation. Furthermore, project data was qualitative and lacked a consistent reporting style. Efforts were made to manually extract and aggregate data whenever possible.

The findings from each line of evidence were triangulated to minimize bias and validate their consistency. Annex G presents the scale used to weigh the significance of qualitative findings.

The evaluation conducted a Gender-Based Analysis+ to assess how people from under-represented groups experienced access to essential skills training differently and to identify potential barriers. The evaluation considered different identity factors in the analysis, such as gender, disability, ethnicity, and race. To support this analysis, focus groups and individual interviews sought specific under-represented groups.

Program background

The Literacy and Essential Skills Program aims to help adult Canadians improve the essential skills needed to be successful in the labour market.

Literacy includes:

• reading
• writing
• document use, and
• numeracy

While essential skills go beyond literacy to include:

• thinking skills
• oral communication
• computer use/digital skills
• working with others, and
• the skills associated with continuous learning

These are defined as the skills required to participate, adapt, and thrive in learning, work, and life.

In the wake of Budget 2021, the Program modernized its Essential Skills Framework and rebranded it as the Skills for Success model. The Model was implemented in 2020 and officially launched on May 18, 2021, along with a fully re-branded program called the Skills for Success Program.

Program design and delivery

The Program provides approximately $25 million annually through 2 main vehicles to fund projects aimed to help Canadians improve their essential skills. Namely, they are the Adult Learning, Literacy and Essential Skills Program^{Footnote 4} and the Employment Insurance Part II – National Essential Skills Initiative^{Footnote 5}. Funding is used to:

• develop new tools and conduct research
• pilot, replicate, and scale innovative practices to skills upgrading with Canadians in varying contexts

Pilot projects test new ways of delivering essential skills training to Canadians and measuring results. Their goal is to improve the quality of essential skills training to meet job seekers', workers', and employers' needs. The Program does not generally fund organizations for the delivery of ongoing training since its primary focus is on funding pilots, many of which include training.

Regarding support for ongoing training, the Program also aims to contribute to the integration and adoption of successful pilot projects. This is referred to as "project sustainability." Funding recipients work collaboratively with partners^{Footnote 6} and stakeholders^{Footnote 7} to test innovative practices. The goal is that once a project ends, the funding recipient or strategic partner(s) will take on the lessons learned and promising practices, and continue to deliver tested practices within their own programming.

Since 2016, project sustainability is an integral consideration prior to making funding decisions. All funding recipients and their strategic partners must have a sustainability plan in place demonstrating how the project will thrive in the long term with the support of other funding sources. It is important to note that funding recipients are considered stakeholders, but not all stakeholders are funding recipients. "Funding recipients"^{Footnote 8} will be used when referring specifically to organizations that have received funding from the Program.

Furthermore, the Program engages with various strategic partners and stakeholders involved in the design and delivery of Labour Market Programs to embed essential skills into their policies and program design. These include federally-funded Labour Market Programs and federal labour market transfer agreements. Engagement also includes working bilaterally with provinces and territories to gather and share information on promising practices and innovative approaches for integrating essential skills into their labour market programming.

Building on the former Essential Skills Framework, the Skills for Success Program continues to fund organizations to design and deliver skills training to help job seekers and workers develop strong skills, become more resilient to labour market changes and have better employment outcomes. These projects support all Canadians, including those facing barriers to employment, to improve their foundational and transferable skills.

Stakeholder engagement activities

The Government of Canada provides approximately $3 billion for skills and employment programs and services through the Labour Market Development Agreements and the Workforce Development Agreements with provinces and territories. The Program engages with employers, industry associations, post-secondary education institutions, and skills training providers. The Program supports active federal labour market programs such as Labour Market Development Agreements and Indigenous Skills and Employment Training Program. Training and employment support services are largely delivered by provincial and territorial governments and supported by additional investments totalling $2.7 billion between 2017 to 2018 and 2022 to 2023. Other project-based programs also provide project-based funding to organizations to respond to the needs of under-represented groups.^{Footnote 9}

Organizational structure

The purpose of the Adult Learning, Literacy and Essential Skills Program, is to provide funding, through grants and contributions, to primarily support the testing, replicating and scaling up of effective and innovative training models that aim to help Canadians improve their essential skills. To manage its strategic investments through Grants and Contributions, the department adopted a collaborative approach between the Program Operations Branch and the Skills and Employment Branch to deliver the Program. The Program Operations Branch is responsible for program delivery, which involves the effective administration of Grants and Contributions throughout the project life cycle. The Literacy and Essential Skills Program within the Skills and Employment Branch focuses on program policy, planning and design, and results measurement.

Program budget

The budgets for the Adult Learning, Literacy and Essential Skills Program (ALLESP) and Employment Insurance Part II – National Essential Skills Initiative (NESI), the Grants and Contributions programs, for fiscal years 2016 to 2017 to 2020 to 2021 are shown in Table 3-1.

Source: ESDC Corporate Management System

Budget 2021 announced funding over 3 years ($298M in addition to $25M ongoing), beginning in 2021 to 2022, in a new Skills for Success Program, the successor of the Literacy and Essential Skills Program.

Evaluation findings

4.1 Rationale for the Skills for Success Model

Key finding

Stakeholders, partners and experts reported the new Skills for Success model to be sound and expect it will respond to the evolving needs of workers and job seekers. However, it was suggested that the new approach will be more impactful with increased employer awareness of and access to reliable assessment tools to help identify skills gaps.

This section focuses on the Skills for Success Program. It responds to formative Evaluation Question 1, which assesses the extent to which the new Skills for Success model responds to the evolving needs^{Footnote 10} of the labour market.

In May 2021, the former Essential Skills Framework was updated and rebranded by the new Skills for Success model. The new model incorporates 2 new skills:

• adaptability, and
• creativity and Innovation

It has expanded the following skills:

• problem solving
• communication
• collaboration, and
• digital

and updated the following skills:

• reading
• writing, and
• numeracy

For a full list of skills in the new model, please consult Annex D. The key differences between the framework and model are highlighted visually in Annex E.

The first evaluation question provides an opportunity to inform the design and implementation of the new Program, and highlights areas for consideration.

Skills for Success Model and the Canadian labour market

Labour market pressures are affecting all sectors of the economy and most regions of the country. As of March 2022, there were more than 1 million job vacancies across Canada, almost double the number prior to the Covid-19 pandemic. Demographic shifts that are rooted in an ageing population and expected retirements are also going to put additional pressure on the labour market over the long term. Over the 2019 to 2028 period, a decline of 1.5 percentage points in the labour force participation rate is projected. ^{Footnote 11} Skills training opportunities can help to mitigate labour market pressures arising from labour market shortages due to inadequate skills and/or skills mismatches.

Skills for Success Model includes foundational and transferable skills

Foundational and transferable skills training are well recognized as supporting:

• social development
• social well-being
• income security, and
• improved employment outcomes  

They are "relevant across the range of life contexts, from education through work to home life and interaction with public authorities."^{Footnote 12} Foundational and transferable skills are not job-specific skills, but rather the basic skills needed to navigate a culture and society, and upon which to build all other skills. These skills are particularly important for under-represented groups or those facing multiple barriers. For example, newcomer women require support to ensure they have the necessary basic skills to function in a new country, ranging from understanding their rights, registering their children to school, accessing job training, or taking medicine as prescribed. For example, literacy is defined as "the ability to understand, evaluate, use and engage with written texts to participate in society."^{Footnote 13}

Foundational and transferable skills also include:

• building literacy and numeracy
• social-emotional skills, (such as adaptability, creativity and innovation), and
• problem-solving

Having these skills supports confidence, and overall resilience.

The OECD's 2012 Program for the International Assessment of Adult Competencies (PIAAC) survey for Canadians aged 25 to 65, the most recent international survey, showed that 49% of Canadians scored at or below level 2 for literacy and 55% scored at or below level 2 for numeracy on the 5-level scale. Level 3 is generally the threshold required to compete in a knowledge-based economy.

Women score evenly with men in literacy, but lower in numeracy. There are more women scoring at or below Level 2 and at or below Level 1 (about 10 percentage points difference in both cases). Disparities in foundational and transferable skills also exist for groups facing multiple barriers. For example, off reserve Indigenous women score lower than non-Indigenous women in numeracy and literacy, but literacy scores are greater than non-Indigenous males. While statistically similar to their male counterparts in literacy, immigrant women score lower than non-immigrant women in Canada and lower than their male counterparts in numeracy.

Overall, stakeholders found resources and support materials to develop the new Skills for Success model to be appropriate. Other Labour Market Programs and program partners indicated a need for tools to support the alignment of their products with the new model.

Evidence shows that the Program engaged and consulted with experts, partners, and stakeholders to inform the Framework renewal. This process was comprehensive and backed by substantial research. Engagement activities took place between June 2019 and March 2020.

Interviewed stakeholders on the Framework renewal described the resources or support materials provided by the Program to develop the new model as relevant or appropriate. It was mentioned that the Program put together documents to facilitate discussions and that the material addressed important skills, such as adaptability and entrepreneurship.

Material distributed by the Office of Literacy and Essential Skills was "well-disseminated, germane, and easily digestible."

Labour Market Programs and program partners indicated a need for tools to support the alignment of their products with the new model.

Overall, stakeholders, partners, and Labour Market Program representatives appreciated the opportunity to provide input on the new Skills for Success model. Provincial and Territorial partners shared mixed views about the engagement process.

All 9 stakeholders interviewed who were involved in the Essential Skills Framework renewal expressed satisfaction with the opportunity to share their input and suggestions. Some described the meetings as "focused, pragmatic, and grounded," "skillfully run and well structured," and inclusive of members with different perspectives.

Overall, partners pointed to positive elements of the engagement process. They received requests for feedback and saw evidence of the new model's relevance. They also highlighted ongoing conversations about evaluation tools for under-represented populations. Meetings at multiple levels took place with contributions from external partners to inform the new Framework's design.

Provincial and Territorial partners shared mixed views. Half of those interviewed (2 out of 4) were not fully satisfied with the engagement process. They highlighted a lack of opportunities for engagement at the launch of Skills for Success model. Others described the process as responsive, respectful of different points of view, thorough, conducted across multiple jurisdictions, and considerate of feedback and input.

The Skills for Success model is expected to respond to the evolving needs of workers and job seekers, and there is widespread recognition by stakeholders that it is sound.

Stakeholders reacted positively when asked about the new skills^{Footnote 14} identified under Skills for Success model, finding them to be important and the model to be sound. They routinely suggested that the new model is expected to respond to the evolving needs of workers and job seekers by helping them understand the skills needed by employers. Responses include:

The new approach [model] was "long overdue," "going in the right direction", "up-to-date approach", "a sound basis", "based on consultation and supported by research", and "excellent Framework"

Stakeholders indicated the model allows job seekers to:

• gauge their current skill set
• determine where they are deficient, and
• look for training to address their deficiencies

Stakeholders suggested that awareness of the importance of essential skills and training would facilitate communication between prospective employers and job seekers by ensuring a common understanding of job-related requirements and expectations. In the following example, this recipient was able to identify that soft skills were needed to be more job ready. As one stakeholder put it:

"These are the basic building blocks and softer skills are often the pre-requisite for developing other skills including certain technical skills".

Stakeholders noted that employers can provide training for specific technical skills but not for socio-emotional skills.^{Footnote 15} These are skills employers expect employees to have.

Some program officials (6 out of 14) agreed that the new model responds to the needs of workers and job seekers. They said that the new model is designed to help individuals prepare for and succeed in the workplace by helping them identify and develop the necessary skill sets.

Overall, experts agree that the Skills for Success model allows job seekers and workers to better judge their skills strengths and weaknesses and how their existing skills relate to prospective job requirements. Ultimately, this helps them navigate the labour market and their careers.

A total of 6 experts^{Footnote 16}, including practitioners and academics, were brought to an expert panel discussion and asked about the Skills for Success model.

Experts agreed that the new model provides job seekers and workers with an understanding of the different skills that are needed to succeed in today's labour market. For example, one expert said:

"The Skills for Success framework [model] provides an opportunity to develop clear definitions and assessments for the development of skills, which is critical to ensuring consistent quality in the skills and employment ecosystem."

Three experts highlighted that the skills needed by job seekers and workers have changed as a result of "a globalized economy". Experts further highlighted that these changes result in an increased demand for advanced levels of:

• literacy
• numeracy, and
• social-emotional skills

All of the above are included in the new model.

Key informants emphasized that the new model needs to be sensitive to the needs of various groups of workers and job seekers, including under-represented groups and those furthest from the job market.

Interviewed stakeholders emphasized that shortcomings in essential skills are more acute among people from under-represented groups. They suggest tools will have to be adapted to the specific needs and circumstances of these groups. The new model was developed with the needs of peoples from under-represented groups in mind. A gain in essential skills is expected to lead to:

• more confidence
• self-assurance
• empowerment, and
• better self-advocacy

Almost all program partners believed the new model to be meeting the evolving needs of workers. Some noted the benefit from emphasizing social-emotional skills training, especially for the workers furthest from the job market. One interviewee stated that the extent to which the model addresses the worker's or potential worker's needs depends on both:

• the degree to which essential skills are embedded into training, and
• the degree to which labour institutions, unions, and labour market participants are connected

Most program officials and several experts emphasized the importance of the new approach model being adaptable to the needs of different groups of workers and job seekers, including peoples from under-represented groups. Program officials reported that the new model can be implemented under varying contexts and adjusted to align with the needs of learners from under-represented groups, including Indigenous contexts.

One expert found that the Skills for Success model's skills definitions and component descriptions were developed to be "flexible and inclusive" as well as to "reflect on the diversity of lived experiences of Canadians with an emphasis on inclusivity". Thus, it provides a "model that can be customized and contextualized to build skills that help vulnerable populations to address evolving needs".

Two experts highlighted the importance of adjusting the delivery and presentation of the skills to the needs of different under-represented groups so that it reflects their realities.

Overall, stakeholders found the new model to be well-researched and aligned with labour market needs. Experts indicated that employers will need access to reliable assessment tools to help identify skills gaps and implement effective approaches to skills upgrading.

The document review shows that the Program consulted employers when creating the new model and found that 67% of them ranked social-emotional skills as the most important skills gap when recruiting new employees. Social-emotional skills are relevant to employers across sectors and are in line with emerging needs resulting from the COVID-19 pandemic. According to the majority of interviewed program officials^{Footnote 17}, the circumstances of the labour market following the pandemic accelerated the need to implement the new model.

Among 11 employers who responded to the survey, 10 indicated that the following 3 new skills would either completely or somewhat meet their needs:

• adaptability
• problem solving, and
• communication

Similarly, most employers (9) said the following 3 other skills would either completely or somewhat meet their needs:

• writing
• reading
• numeracy

Most program officials indicated that the new model helps individuals focus on the skills that are in-demand in the labour market. All agreed that the new model will respond to the needs of employers, and the majority (5 of 7) indicated that it was designed based on employer feedback. Furthermore, Social Research and Demonstration Corporation research suggests that Skills for Success aligns well with employer needs and that the conceptualization of skills, as repeatable^{Footnote 18} processes or behaviours, provides the necessary language to generate employer buy-in.

Stakeholders suggested that the new model provides employers with a conceptual skills model and a common language that allows them to clearly identify and articulate their essential skills needs. For example, the new model allows employers to:

• precisely identify the types and the degrees of skills they need
• validate and verify if their current and prospective employees have needed skills
• assess the strengths and weakness of their employees' skill sets, enabling them to focus on areas for growth

The majority of Labour Market Program representatives and partners (3 of 5) and half of interviewed Provincial and Territorial representatives (2 of 4) likewise felt that the model addresses the evolving needs of employers. Both groups also specifically noted that the emphasis on social-emotional, digital, and transferrable skills, especially in light of the pandemic, was positive.

A Labour Market Program representative also pointed out the importance of raising awareness of the Skills for Success model among employers. Similarly, one of the provincial and territorial representatives stated improvements are necessary to make the language, rather than only the content, relevant to employers.

Most experts indicated that Skills for Success model presents an important tool that employers can use both in recruiting efforts and in training programs to foster skills development.

One expert stated that:

"Employment agencies can use this model to help unemployed people better understand the issues and expectations of the labour market."

Another expert underlined the fact that by including social-emotional skills, the new approach gives employers leverage to identify these skills as essential for successfully completing work-related tasks. To ensure relevancy and enable employee upskilling and retention, one of the experts noted that the new approach should be presented in a way that aligns with the employers' context.

One expert emphasized that while the Skills for Success model is useful for employers, they will need access to reliable assessment tools, similarly to job seekers and workers. This would help employers identify skills gaps and implement "efficient, effective and low-risk approaches to skills upgrading," particularly for social-emotional skills. Another expert mentioned that assessment tools will need to be user-friendly and accessible to the workforce in a variety of settings to ensure that they are implemented "as close to reality as possible."

In general, stakeholders, partners and experts expect the new model to be more impactful with increased employer awareness and emphasis on pre- and post-assessments to determine participants' proficiency levels and areas for improvement, where applicable.

Several experts believe that the impact of the new Skills for Success model will be limited until job seekers and workers are able to assess their current skill level easily and objectively. This would enable the identification of skill gaps and point to specific skills that need upgrading to ensure labour market success.

Experts also emphasized the importance of pre- and post-assessments to determine participants' proficiency levels. One expert stated that pre-assessments are useful to identify the areas and skills that need improvement and that determining baseline proficiency levels can enable a more objective measure of skill increase. However, another expert said that while proficiency in certain skills (such as basic literacy and numeracy skills) can be objectively measured, other skills (such as social-emotional skills) are more likely to be subjectively determined by employers, resulting in challenges with accurately measuring skill gain.

Experts expressed that the new approach is promising, but not yet widely known or applied. While this may be partially explained by the recency of the approach, an expert observed a lack of employer engagement, especially with small- and medium-sized enterprises. According to this expert,

"One of the most significant problems in the ecosystem is the number of programs which have not got significant employer engagement."

One expert stated that organizations delivering essential skills training to individuals in pre-employment have "zero connections" to employers, which makes them question the extent to which essential skills are "understood and appreciated by employers." Experts noted the importance of ensuring that skills development training is connected to employer and employment needs.

Experts also cautioned against delivering essential skills that are not in demand or relevant to the needs of employers. Jobseekers and workers who learn these skills may not apply them and eventually "lose" them.

Stakeholders and Labour Market Program partners provided several recommendations for increasing employer awareness of the approach's benefits. Overall, the approach will be effective as long as there is buy-in from employers and to the extent that employers can adapt it to their individual needs and circumstances.

The Skills for Success model aligns with other essential skills approaches or frameworks. Overall, there is a need to develop ways to implement the approach to meet the specific needs of various audiences.

Interviewed stakeholders observed that the approach compares favourably with other approaches. It was noted that the Program did a lot of comparative research to validate the new approach. Specifically, stakeholders indicated that:

• the model compares well with models in other commonwealth countries, although it still needs instructional content to be developed
• the model aligns with other international models, including the Organization for Economic Co-operation and Development (OECD) models and the Programme for the International Assessment of Adult Competencies (PIACC). It is not yet clear how and if it will differ from others with regards to the measurement of new social and behavioural skills

Almost all interviewed stakeholders said that they plan to implement the Skills for Success model. Overall, there is a need to develop ways to implement the model to meet the specific needs of their audience. Some were unsure of how to operationalize the model by developing tools applicable to the industry or trade.

In interviews with program officials, Labour Market Program representatives and partners, and Provincial and Territorial partners, most reported that extensive research and consultations with various jurisdictions and organizations^{Footnote 19} had been undertaken to ensure that the new model aligned with other skills models. Furthermore, in 2 interviews, the Skills for Success model was said to place Canada ahead of the USA and International Labour Organization. Nevertheless, some partners felt the Skills for Success model "didn't feel ready" at launch due to missing guidelines and direction, and pointed to added pressures associated with the timing of the launch.

One expert flagged the lack of "valid, reliable, and interpretable measures" of the social-emotional skills included in the new model. Another suggested that Canada should learn from the "knowledge, insights and best practices" of some G7 countries who have integrated skills models into all spheres of the economy and academic pathways.

4.2 The Program's contribution to identifying, testing and implementing innovative models in essential skills training

This section relates to Evaluation Question 2 which assesses the extent to which the Program supports partners and stakeholders in identifying, testing, and implementing innovative models in essential skills training.

It relates to the Program's expected intermediate outcome "Partners and stakeholders leverage essential skills resources, products, tools and innovative practices in their training curriculum and interventions".

Please note that the remainder of this report focuses on the Literacy and Essential Skills Program, before the introduction of the Skills for Success Program, and therefore the focus of the following sections will be on assessing the achievement of outcomes under the former Essential Skills Framework.

The Program funded different kinds of projects over the evaluation period.

The Program funded 70 projects over the evaluation period to develop, test, and implement innovative models in essential skills training. These projects fall under the former Program's skills Framework.

While not officially defined as such, the Program's funded projects generally fit into 3 types or categories:

• 72% of projects focused on training^{Footnote 20}
• 24% focused on research^{Footnote 21}
• 24% focused on support^{Footnote 22}

In addition to the funding received, funding recipients appreciated the support provided by the Program, including network facilitation, and flexibility offered during the pandemic.

Funding was the most frequently identified kind of support received from the Program, with a few funding recipients characterizing it as the only or primary type of support. Interviewees mentioned:

• Facilitating networking: The Program helped funding recipients make connections with other organizations and stakeholders
• Providing flexibility: The Program was often described as very flexible and accommodating when project changes were required, especially with projects facing challenges related to the COVID-19 pandemic

Funding recipients are making good use of some of the Program's resources, but may not be aware of all the tools available to them.

Key informant interviews with program officials and an examination of program documents reveal several resources produced by the Program to support partners and stakeholders.

In addition, the Program has put on events to further support partners and stakeholders. For instance, some program officials referred to the Essential Skills Forum as the most common setting for facilitating partnerships between stakeholders prior to the pandemic.

Program officials referenced the Essential Skills Playbook most frequently, but funding recipients did not mention it at all when asked which Program resources they used. The only Program resources identified by interviewed funding recipients when asked were:

• the essential skills profiles
• the essential skills assessment template, and
• the former Essential Skills Framework

Interview participants also referred to resources from some of the organizations that the Program funds for research, and a few participants also obtained research reports through reaching out to the Program.

The Program assisted funding recipients facing challenges posed by the pandemic.

The only challenges faced near-universally by funding recipients who were interviewed for this evaluation, related to the COVID-19 pandemic. Primarily, the challenges related to pivoting to an entirely online format for project training delivery and dealing with reduced staff.

Addressing challenges posed by the pandemic was the most common reason for funding recipients to receive assistance from the Program. Typically, it involved amending the terms of their contribution agreement, such as:

• the timelines
• deliverables, and
• allocation of funds

Key informants routinely described the Program as 'very flexible' and 'accommodating' when it came to making these kinds of amendments. Beyond assistance and flexibility with amending agreements, funding recipients described their Literacy and Essential Skills Program analysts as 'helpful'.

Funding recipients highlighted the following ways that their Program analyst helped them:

• transition to a fully online training format
• define domains for new skills
• connect with others facing similar challenges who may be able to assist
• resolve inter-jurisdictional issues between levels of government related to the project

While a few funding recipients did not need assistance from the Program to address these challenges, none of the key informants identified any other types of support the Program could provide to help them in the future.

4.3 The Program's influence on the integration of proven approaches into to essential skills training

This section relates to Evaluation Question 2.1 which assesses the extent to which the program influenced partners and stakeholders to integrate proven approaches to essential skills training.

As it is a sub-section of Evaluation Question 2, it also relates to the Program's intermediate outcome "Partners and stakeholders leverage essential skills resources, products, tools and innovative practices in their training curriculum and interventions".

The Program has a few key channels for communicating proven approaches to stakeholders.

Key informant interviews with program officials and a review of program documents reveal that program communication and engagement activities with partners and stakeholders occur through a few key channels. In the absence of pre-determined benchmarks for communication, it is important to note that this section is simply describing the process and not assessing whether communication strategies were implemented as planned.

First, the Program communicates on a regular basis with funded project recipients through bilateral meetings^{Footnote 23} that are separate from the regular monitoring cycles performed by the Program Operations Branch.

Prior to the pandemic, the Program held the Essential Skills Forum. This annual event (since 2017) brought together government representatives, service providers, practitioners, researchers, evaluators, and employers. Over 100 participants attended the most recent event in 2019.

Also, program officials identified a panel of program managers and analysts responsible for identifying proven approaches and sharing this information with partners and stakeholders.

Finally, the performance measurement working group allows project recipients to network and learn about the successful approaches of other projects.

Partners and stakeholders indicated the need of a more active, consolidated and structured line of communication with the Program.

Funding recipients, partners in Other Labour Market Programs, and Provincial/Territorial partners perceived the Program's communications activities, especially as they relate to proven approaches to delivering essential skills training, differently.

Funding recipients

All funding recipients interviewed said that they communicated with the Program from the point of signing the contribution agreement to the delivery of their project activities. Overall, they unanimously described the Program's communications and engagement activities as 'helpful.'

Other Labour Market Program partners are aware of the communication activities of the Program, but they also describe different experiences and some communication gaps. Partners, interviewed for this evaluation, discussed internal communications rather than specific formal activities delivered by the Program. While some interviewees could describe specific formal communication activities, there were a few who were either not aware of the Program's communication or felt that certain activities (such as communities of practice) had suffered during the pandemic.

Two partners recommended a more active and structured line of communication with the Program. This would allow for both:

• synergy along project lines
• increased transparency to avoid last-minute consultation

When asked specifically about their satisfaction with Program communication activities, interviewed partners again gave mixed responses. Interviewed partners reported a reliance on other opportunities to gather insight on proven approaches to essential skills training.

Provincial and Territorial partners

When asked about their levels of satisfaction with Program communication and engagement activities, Provincial and Territorial partners also gave mixed responses. Satisfied partners usually referred to the "open collaborative working relationships" fostered through communities of practice and the Essential Skills Fora. The main dissatisfaction coming from the majority of these partners (2 out of 3) arose from a perceived lack of a consolidation of program communication and interactions^{Footnote 24}. For example, 1 provincial/territorial partner indicated inconsistencies in communication across different aspects of the program, indicating that it could be more fluent and cohesive.

There is a lack of awareness by partners of Program materials related to proven approaches and a need to consolidate tools to facilitate access to this type of information.

According to program officials, the basis of the Program is to help partners and stakeholders develop, test, and implement proven approaches in essential skills training. This is primarily done via multi-year project funding, but the Program also looks to increase stakeholder capacity by creating opportunities for information exchange. The Program has developed resources, products, and tools that are used to communicate proven approaches with partners and stakeholders.

Key informant interviews with Other Labour Market Programs and Provincial and Territorial partners did not reveal overwhelming evidence of Program support in relation to proven approaches. Program support was mentioned in only 2 interviews with other Labour Market Program partners. Provincial and Territorial partners did not have any substantial examples of Program support.

When asked about Program resources to support the integration of proven approaches of delivering essential skills training, the results were again mixed. For instance, of the two Other Labour Market Program partners who spoke on the issue, one felt they had sufficient information on proven approaches and where to find additional help. The other partner saw a disconnect between the Skills and Employment Branch and the Program Operations Branch and felt that resources were limited.

The results were also mixed for Provincial and Territorial partners. Half of them (2 out of 4) felt they could not utilize documented evidence relating to proven approaches. Some of the barriers that they identified included:

• the lack of awareness of Program materials specifically related to proven approaches
• time constraints in looking through multiple different tools
• a need for a single location where evidence on proven approaches can be compiled
• missing Skills for Success documentation on assessments of the skills in the new model

Funding recipients pointed to the Program supporting them in learning about proven practices in essential skills training. Funding recipients routinely reported that the Program's support helped them identify or learn about proven approaches in essential skills training. They described the support as 'indirect,' in that the Program (in their awareness) was not the source of any proven approaches or practices but supported their learning about these practices. Specifically, these supports included:

• funding pilot projects to help validate proven approaches in the delivery of essential skills training
• sharing information, including the list of Program-funded projects, related to the delivery of services and training
• providing networking opportunities through annual meetings and fora to help build new partnerships with stakeholders

Overall, almost all funded project recipients were aware of proven approaches or promising practices from successful pilot projects. This was attributed to:

• participants' experience in the field (including their own work)
• research
• the work and partnership of other institutions and organizations, and
• shared knowledge from connections and networking opportunities

There is evidence that program activities have supported the development of partnerships and, in some cases, the potential integration of proven approaches.

When asked if Program activities led to the development of partnerships leading to the integration of proven approaches, program officials pointed to their facilitation mechanisms. The following were listed as means through which the Program promotes the integration of essential skills proven approaches:

• the Essential Skills Playbook
• the Essential Skills Forum
• the establishment of working groups and committees, and
• bilateral conversations with funded organizations

When representatives from Other Labour Market Programs were asked a similar question, none of the respondents could give evidence of partnerships leading to the integration of proven approaches. Nevertheless, partnerships were formed that did not necessarily lead to approaches being integrated. Half (2 out of 4) of the interviewees thought there were benefits from these partnerships, but that the connections were not well established, or the Program had to be routinely pursued for interactions. The other half presented generally positive connections being formed through regular policy channels.

Finally, when the question was posed to Provincial and Territorial partners, half of them (2 out of 4) could describe examples leading to the integration of proven approaches.

• In one example, a partnership established with another province via a Community of Practice led to the signing of a 3 year funding agreement with the Program on a project for individuals with learning disabilities.
• In another, a partner fostered multiple partnerships with people they met at the Essential Skills Forum. One of which led to a follow-up contract through their pilot project. Another allowed them to gain knowledge around creative thinking skills from a third-party organization.

There is a continued need to encourage funding recipients, and other interested parties to develop sustainability plans.

Pilot project replication is not formally monitored, but some evidence indicates that it occurs. Interviews with program officials revealed that there are no formal mechanisms to track projects after the funding ends.

The majority (4 out of 6) of program officials interviewed indicated that potential funding recipients are asked to provide a 'sustainability plan' explaining how their project could be sustained beyond the funding period. These reports are not monitored once the project is complete, but most program officials said that the Program informally interacts with recipients after the funding period and has anecdotal evidence of project continuation.

An analysis of the Program's administrative data found that out of the 70 active projects over the evaluation period, about 40% were replicated and/or expanded.

However, given the comments from program officials above, it is unclear whether these numbers are comprehensive.

Overall, there is a lack of data available to inform the status of successful projects, over time, after their funding ended.

However, approximately half of funding recipients said that they or their organization discussed a plan to sustain their project with the support of partners and/or stakeholders (for example,  post-secondary institutions, non-profit organizations). Funding recipients who said they have not (yet) discussed a plan to sustain their project explained that their project is not far enough advanced for such discussions to take place.

4.4 Program's influence on the integration of essential skills into Other Labour Market Programs

This section relates to the Evaluation Question 3.1 which assesses the extent to which the Program influenced the integration of essential skills into other Labour Market Programs.

This relates to the Program's immediate outcome 3.3 "Other Labor Market Program's integrated essential skills into their policy and program design."

Evidence shows that Labour Market Programs have integrated essential skills into their training, supports, and services.

Program officials, Labour Market Programs, and partners who participated in key informant interviews reported essential skills integration in training and support delivered by Labour Market Programs across the country. Almost all (6 out of 7) Labour Market Programs and partners reported some level of integration of essential skills trainings and supports.

Several government-funded Labour Market Programs managed by Provinces and Territories have incorporated essential skills into their training programs and services. A review of program documents revealed a list of the programs implementing literacy and essential skills into programming, including provincial and territorial transfers^{Footnote 25} and other programs with direct federal funding from the Program^{Footnote 26}. Some specifically target essential skills development, while others provide adjacent training.

Provincial and Territorial partners, interviewed for this evaluation, reported that further integration is at their discretion according to what makes sense for their labour market. Essential skills are also included in the Performance Measurement Framework strategy in place within jurisdictions, as well as when Provinces and Territories report on programs offered under Labour Market Development and Workforce Development Agreements. Essential skills were reported as being well-integrated into industries' occupational standards and job profiles. There was also enthusiasm regarding the integration of essential skills into other labour market tools and working collaboratively with the Program on other opportunities to integrating essential skills.

The Program has provided several opportunities for collaboration and engagement with Other Labour Market Programs.

During the evaluation period, the Program established 3 new fora to facilitate information-sharing and collaboration among stakeholders and to implement essential skills into labour market and adjacent programming.

The Program engaged with partners and stakeholders through working group meetings to:

• share essential skills content
• raise awareness about innovative and proven essential skills approaches, and
• influence the integration of approaches to meet the needs of peoples from under-represented groups

Program documents revealed several other examples of collaborations. For example, the Program held meetings with various groups (for example, provincial representatives, employment services, Labour Market Programs, and other government programs) to promote learning opportunities, the use and awareness of assessment tools and resources, and performance management strategies.

Program officials referred to collaboration and engagement activities with various groups. That said, they emphasized the importance of continuing to engage with partners and stakeholders and promote awareness of the new approach.

One representative from Other Labour Market Programs reported having received limited Program support in the past. They suggested that the Program avoid focusing on incorporating essential skills as a requirement of projects funded through other Grants and Contributions programs without providing support on how to do so. Two program officials suggested the integration of essential skills into their program's Calls for Proposals.

4.5 Influence of the integration of essential skills on access to essential skills training, including for people from under-represented groups

This section relates to Evaluation Question 3.2 which evaluates how the integration of essential skills influenced access to essential skills training, including for people from under-represented groups.

Approximately half of program-funded projects targeted members of under-represented groups, but, despite recent improvements, aggregate-level data related to participants in essential skills interventions is still not available.

Program administrative data identifies 5 categories of under-represented groups, including women, youth, Indigenous Peoples, newcomers, and members of an Official Language Minority Community. The following is the breakdown of under-represented groups amongst the funded projects:

• 31 targeted Indigenous People (44%)
• 13 targeted youth (18%) and 13 targeted newcomers (14%)
• 14 targeted women (14%), and
• 6 targeted Official Language Minority Communities (8%)

The Program made improvements starting in 2019 to 2020 in providing instructions and definitions of indicators for funding recipients to submit demographic information on project participants. These reports include a breakdown of:

• Indigenous Peoples
• newcomers, and
• persons with disabilities

However, the administrative data review showed a lack of consistency in the way the information was shared by different funding recipients who led the projects, thereby preventing the aggregation of results. There is therefore a need to clarify requirements provided to funding recipients on reporting on participant-level data to identify overall reach, including to peoples from under-represented groups.

Funding recipients do not rely on the Program for information that could assist their efforts with respect to under-represented groups, but there is room for both sharing available information with funding recipients regarding assisting newcomers and encouraging networking among funding recipients.

Most funding recipients said they neither received nor sought any information from the Program to help them better respond to the needs of under-represented groups. One interview participant described the Program as 'very hands-off' in this regard but available if you needed them. The few who did seek information obtained resources regarding best practices in dealing with members of vulnerable groups and contact information for other organizations or practitioners with experience in this area. In each case, the information was described as 'helpful'.

A few interviewed funding recipients identified information and support they would have liked to receive from the Program. This included:

• Information, support, and advice about assisting newcomers who need support with language, childcare, transportation, and accessing online tools
• Help networking with others to learn from about best practices for responding to the needs of members of under-represented groups

There is insufficient data to provide clarity regarding employers' access to training and associated needs.

Most employers (6 of 7 respondents) reported hearing about training from service delivery organizations. The most common reasons for employers participating in training are to support a local organization and address a shortage of qualified workers (see Table 4-1 below). Caution must be exercised in the interpretation of any findings related to the employer survey given the low survey response rate (7%).

Source: Employer Survey Technical Report, May 2022

Administrative data showing the number of tools ordered, downloaded and viewed by type and category indicate that employment organizations, post-secondary institutions, and other federal departments most frequently ordered and viewed essential skills tools. Most orders and downloads were concentrated in Ontario (47%), followed by British Columbia (11%) and Quebec (10%).

Participants in program-funded essential skills interventions reported some access challenges, particularly people from under-represented groups accessing online training.

Three participants in essential skills interventions, interviewed for this evaluation, identified the time commitment associated with the frequency and duration of the training program as an access-related challenge. Some focus group participants identified childcare as an important factor to be considered by adult training opportunities.

Several focus group participants said that in-person training was critical to its success. They felt that an online format would not have been conducive to building connections with others and improving their communication and interpersonal skills.

Focus groups and interviews with members of under-represented groups also revealed other access challenges, such as:

• Internet problems
• communication issues with training staff
• difficulty integrating Indigenous-related activities online
• difficulties accessing or using online training portals
• timing of the training (particularly for parents), and
• additional issues related to mental health

Some interviewees found the training they participated in to be 'very accessible,' and reported having established positive relationships with their instructor.

Most participants in program-funded essential skills interventions (79%), surveyed for this evaluation, reported experiencing no challenges with training. Indigenous participants (particularly Métis) and participants with a disability reported more likelihood to encountering challenges (68% and 65%, respectively). About a fifth of Indigenous participants (21%) and participants with a disability (20%) reported that they did not have the necessary technology to access training. In addition, one-quarter of participants with a disability (25%) reported that they could not easily get to their training location. This challenge was also reported by 18% of Indigenous participants (particularly Métis). See Figure 4-1 below for the percentages of participants having experienced challenges with the program-funded essential skills interventions, by under-represented groups.

Source: Participant's Survey Technical Report, May 2022

Text description of Figure 4-1

Types of challenges with the program-funded training experienced by under-represented groups	Percentage of overall participants who experienced challenged	Percentage of women who experienced challenges	Percentage of Youth who experienced challenges	Percentage of Indigenous participants who experienced challenges	Percentage of visible minorities who experienced challenges	Percentage of persons with disabilities who experienced challenges	Percentage of participants with no vulnerability
Didn't have the necessary technology to access the training (if online)	10%	13%	10%	21%	11%	20%	6%
Didn't have the necessary skills to do well in the training (training was too advanced)	3%	4%	N/A	3%	5%	N/A	1%
Wasn't very interested in the training	5%	4%	3%	9%	4%	10%	6%
Couldn't easily get to the training location	4%	4%	5%	18%	5%	10%	1%
Couldn't afford to take the training (due to child care costs, transportation costs, etc.)	4%	5%	5%	6%	4%	10%	3%
None of the above	79%	75%	79%	68%	77%	65%	82%

While visible minority participants had similar results to the total, Black respondents were more likely to report challenges accessing the necessary technology and affording the training.

Essential skills training is extending its reach, including to people from under-represented groups.

In focus groups, participants became aware of the essential skills training in a variety of ways. Most participants (13 out of 17) heard about the program-funded training either through an organization (such as, a training provider) or institution (such as,  a school). Other participants were referred to the training program by:

• a friend
• a family member or
• their business networks

The majority of Indigenous participants (4 out of 7) completed a training program offered by an Indigenous employment and training organization and became aware of the training through their previous connection with the organization.
Half of interviewed newcomers (2 out of 4) reported learning about their training program from:

• an organization supporting newcomers
• a service delivery agency, or
• the training or academic institution they were enrolled in at the time

Participants become aware of essential skills training in various ways:

• heard about the training through their networks
• received an email from a contact or service delivery organization
• heard about the training at an industry event
• heard about the training through their organization

Source: Focus groups and interviews of 17 participants (from under-represented groups) in program-funded essential skills interventions conducted for the purpose of this evaluation

In a survey of participants in program-funded essential skills interventions, out of 196 respondents, many participants (44%) found that the program-funded training they participated in was somewhat easy to find. This was reported by most Indigenous participants (62%) and youth (54%) compared to other respondent groups. However, many respondents from other groups including "non-under-represented" (34%), visible minority (36%) and participants with a disability (47%) reported that the training was not very easy to find.

Many participants (36%) heard about the training from the organization that delivered it. Among female participants, those who were Indigenous or had a disability were more likely to report hearing about the training online. Very few participants across all groups reported hearing about the training from their employer or from an employer where they wanted to work. Participants from visible minority groups (43%, particularly Black participants) and youth (36%, particularly visible minorities) were more likely to have heard about the training from a friend. Finally, many Indigenous participants heard about the training either from the organization that delivered it (31%), online (29%), or from a friend (26%).

4.6 Partners' and Stakeholders' capacity to deliver essential skills training and the Program's influence

This section relates to Evaluation Question 4, which assesses the extent to which the Program has contributed to improving partners' and stakeholders' capacity to deliver essential skills training.

Provinces and territories have largely adopted the Essential Skills Framework and developed new or used existing essential skills programs, tools, and products.

According to Program documents, the Essential Skills Framework is adopted 'as is' by 7 Provinces and Territories. An additional 2 (Alberta and New Brunswick) use a different name for their frameworks but are "completely aligned with the framework." Of the interventions delivered through Labour Market Development Agreements, in 2 of the 5 provinces surveyed, essential skills comprised more than half of the skills development interventions. The other 3 reported that at least one-fifth of interventions involved essential skills.

In key informant interviews, most (3 out of 4) provincial and territorial partners reported adopting the essential skills model or using essential skills programs, tools and products. This includes:

• using essential skills tools and products, such as measurement and program design tools
• integrating the framework into their projects, contracts, and programming

The integration of essential skills by provincial and territorial partners is two-fold. First, essential skills assessments are used to tell whether clients approaching them for funding require upskilling prior to other training and to conduct pre- and post-assessments for those facing employment barriers. Second, essential skills are used at the community level when working with employers and industry partners to help individuals move into and sustain employment.

One key informant spoke to the issue of organizations (such as colleges) adopting the Essential Skills Framework and carrying out training without a formal curriculum or a full understanding of the model or complexity levels. This occurs with increases in program funding. Thus, while the importance of essential skills training may be recognized, it is not fully understood.

Partners and stakeholders had mixed views on whether the Program's support improved their capacity to deliver essential skills training.

The most frequently identified Program support that helped improve funding recipients' capacity to deliver essential skills training was their pilot project funding. Other Program supports mentioned include:

• flexibility adjusting timelines, deliverables, and the allocation of resources
• acting as a 'matchmaker,' 'connector,' or 'facilitator' of networking opportunities
• providing additional funding for the purchase of computers or laptops
• sharing useful evaluation tools^{Footnote 27}
• giving support on an as-needed basis to deal with issues or problems that arise

Only a few participants identified further supports the Program could offer. They include:

• making them aware of effective evaluation tools
• creating or providing access to culturally-adapted tools for essential skills training and evaluation
• fostering more training opportunities for practitioners^{Footnote 28}
• providing more networking opportunities for the sharing of best practices

Expert panelists stressed that the success of learners is dependent on service providers' ability to deliver essential skills programming by targeting the specific needs of learners, empowering and motivating them to participate. They felt training providers require support to deliver training due to essential skills gaps and challenges delivering the necessary tools.

Other Labour Market Programs and partners in one interview noted that the suite of tools developed prior to 2006 to support training lacked applicability to all occupations. A few expert panelists likewise did not feel the Program's tools were particularly useful. However, two other experts attributed positive impacts of the new and previous skills approaches to the development of relevant tools and resources. One of these experts also stated that the "loss of the essential skills profiles deprived education and training providers of a tool to communicate the economic importance of investment in Essential Skills."

Half of interviewed program officials indicated that the organizations funded by the Program already have the capacity to deliver essential skills training and interventions. Those organizations not receiving funding may be less able to build capacity. They suggest that the Program broaden the funding pool in order to increase sector capacity.

Experts suggest enhancing organizations' capacity to deliver and embed essential skills training into their programming.

Panel experts caution that inconsistency in skills training across service providers results in individuals remaining ill-equipped to succeed in the labour market. Some experts attributed these inconsistencies to service delivery organizations' lack of a clear understanding of how to embed essential skills in their training programs. They suggest ensuring that service delivery organizations receive standardized training and support to promote a uniform understanding and application of essential skills based on consensus around skill descriptors and competencies. Experts also emphasized that these issues are compounded by the fact that "the [current] incentive structure [is tied] to bums in seats," rather than performance. This results in service delivery organizations having "no incentives at all around accountability" to ensure the success of their programs.

4.7 Participants' and employers' training needs

This section corresponds to Evaluation Question 4.1 and assesses the extent to which participant and employers' needs were met.

Participants, including those from under-represented groups, are satisfied with the essential skills training they received.

Most survey participants (89%) reported being very happy or happy with the training they received. This holds true across all respondent groups, including visible minorities (89%), Indigenous participants (78%), women (92%), and persons with a disability (80%).

Over half of peoples from under-represented groups (10 out of 17), interviewed for this evaluation, were "completely satisfied" with their training program. For instance, Indigenous participants highlighted:

• positive relationships with Indigenous training instructors
• cultural workshops, combining personal and work-related topics
• discussions about residential schools, including managing intergenerational trauma
• life skills and self-management skills, including the ability to reflect on personal barriers and how to overcome these
• the training being widely applicable/relevant to all participants despite the significant variation in age

The following training topics were highlighted by newcomer participants as being particularly beneficial:

• communication skills (for example, how to communicate effectively and professionally)
• information on online job search tools and skills needed in the Canadian labour market context
• job application skills including interview skills, resume and cover letter writing

In contrast, 6 out of 17 respondents were "mostly satisfied" with the training and pointed to areas of satisfaction and dissatisfaction. These participants included 3 newcomers, one visible minority woman, and two Indigenous women (one with a disability). Areas of satisfaction reported by participants included the ability to:

• gain a better understanding of the industry (4 of 17)
• develop business skills (3 of 17)
• network with others in the same field of work (2 out of 17)
• build relationships with peers (2 out of 17)
• ask questions and have interactive discussions in a comfortable training environment (3 of 17)

Areas of dissatisfaction raised by these participants included:

• training topics covered in the courses being unnecessary, not applicable, or repetitive (3 out of 17)
• length of the training program and necessary time commitment (2 out of 17)
• no tangible course materials, resources or documentation provided to participants (2 out of 17)
• training instructors not understanding the challenges experienced by participants with mental health issues (1 out of 17)
• industry-specific training topics in high-demand not covered by the training program (1 out of 17)
• the cost of the training for newcomers (1 out of 17)

Among survey participants who reported that the training did not meet their needs, the most common reason was because it was 'too basic'. This sentiment was echoed by peoples from underrepresented groups interviewed for this evaluation.

Areas of dissatisfaction raised by one of the participants (newcomer) who reported being completed dissatisfied included:

• training instructors were not able to address the various needs of students with different backgrounds
• industry-specific topics in high-demand (such as topics related to cybersecurity, data analysis, and web development) were not covered by the training
• the participants were not well supported in the transition from in-person to online training which had negative impacts in meeting the skills-related needs of participants (particularly with respect to interpersonal and communication skills)

Essential skills training helped participants from under-represented groups overcome barriers to learning and employment. Participants reported gaining important skills, enrolling in additional training, finding jobs, and building their confidence.

Peoples from under-represented groups, interviewed for this evaluation, reported significant learning and employment gains as a result of their participation in essential skills training.

Learning

Of the participants who reported being in school or planning to enroll in the next 6 months, 36% (32 of 90) indicated the training helped them a lot to enroll in additional training.

Employment

Close to 30% (50 of 196) of participants reported the training helped them find a new or better job by helping them overcome challenges at work.
25% (49 of 196) of participants indicated that the training helped them learn a new skill.

Support

17% (33 of 196) of participants reported that the organization delivering the training helped participants find a job.

Two examples of "Learning Journey Maps" based on in-depth interviews with participants from under-represented groups, interviewed for this evaluation, can be found in Annex I. Each map tells the story of a participant's experience with Essential Skills training funded by the Program. The Learning Journey Maps demonstrate the participants' skill gains in:

• learning
• social and cultural relationships
• interpersonal skills
• networking
• health and personal wellbeing
• communication skills, and
• professional development

For example, one mentions the participants' improvement in communication skills, and having become more comfortable speaking in a group setting.

"The training made us feel very confident because [now] you know what to do, you have no fear; you know what's going on even though you don't know the people that will interview you."

In another Learning Journey Map, the participant reports having taken the initiative to register for another, industry-specific course after having participated in an essential skills training.

"Training is always part of my schedule; I need to have that constant source of new material. The internet is great, but it's good to have people and their experiences. This is very valuable; this is how connections are made."

A majority of participants from under-represented groups, explained how the training helped them overcome barriers to employment. Almost all participants indicated that the training helped them strengthen their:

• communication
• interview skills, and
• job search skills

In particular, newcomers reported:

• learning how to network
• learning how to write cover letters and resumes, and
• gaining an increased level of confidence in interviews

In addition, a majority of participants said that their training helped them to pursue further education. These participants largely reported a broadened understanding of the essential skills required to obtain employment and the benefit of taking additional courses to achieve their career goals.

4.8 Participants' essential skills and labour market outcomes and the Program's contribution

This section corresponds to Evaluation Question 5 which assesses the extent to which the program's activities, such as collaborations and engagements with stakeholders and partners, contributed to improving participants' essential skills and labour market outcomes.

Most participants self-reported improvements in social or interpersonal skills, communication skills, people skills, as well as an increase in their self-confidence at work or in looking for a job, and their mental health and wellbeing.

Some peoples from under-represented groups (7 out of 17), interviewed for this evaluation, reported improvements to their resume writing and interview skills, writing and numeracy skills, digital and computer skills, organizational skills and their ability to identify and overcome employment barriers.

Of those interviewed participants, 3 business owners or Chief Executive Officers, reported improvements to their networking skills and business management skills, including business trends and terminologies. Their participation in essential skills training contributed to an increased sense of self-sufficiency in managing their own businesses.

All 17 participants from under-represented groups, interviewed for this evaluation, reported some degree of increased self-confidence or self-sufficiency as a result of their participation in program-funded training. Also, the majority of participants (10 out of 17) reported improvements in communication skills. Most interviewed newcomers (6 out of 8) reported increased confidence, A few participants also reported improvements in their physical and mental wellbeing. As a result of their participation in the trainings, newcomers reported:

"Being more comfortable and less nervous or afraid to search for a job", "increased confidence in speaking my mind and sharing opinions and thoughts", "stronger grasp of the language through the training", "increased confidence in the current job", "increase confidence in pursuing advocacy-related work for people with disabilities", and "increased confidence in their life in general"

Surveyed participants reported that the most common skills gained as a result of their training were:

• social or interpersonal skills including team working skills (45%)
• communication skills (40%), and
• people skills (36%)

Participants also mentioned improvements in business development skills, skills specific to their employment, and improvements in their cultural knowledge (both their own culture and Canadian workplace culture).

Survey findings also indicate that all participants reported some degree of increased self-confidence or self-sufficiency as a result of their training, and that more than half of participants reported improvements in other social-emotional skills. As a result of their participation in a program-funded essential skills training, surveyed participants reported the following:

• 40% respondent experienced an increase ("a lot") in their self-confidence at work or in looking for a job improved due to their participation in the training
• 34% of respondents indicated that their self-confidence in their life overall improved "a lot"
• 20% of respondents reported improved personal wellbeing (20%), and
• 18% reported improved mental health and wellness

The survey also reveals that respondents with a disability were most likely to report that the training did "not at all" help to improve their self-confidence to look for a job or improve their personal wellbeing. Indigenous respondents and respondents with a disability were equally the most likely to report that the training helped "not at all" improve their self-confidence in life overall.

More details on the survey findings are presented in Figures H0-1 and H0-2 in Annex H.

There is evidence of improvements to participants' ability to obtain and retain employment 1 year post-intervention.

Of 196 survey respondents, some (46%) were unemployed at the time of their training. When they took the survey, only 27% of participants reported not having a job.

Some survey respondents reported getting a new (31%) or better (16%) job one-year after completing the training.

47% of participants reported getting a new job, or getting a better job with their existing employer one-year after completing the training.

Furthermore, the majority (52%) of participants reported retaining their job because of the training.

Over 40% of respondents from under-represented groups reported being unemployed when they took their training. At the time of the training, people from under-represented groups reported higher rates of unemployment (see Figure 4-2) except for participants who did not self-identify as belonging to one of the under-represented groups. Overall, 47% of all respondents (including peoples from under-represented groups) reported not having a job at the time of the training. In contrast, at the time of the survey (see Figure 4-3), peoples from under-represented groups reported higher rates of employment. Overall, 57% of all respondents reported having a job and 27% not having a job.

Participants with a disability (50%) and women (45%) most frequently reported that the training did not help them get a new or better job. Youth (50%), Indigenous participants (48%, particularly First Nations respondents) and visible minority participants (48%, particularly Black respondents) more frequently indicated that the training helped them get a new job.

Additional findings on participants' ability to retain employment one-year post intervention across under-represented groups, with a lens on intersectionality, are presented in Figures H0-3, H0-4, and H0-5 in Annex H.

A majority of interviewed participants from under-represented groups (11 out of 17) were unemployed prior to completing the essential skills training. While a majority (6 out of 11) of these participants indicated that the training helped them obtain a new job, (4 out of 6) of these individuals reported losing it shortly after due to the COVID-19 pandemic.

There is anecdotal evidence from program officials, funding recipients, Other Labour Market Programs, partners, employers and experts that participation in essential skills interventions has a positive impact on employer outcomes.

The funding recipients interviewed were more able to track or report on essential skills acquisition than employment outcomes since their projects did not always involve training nor an employment-related component, and tracking participants following training was not part of their projects.^{Footnote 29} One participant whose project tracks the employment status of trainees observed that:

"All the participants entering the program were on income assistance, but none of them are after the training, and those working are more likely to have full-time jobs than part-time jobs. Some have bought cars and houses as their incomes have increased, while others have gone on to enroll in post-secondary education."

A few program officials (3 out of 14) said that the Program's activities and supports improved the outcomes for employers who participated in the Program's funded projects.

Three Other Labour Market Programs and partners reported positive employer outcomes^{Footnote 30}. These include a shorter unemployment duration, positive wage effects, and continued integration of essential skills within employer tools.

Some experts identified the positive impact of investing in skills development among employees on employers' business outcomes.

Two experts highlighted the difficulty in assessing the extent of employer engagement and their business outcomes due to a lack of publicly available information on funded projects.

Of the surveyed employers, one indicated that the training helped their employees improve their:

• team working skills
• people skills
• organization skills
• time management skills, and
• ability to think critically

However, this employer indicated that the training did not help at all to improve employees' writing and reading skills, mathematical skills, ability to read a document to get information, or computer program skills.

A few other employers (2 out of 11) reported hiring training participants. Figure H0-6 in Annex H presents additional findings on how trainings helped both existing and new employees based on employers' reports.

Most funding recipients (8 out of 15) reported that their funded essential skills interventions were able to achieve improved results related to employers and/or workers/job seekers. Some qualified their assessment, observing that it was based on preliminary data or anecdotal evidence only, but felt comfortable providing it, nonetheless. In the case of workers/employees, the improvements in skills acquisition include:

• increases in confidence in understanding new skills
• demonstrating new skills appropriately in the workplace
• improved numeracy and reading skills, and
• increased confidence in various social-emotional skills

Out of the 11 employers who responded to the survey, 6 reported being "somewhat satisfied" with their organizations' experience with the essential skills intervention, and the remaining (5 out of 11) reported being "very satisfied". Few surveyed employers (2 out of 11) suggested that the training could be longer to better meet their needs.

In terms of impacts on employers, interviewed stakeholders mentioned that employers who took part in an essential skills training intervention were generally:

• satisfied
• had an interest in extending the training
• planned on hiring individuals trained in their program; and
• improved productivity in the sector targeted by the essential skills intervention

Of the 5 interviews with program officials that commented on this question, most (4) reported benefits to employees and employers. These benefits include:

• increased skills
• greater work attendance and
• greater engagement at the workplace

Only 2 out of the 5 provincial and territorial partners interviewed commented on this question, with mixed responses. Both reported that employees can benefit from essential skills interventions, but there remain some challenges. Additionally, they highlighted some recent approaches including:

• connecting essential skills to employee outcomes
• focus on what an individual can already do to bridge the gap, or
• focus on what employers need to do to bridge the gap

Provincial and territorial partners are still figuring out how to market essential skills to employers and the essential skills model facilitates these conversations.

Conclusions and recommendations

The evaluation assesses the Program's contribution to achieving the expected outcomes, including the ultimate outcome related to improvements in essential skills and labour market outcomes of participants in essential skills training. The evaluation also assessed the adequacy of the design of the new "Skills for Success" model. The main evaluation findings, as well as the recommendations, are below.

The main evaluation findings include:

• recognition from program officials, Other Labour Market Programs, partners, stakeholders and experts that the Skills for Success model is sound, is designed to address the evolving needs of the labour market, and aligns with international essential skills approaches
• participants in program-funded essential skills training interventions, including participants from under-represented groups, are satisfied with the training they received. They reported that the training helped them overcome barriers in learning and employment, gain important skills, and increased their sense of wellness and wellbeing
• surveyed participants reported that the most common skills gained as a result of their training were:
  • social/interpersonal skills including team working skills (45%)
  • communication skills (40%), and
  • people skills (36%)
• among surveyed participants 31% reported having found a new job one year after their participation in the essential skills training and 16% reported having found a better job.

The evaluation identified the following recommendations for the Program moving forward. The Program could:

• continue to strengthen performance management practices with funding recipients, including exploring a way to measure skill levels of training participants, to provide regular and robust evidence to support policy analysis and program decision-making
• increase communication and engagement within ESDC and with partners and stakeholders, particularly employers, to foster awareness of the Skills for Success Model and the tools to support the implementation of the new model
• continue to improve access to foundational and transferrable skills training opportunities and to raising awareness of these opportunities particularly to under-represented groups

Management response and action plan

Overall management response

Management accepts the recommendations outlined in the Evaluation of the Literacy and Essential Skills (LES) Program (now the Skills for Success Program) and will be engaging in further analysis on how its findings can inform ongoing adjustments to program design and delivery. Insights gained through the evaluation's lines of evidence, including the perspectives of employers and groups who are under-represented in the labour market have confirmed the importance of program changes made in recent years and provide additional insights for considerations going forward.

Significant efforts were already in progress during the time of the evaluation to address areas of focus highlighted in the report. This includes strengthening performance management practices and increasing communication and engagement with stakeholders representing under-represented groups, other Governments and with employer organizations to foster awareness of the Skills for Success (SFS) Model and the development of tools to support its implementation.

Budget 2021 announced an investment of $298 million over 3 years for the Skills for Success Program. As a result, the Program developed an Investment Strategy involving several intake processes including solicited proposals from provinces and territories and from key organizations. In addition, the Program launched an Open Call for Proposals (CFP) in January 2022 under 2 streams: Tools and Training and Research and Innovation. Projects funded under the Investment Strategy will contribute to respond to elements in each of the 3 recommendations

Recommendation 1

Continue to strengthen performance management practices with funding recipients, including exploring a way to measure skill levels of training participants to provide regular and robust evidence to support policy analysis and program decision-making 

Management response 

Management agrees with the recommendation.  There is a need to encourage and incent funding recipients as much as possible within this context, and other Labour Market Programs, to collect information on participants' skills acquisition and labour market outcomes, as well as participant demographics.

As a model for the data that could be collected, a 2-year initiative run under the Adult Learning, Literacy and Essential Skills Program's Terms and Conditions, the Women's Employment Readiness Pilot Program, is collecting participant-by-participant data and up to 21 identify factors of participants. Incentives were provided to funding recipients in the form of up to 5 percentage points more for administrative costs to offset the costs of expanded data collection and reporting. This data collection framework, and lessons learned, will be used going forward to inform the Skills for Success Program.

Additionally, the Skills for Success Program works on an ongoing basis to strengthen performance management practices with funding recipients. Project recipients delivering training are required to collect and report data on project participants' demographics, as well as outcome-related data such as training completion, participants' skills acquisition, labour market outcomes and employers' business results.

The Program has been implementing changes to improve the consistency of reporting by funding recipients. It has developed a new data collection tool that includes definitions of demographic information to ensure common understanding across funded projects. The Program will also provide funding recipients with information sessions and functional guidance to help with the consistency and quality of the reporting.

As outlined in the management action plan of the previous evaluation, the Program renewed its performance indicators as part of the Performance Information Profile (PIP) which was approved officially June 2022. The PIP has identified clear actions and activities to meet expected outcomes of the current evaluation recommendations.

1.1 SFS Data Collection Strategy

The objective is to gather information that demonstrate skills and employment outcomes for participants, including from under-represented groups. The strategy incorporates a survey that will be integral to studying the impact of foundational and transferable skills training on participants, and a fuller complement of support tools and training sessions to improve consistency. It will also support the identification of promising skills development approaches.

In-progress

Start date - April 2024

1.2 Clearer annual reporting requirements for funding recipients

Agreements from recent intake processes (Solicited organizations and P/Ts and the CFP) will require organizations to report annually on aggregate or disaggregate information related to individual demographics and outcomes in a template provided by ESDC. 

In progress

First annual reports - April 2024

1.3 Feasibility study on a "Made-in-Canada" skills survey

The Program's launch of the new SFS Model provides broad direction to support Canadians in upskilling, reskilling and getting back to work. The Program for the International Assessment of Adult Competencies (PIAAC) is an international assessment of core skills that examines proficiency in literacy, numeracy and problem solving in technology-rich environments. It includes questions that allow participant results to be classified by a range of factors that can influence skill outcomes including education, employment status and age.

However, PIAAC does not include a range of demographic identifiers that are important in the Canadian context, including groups that are under-represented in the Canadian labour market. The skills measured only partially align with the SFS Model, and it is only conducted once every 10 years. In order to provide robust evidence to support policy analysis and program decision-making, the Program will explore ways to measure Canadians' skills levels on a more regular basis.

A feasibility study will provide insights on the scope and need for more frequent and broader data on the skills levels adult Canadians.

Activities include:

• Conduct a feasibility study for a Canadian survey of foundational and transferable skills comparable to the PIAAC survey but enhanced to include each of the nine skills outlined in the SFS Model.

In-progress

Due date for completion of feasibility study: April 2023

Recommendation 2 

Increase communication and engagement within ESDC and with partners and stakeholders, particularly employers, to foster awareness of the Skills for Success Model and the tools to support the implementation of the new model

Management response 

Management agrees with the recommendation. The Skills for Success Program will continue to raise awareness of the SFS Model and its tools and conduct communication and engagement activities with partners and stakeholders to increase the uptake and adoption of the Model. Stakeholders and partners are encouraged to use available resources and adapt them to their needs and the reality of the audience they serve. More recently, ESDC launched the new SFS Model and SFS website in May 2021 which provides Canadians and employers with information about the new model and links to training resources.

The 2022 CFP outlined that funded projects would allow the SFS Program to respond to current skills and labour market issues and enable the broad adoption and uptake of the Skills for Success Model.  Eligible projects have to demonstrate how they would enable the broad adoption and uptake of the Skills for Success Model. This is done by including a dissemination and sustainability strategy to share project results with organizations and employers.

Moving forward, the program will continue to update, expand and promote the content of the SFS Program's website. This will include information products, a compendium of tools and resources based on the SFS Model, research reports, as well as a variety of training resources for education and training providers to draw from to incorporate into their own programs. Most of these resources will likely stem from projects funded by the program. The Program will also continue to engage with stakeholders, including funding recipients, employers and employer organizations, labour/unions and organizations representing job seekers and workers, including from underrepresented groups.

Management action plan

2.1 SFS engagement strategy

Over the next 2-years, the SFS program will develop and implement an enhanced communication and engagement strategy. The objective of the strategy is to increase awareness and adoption of the SFS model particularly by employers and organizations supporting groups who are under-represented in the labour market.

Planned activities include:

• Implement communication and engagement activities to foster awareness of the new SFS Model and related resources to other relevant labour market actors:
• Stakeholder organizations focusing on skills development and serving under-represented populations, namely: women, persons with disabilities, Indigenous people, members of official language minority communities, newcomers, and visible minorities including Black Canadians and other racialized groups.
• ESDC's programs:
• Labour Market Development and Workforce Development Agreements (LMDA/WDA)
• Indigenous Skills and Employment Training (ISET) Programs
• Strategic Partnerships Funds (SPF),
• Youth Employment and Skills Strategy (YESS)
• Opportunities Funds (OF)
• Other Government Departments:
• Immigration, Refugees and Citizenship Canada (IRCC)
• Innovation, Science and Economic Development (ISED)
• Provinces and territories (PTs)

Completion date: In-progress

2.2 Strategy to engage employers

The SFS Program will develop an engagement strategy to reach employers.

Activities include:

• Creating awareness of the available SFS Model  tools and promising approaches
• Developing an accompanying marketing approach and communication tools such as infographics, videos, case studies, success stories, and include more content for employers on the website; and
• Implementing the strategy, working through sector councils and other employer organizations

Completion date: TBD

2.3 SFS profiles (updated Essential Skills (ES) Profiles)

Key findings identified that ES Profiles were an important resource used by education and training providers. The Program is in the process of updating the top-searched occupational profiles in a variety of sectors to align them with the new skills model. The Program is also developing guidance documents and a practical webinar series to promote and support ways on how training or sectoral organizations can develop or update their own SFS Profiles. These materials will be published over the next 2 years.    

Completion date: 2021 to 2024

2.4 Update Website

The website provides Canadians with assessment tools and training resources aligned with the SFS model at no cost. It also offers information on funded projects and a variety of training resources for training providers to draw from to incorporate into their own programs, as well as research reports. As more training resources and research material are developed, ESDC's Skills for Success website's content will be updated.

Planned activities include:

• Updating the website on an ongoing basis as training resources and research material are made available
• Tracking and measuring website traffic through views/downloads providing an indication of its reach; and
• Exploring options to gather user feedback to gather intelligence on usability and usefulness of the website

Completion date: On-going

2.5 Fostering networks

Moving forward, the Program will identify and leverage existing networks to share its work more broadly with those supporting under-represented groups and involved in skills training, career development, counselling and human resources.

Planned activities include:

• Updating an environmental scan of organizations serving groups who are under-represented in the labour market
• Developing an inventory of events that bring together professionals in career development and counselling, skills training and Human Resources; and
• Developing a strategy to share the Program's work broadly during these events. This would include promoting the website

Completion date: TBD

Recommendation 3

Continue to improve access to foundational and transferable skills training opportunities and to raising awareness of these opportunities particularly to under-represented groups

Management response 

Management agrees with the recommendation. The Program will continue to improve access to skills training opportunities for Canadians, including those from under-represented groups. According to the 2012 OECD Programme for International Assessment of Adult Competencies (PIAAC) survey, which is the most recent information available, approximately 45% of employed workers are at level 2 or below for literacy and 50% are at level 2 or below for numeracy in Canada. The numbers are even higher, at 56% and 66% respectively, for unemployed Canadians. These numbers highlight the importance of foundational and transferable skills going forward, as more than 97% of jobs created in the last 20 years require level 3 literacy or higher.

To address the foundational and transferrable skills needs of Canadian, Budget 2021 committed to provide 90,000 new training opportunities over 3 years. Of this investment, 50% is supporting projects that address the skills needs of under-represented groups, including: women, persons with disabilities, Indigenous people, members of official language minority communities, newcomers, and visible minorities including Black Canadians and people who are racialized.

Going forward, the Department will leverage other labour market programs within ESDC and continue to leverage other stakeholders to integrate the Skills for Success model in their programs to provide foundational and transferrable skills training to more Canadians to help fill labour shortages and address skills mismatches for employers.

The Women's Employment Readiness Pilot Program also provides and tests foundational and transferable skills training, coupled with wrap-around supports, to 4 groups of women facing multiple barriers: Racialized and/or Indigenous women; women with disabilities; women from the LGBTQ2 community; and women with prolonged workforce detachment. The pilot also tests models to improve workplace inclusivity. The results of the pilot will be used to inform systemic changes to skills and employment programming, including the SFS Program, to improve access for these groups of women.

Management action plan

3.1       Leveraging other labour market programs to deliver foundational and transferrable training and raising awareness amongst underrepresented groups through a Diversity Community of Practice

 Moving forward, the Program will explore leveraging existing federal labour market programming to improve access to foundational and transferrable skills training for Canadians, including underrepresented groups.

In order to improve awareness and access to foundational and transferable skills training opportunities, the Program will also explore the feasibility of setting up a community of practice for under-represented population groups. The community of practice could help develop a communication strategy, help identify priorities, assist with outreach with organizations, and provide recommendations on how to best improve access to training opportunities for under-represented populations. The community of practice would be an opportunity for dialogue and a key hub of communication and sharing of promising practices, tools and training resources, as well as research material developed through Program funding.

Planned activities include:

• Reviewing existing federal labour market programs to assess the feasibility of incorporating Skills for Success model as part of the training being offered to Canadians; including underrepresented groups
• Exploring the feasibility of setting up a community of practice for under-represented population groups         

Completion date: TBD

Annex A: Evaluation questions and lines of evidence

Relevance

Evaluation question 1: To what extent does the new Essential Skills approach ("Skills for Success") responded to the evolving^{Footnote 31} needs of the labour market?

Performance

Evaluation question 2: To what extent has the Program supported partners and stakeholders in identifying, testing and implementing innovative models in essential skills training?^{Footnote 32}

Evaluation question 2.1: To what extent has the Program influenced partners and stakeholders to integrate proven approaches to essential skills training?

Lines of evidence:

• document review
• literature review
• administrative data review
• key informant interviews

Evaluation question 3: To what extent has the Program contributed to broadening access to essential skills training and services across Canada?

Evaluation question 3.1: To what extent has the Program influenced the integration of essential skills into other Labour Market Programs?^{Footnote 33}

Evaluation question 3.2: How did the integration of essential skills influence access to essential skills training, including for people from vulnerable groups? ^{Footnote 34}

Lines of evidence:

• document review
• administrative data review
• key informant interviews
• focus groups

Evaluation question 4: To what extent has the Program improved partners' and stakeholders'^{Footnote 35} capacity to deliver essential skills training?^{Footnote 36}

Evaluation question 4.1: To what extent are participant and employer training needs met?

Lines of evidence:

• document review
• literature review
• administrative data review
• key informant interviews
• expert panel
• participants' survey

Evaluation question 5: To what extent have the Program's activities, such as collaborations and engagements with stakeholders and partners, contributed to improving participants' essential skills and labour market outcomes?^{Footnote 37}

Lines of evidence:

• document review
• literature review
• administrative data review
• key informant interviews
• focus groups
• expert panel
• participants' survey

Annex B: Previous evaluation findings

The most recent evaluation of the Program covers the period from 2011 to 2012 to 2015 to 2016. Key findings include the following:

 Relevance of the Program

• There is a continued need for literacy and essential skills programming and activities in Canada
• There is a need and a role for the federal government in literacy and essential skills programming
• The Office's activities do not duplicate those of other funders
• There is a need for the activities, including those not related to project funding, such as research and analysis and the development of partnerships and networks

Effectiveness of the Program

• Communication with provinces and territories has been limited but shows recent signs of improvement
• Some efforts have been made to facilitate knowledge sharing and networking, but more can be done
• The activities have contributed to many of the expected outcomes, although it is difficult to determine their role in progressing towards systemic change
• The Adult Learning, Literacy and Essential Skills Program and the National Essential Skills Initiative projects have achieved results but delays in funding decisions have tempered results and affected networks
• Performance information (for example, indicators, logic model, and data collection tools) exists, but is not systematic or uniform. Given the evolution of the program mandate, performance information could benefit from updating

Efficiency of the Program

• The evidence is mixed as to whether the operational transition of literacy and essential skills grants and contributions delivery has had an observable impact on efficiency

The previous evaluation generated 3 recommendations:

• Consider working with provinces, territories and partners to develop formal partnership strategies to support stakeholder network development and the sustainability of effective approaches
• Continue to improve communications with provinces and territories, partners and stakeholders; and
• Update the performance measurement information and related tools to reflect recent changes

Annex C: Literacy and Essential Skills Program logic model

Annex D: Skills for Success

Skills for success

• Are the skills needed to participate, adapt, and thrive in learning, work and life. These skills are becoming increasingly imperative for all Canadians in a rapidly changing labour market that is characterized by new technology, globalization and diversity. Canada required a skilled workforce, inclusive workplaces and lifelong learning to be innovative, competitive and inclusive
• Include foundational skills – those skills on which all other skills are built and social-emotional skills-the human skills required for effective social interaction. These skills can overlap and interact with each other, and with other technical and life skills
• Are recognized by extensive research drawn from Canadian and international sources as influencing a wide range of personal, professional and societal outcomes. They are transferable, teachable and assessable. They are inclusive and can be adapted to different contexts and users
• Are for everyone – employers, workers, training providers, governments and communities. They are critical to Canada`s economic and social prosperity      

Skills for Success Criteria

Transferable

Applicable across all jobs and within a variety of contexts

Teachable

Applied in learning or training contexts and linked to outcomes

Assessable

Quantitative or qualitative, demonstrable, measurable, comparable, consistent

Recognized

By Canadian and international research for influencing outcomes

Durable and enduring

Responsive to the evolving labour market

Skills for Success includes the following essential skills:

• Numeracy
• Communication
• Creativity and Innovation
• Problem solving
• Reading
• Digital
• Collaboration
• Adaptability
• Writing

Annex E: Comparison of Skills for Success with the old skills framework

Annex F: Methodology and limitations

The evaluation's methodology is based on 7 lines of evidence. These include an administrative data analysis, a document review, a literature review, key informant interviews, surveys, focus groups, and an expert panel. Each line of evidence and its limitations are described below.

1. Administrative data and file review

A total of 70 projects received funding from the Program to develop, test and implement innovative models in essential skills training during the evaluation period

The administrative data and file review serves several purposes. Specifically, it:

• examines the quantitative and qualitative data the Program gathers about program outputs and outcomes.
• assesses the quality and consistency of the data, and
• identifies improvements made since the last evaluation.

Over 80 documents with administrative data were considered. These documents include:

• Pilot project information sourced from the Common System for Grants and Contributions
• Annual Performance Information Analysis and related documents from 2018 to 2021
• Provincial and Territorial program profiles
• Data and files related to other labour market programs in Canada
• Other relevant sources on internal program activities

A framework was developed to address the evaluation questions with the administrative data. Data was analyzed both qualitatively and quantitatively, where applicable.

Limitations

Much of the project data is qualitative and lacks a consistent reporting style. Manual calculations were therefore required to extract and aggregate data where possible. There was also little data on both national access to essential skills training and use of essential skills by other Canadian Labour Market Programs.

Furthermore, pilot project performance results data were only available for the second half of the evaluation period.^{Footnote 39} This limited the observation of overall trends for project-specific performance indicators over time. Finally, the data reviewed did not allow for the assessment of the impact of provincial and territorial collaboration on service delivery.

2. Document review

Over 170 documents reviewed

The document review considered key documents that provide insight into the Program's mandate, operations, and reporting. It also noted changes to the Program during the evaluation period. In addition, documents from provincial and territorial essential skills programs helped inform the Program's national context. Over 170 documents were reviewed. This includes materials both external and internal to the Government of Canada.

Limitations

There were no significant limitations in the preparation of the document review. However, some of the evaluation questions could not be answered based on the content of the documents. Also, a few documents required contextual clarification from the Program.

3. Literature review

A total of 30 references cited

1. Academic papers: 13
2. Grey literature: 17

The literature review assessed data from a range of different academic and 'grey' sources. These sources helped inform the context in which the Program operates. The review identified and reported on literature related to evolving labour market needs in Canada. It also explored domestic and international best practices in essential skills. Over 60 sources were consulted, and about 30 could be used to address the evaluation questions.

Limitations

There were a few gaps in the literature. Few sources commented directly on the impact of federal literacy programming in Canada. Canada-specific discussions of the skills and labour market standings of certain vulnerable groups were also rare. As a result, some of the papers reviewed were dated or did not give a complete picture of the Program or topics of interest. Thus, only evaluation question 1 could be answered with the literature review.

4. Key informant interviews

A total of 60 key informants were interviewed:

3. Program officials: 14 individuals
4. Partners: 12 individuals
5. Stakeholders and funding recipients: 34 individuals

Key informant interviews were conducted with various groups including program officials, partners, stakeholders, and funding recipients.These key informant groups represent a range of views, both internal and external to the Government of Canada. Specifically, these included:

Program Officials

6. Program officials in the Skills Employment Branch and Program Operations Branch at ESDC (14 individuals)

Partners

7. Representatives from Other Labour Market Programs within and/or outside ESDC and other partners (8 individuals)
8. Provincial and territorial partners from different provinces and territories including representatives responsible for adult education and training and P/Ts responsible for trades training (4 individuals)

Stakeholders and funding recipients

9. Service providers and practitioners, including employers (13 individuals)
10. Other individuals, groups, or organizations external to federal and Provincial/Territorial governments who have expertise in essential skills policy and programming (6 individuals)
11. Funding recipients' representatives (15 individuals)

Both individual- and group-level analyses were conducted with interviewee responses to identify key themes, divergent opinions, and other information of note.

A purposive sampling method was used to select interviewees based on level of experience, type of work, depth of knowledge and years of experience with the Program, and geographic area.

Limitations

Program Officials

The level of familiarity with the Program varied by interviewee, with some being familiar with the Program, some with one of the funding programs, and some with little familiarity. Therefore, not all interviewees could comment on all of the evaluation questions and indicators. Where an interviewees was unable to comment on an indicator, the total number of interviews that were included in the analysis of findings for that indicator were reported.

Stakeholders

This qualitative research was designed to reveal a rich range of opinions and interpretations rather than to measure what percentage of the target population holds a given opinion. As such, the results indicate participants' views about the issues explored, but they cannot be generalized to the full population of stakeholders. Thus, the results of this research were used to substantiate and supplement findings from the other lines of evidence.

5. Survey

A total of 207 surveys completed

12. Participants' completion rate: 196 from a sample of 846 – response rate 23%
13. Employers' completion rate: 11 from a sample of 146 – response rate 7%

The survey was administered online to gather feedback from participants in essential skills interventions and employers to answer questions about the Program's contribution to improving essential skills and labour market outcomes. This includes participants in pilot projects and other Labour Market Programs with essential skills integrated. Two separate survey instruments were created for:

14. Individual participants such as job seekers and workers, including those facing barriers to employment; and
15. Participating employers.

The surveys for participants and employers consisted of 33 and 24 questions, respectively. Selective sampling was used to target individual participants and employers that participated in essential skills training between April 2016 and March 2021.

Limitations

• Recruitment. A contractor reached out to a total of 73 funding recipients to request lists of individual participants and employers that participated in essential skills training. Of those, only 10 provided the names and contact information of training recipients.^{Footnote 40}
• Also, the sample of eligible participants per organization varied widely. Some organizations provided long participant lists (over 100 individuals) and others very short lists (less than 10 individuals). Survey findings may therefore be more representative of the experiences of individuals from organizations with larger samples.
• Sample size. Findings for the employer survey should be interpreted with caution in light of the very small sample size (N=11). While 11 employers completed the survey, only 7 respondents recalled information related to the training program and answered the corresponding survey questions.
• Recall bias. Survey respondents that participated in essential skills training earlier in the funding period (closer to April 2016) may not remember their training experiences as accurately as those that participated recently. Nevertheless, all survey respondents were provided with the name of the training program they participated in, the name of the organization that delivered the training, as well as training program dates (where available).
• Immigrant status. A higher-than-expected proportion of participant survey respondents are Permanent Residents. It is possible that some respondents may have selected the "Permanent Resident" response option by default since no "Canadian citizen" option was provided in the survey question. These data are therefore excluded from cross tabulation analyses examining vulnerable groups.

6. Focus Groups and Interviews with Peoples from Under-Represented Groups

A total of 17 participants took part

• 4 focus group discussions: 10 individuals
• in an interview^{Footnote 41}: 7

Focus groups were held to assess:

• The extent to which the Program meets the needs of participants;
• The Program's contribution to broadening access to essential skills training, particularly for people from vulnerable groups; and
• Participants' improvements in essential skills and labour market outcomes with a focused lens on people from vulnerable groups.

Survey participants were invited to take part in a follow-up focus group or interview. From the sample of participants that agreed to be contacted either for a follow-up focus group or an interview, individuals from vulnerable groups were identified based on survey responses.

A total of 10 participants took part in one of 4 focus groups, and 7 individuals participated in individual interviews. The 4 focus groups were grouped by characteristic. Specifically, they were themed on visible minority (4 participants), disability (2 participants), immigrant status (2 participants), and Indigenous Persons (2 participants).

Overall, in the focus group discussions and interviews, participants self-identified as:

• Women (9)
• Other gender (3)
• Indigenous (7)
• Visible minority (9, 8 of whom were also newcomers)
• Living with a disability (6)
• Immigrant (2)

Participant employment statuses also varied:

• Employed (7)
• Unemployed (5)
• In school (2)
• Medical leave (1)
• Retired (1)

Limitations

Sampling limitations prevented the inclusion of individuals that selected French as their official language of choice. Almost all potential focus group or interview participants identified English or English and French as their preferred official language(s). There were recruitment challenges for French participants, limiting the range of sampling desired (or something like this.

Fewer participants than expected participated in a focus group discussion.^{Footnote 42} As a result, findings were analyzed across individual participants that took part in either a focus group or an interview. (Put this paragraph first).

Expert Panel

A total of 6 experts consulted

• Academics: 3
• Practitioners: 3

The expert panel aims to leverage the knowledge of 6 experts in essential skills to add credibility to the evaluation findings. Panelists were selected from members of Program working groups and advisory groups, so long as they met the following criteria:

• At least some knowledge of OLES and their work over the last 5 years;
• Mix of regions;
• Mix of academics and practitioners;
• Mix of areas of focus/discipline; and
• Not a recipient of funding issued by OLES.

Once the experts were selected, the panel was implemented in 2 rounds. For round 1, experts were sent questions and asked to provide written responses. For round 2, a videoconference was held to gather feedback from the experts on the responses shared during round 1. Responses were then compared across the 6 experts, highlighting areas of convergence and divergence.

Limitations

An Indigenous perspective was absent from the panel. While an expert with an Indigenous focus was invited to the panel, they were not available to participate. Despite efforts made, a replacement Indigenous expert with sufficient knowledge of the Program could not be located in time to replace them. As such, the panel responses may not be fully representative of the diverse views of Canadian experts.

Note: The mixed-methods approach employed in this evaluation, and the triangulation of findings from seven key lines of evidence, serves to minimize bias,validate findings, and address methodological limitations.

Annex G: Scale for qualitative results

The scale below was used to indicate the significance of qualitative results stemming from several lines of evidence. Factors such as a respondent's position, areas of responsibility, and years of experience were considered.

"All/almost all"

Findings reflect the views and opinions of 90% or more of subjects in the group.

"Large majority/most"

Findings reflect the views and opinions of at least 75% but less than 90% of subjects in the group.

"Majority"

Findings reflect the views and opinions of at least 51% but less than 75% of subjects in the group.

"Half"

Findings reflect the views and opinions of 50% of the subjects in the group.

"Some"

Findings reflect the views and opinions of at least 25% but less than 50% of subjects in the group.

"A few"

Findings reflect the views and opinions of at least 2 but less than 25% of subjects in

Annex H: Additional quantitative and qualitative findings

This section provides additional quantitative and qualitative findings grouped by indicator and data collection tool.

There is some evidence of improvements in their ability to obtain and retain employment within one year-post-intervention but it is not possible to draw a conclusion on the Program's attribution to the improvements.

Figures H0-3, H0-4, and H0-5 present quantitative findings on participants' ability to retain employment within one-year post intervention across all groups, including under-represented groups, with a lens on intersectionality.

As shown in the figures below, among female participants, disabled respondents were more likely to say the training did not help them get a new or better job. Among youth participants, visible minorities were more likely to say they got a better or new job and Indigenous participants were more likely to say they got a new job.

Most respondents across under-represented groups (>60%), particularly visible minority respondents (84%) reported that the training helped them to keep their job for at least one year after they took the training. Indigenous participants (62%), youth (61%) and visible minority participants (53%) were more likely to report no longer working at the same job they got as a result of the training.

Figure H0-4 below presents additional findings on how trainings helped both existing and new employees based on employers' reports.

Annex I: Learning journey maps

Participant Learning Journey (1)

Participant Profile

• The participant is a visible minority, newcomer woman, residing in Alberta.
• She is currently working full-time.
• She completed a training program focused on language and literacy essential skills and employment-related skills.

1. Participant Training Needs

The participant was a newcomer to Canada trying to secure a job. To achieve this goal, she sought to gain more knowledge about Canadian culture and improve her job searching and interviewing skills in a Canadian context.

2. Training Program Identified

The participant was taking English language classes at the local college when she heard other students talking about the essential skills training program. College staff formally introduced the training program to all students, informing them that anyone could apply to register for the training. She was not accepted into the training program when she first applied because the program was full. She was accepted into the program after applying a second time.

3. Training Overview

The training was offered 5 days/week for 3months. The training was held in-person at the local college, which was very accessible. The participant had already been taking English language classes at the local college.

The training program covered a number of employment-related topics including:

• Job search skills
• Resume writing
• Professional communication skills how to conduct yourself in an interview and on the job)
• Interpersonal skills (how to work with others in a group)
• Problem-solving skills
• Workplace culture in Canada

The participant did not experience any challenges with the training.

4. Participant Experience

The participant enjoyed the training and particularly liked the instructor, whom she said was very good at preparing participants for potential job interviews.

"I learned new things I didn't know before…like I hadn't done an interview here [In Canada] before…the criteria is different."

"The training made us feel very confident because [now] you know what to do, you have no fear; you know what's going on even though you don't know the people that will interview you."

The training program helped participants prepare and review their resume and connected them with prospective employers.

"They helped you find companies that were looking for employees. I interviewed with those companies and got a job with one of them."

5. Training Outcomes

The participant experienced improvements in five main areas:

Communication skills

Particularly in group settings including job interviews and in the workplace.

"The training prepared me to feel confident in an interviews and at the workplace."

Interpersonal skills

Including group work and problem-solving.

Professionalism in a Canadian context

For example, how to communicate with employers, how to conduct yourself during an interview and at work.

Employment

She secured a job with one of the companies immediately after completing the training.

Knowledge

She was able to share the skills she acquired during the training with her friends and family.

"I benefited from the training, but not only me, my friends and family too because I shared my learning with them."

6. Ongoing Needs/Future Considerations

The participant secured a job after completing the training, but was let go a few months later due to the onset of the COVID-19 pandemic. She completed additional training programs after completing the essential skills training. She is currently working as a caregiver. She is very interested in pursuing healthcare aid training to get a better job. She was also open to any other training courses that could improve her skills.

Participant Learning Journey (2)

Participant Profile

• The participant is an Indigenous single father living with a disability, residing in British Columbia.
• He is currently self-employed running a small business
• He completed a training program focused on essential skills in supply chain management

1. Participant Training Needs

The participant launched his own business to become more self-sufficient. "As a person with a disability, not all options are available to me. Self-employment was really appealing because I could customize it to my disability.

He sought to improve his professional knowledge and business management skills to better position the company in the supply chain and ensure its success. He also sought to create professional networks by connecting with fellow small business owners.

2. Training Program Identified

The participant heard about the supply chain training at an industry event. The training program was recommended by representatives of an industry trade association organization.

"The training was important to remain engaged and be proactive with career options."

3. Training Overview

The training was offered in-person 2 days/week or 6 weeks. The participant had to travel to the training site by ferry and taxi, and eventually carpooled with fellow classmates. He received some financial assistance from an Indigenous organization to support his travel expenses, but still needed to arrange for child care. He was grateful that his friends and neighbors could babysit.

" As single parent it [traveling] was challenging, but I was driven to increase our financial situation. Neighbors and friends helped, but was a bit stressful."

The training program focused on business management skills, including how to help businesses stimulate demand for services and how these services fit into the supply chain. The training covered the following topics:

• Business standards to reflect quality of services (for example, standard certifications, corporate branding)
• Interpersonal skills (for example, working in a group; creating synergies with other businesses)
• Communication skills (for example, drafting presentations, public speaking/sales pitches)

4. Participant Experience

The participant was very satisfied with the training, and how it allowed him to network/build relationships with other companies and develop business/synergies. "The training was crucial because it created the network I needed to immerse myself in the business world."

He acquired a lot of knowledge from the instructors and fellow classmates and indicated that the training was extremely valuable and exceeded his expectations in this respect. The instructors shared experiences that participants could relate to and the training format allowed for participants to develop interpersonal and communication skills.

"My biggest take away from the training was the networking. The training was beneficial because of the networking aspect."

5. Training Outcomes

The participant experienced improvements in 4 main areas:

Business knowledge

The training covered foundational business concepts and terminologies, which gave him a better grasp on how to run a business.

Soft skills

The training was structured in a way that enabled relationship building between participants and instructors, and allowed for the development of interpersonal and communication skills.

Networking

The training provided valuable opportunities to network with other small businesses in the region. As a result, he was able to form a relationships with a small number of companies and generate a new clientele for his business.

Self-sufficiency

The training helped him diversify his business opportunities and propel his business to be better positioned in the industry. This has allowed him to achieve greater financial stability.

6. Ongoing needs/future considerations

The participant operates in a relatively new industry that is in continuous development. He is always looking for industry specific training opportunities to remain up-to-date with the latest industry trends that help propel his business. He is currently registered for an industry-specific course, which he believes will help his company generate more business.

He highlighted that the supply chain has changed since the onset of the COVID-19 pandemic and believes that a supply chain training program offered now could provide valuable insights in this new context. He would be interested in taking this type of training in the future, but noted that the training would need to be offered in-person to yield the same value.

"Training is always part of my schedule; I need to have that constant source of new material. The internet is great, but it's good to have people and their experiences. This is very valuable; this is how connections are made."