Evaluation of Literacy and Essential Skills

From: Employment and Social Development Canada

Executive summary

This evaluation report assesses the relevance and performance of the activities of Employment and Social Development Canada (ESDC) related to the development of workplace literacy and essential skills, through the Office of Literacy and Essential Skills (hereafter referred to as “the Office”) and the grants and contributions programs funded under the Consolidated Revenue Fund - Adult Learning, Literacy and Essential Skills Program and Employment Insurance Part II - National Essential Skills Initiative.

The evaluation covers fiscal years 2011 to 2012 to 2015 to 2016 and focuses on how the Department and its programming have supported the development of workplace literacy and essential skills in Canada, and how it has more recently, 2015 to 2016, shifted its focus to systemic change by encouraging the integration of literacy and essential skills into labour market programming, services, and policies.

The evaluation was limited by the availability of performance information. Performance measures are currently being reviewed and should better support the next evaluation. This evaluation addresses key questions related to the continued need for the Program, whether the activities contribute to the achievement of workplace literacy and essential skills outcomes, if the shift in funding approach is making progress towards supporting systemic change in the labour market, and the availability of quality performance information. The structure of the program involves providing support to non-governmental organizations to develop and disseminate tools and supports that contribute to improving literacy and essential skills for the work place. Given this program design, it is indirect support which is provided to Canadians; and for this reason, net impacts could not be assessed.

Key findings

Relevance of the program

There is a continued need for literacy and essential skills programming and activities in Canada.
There is a need and a role for the federal government in literacy and essential skills programming.
The Office’s activities do not duplicate those of other funders.
There is a need for the activities, including those not related to project funding, such as research and analysis and the development of partnerships and networks.

Effectiveness of the program

Communication with provinces and territories has been limited, but shows recent signs of improvement.
Some efforts have been made to facilitate knowledge sharing and networking, but more can be done.
The activities have contributed to many of the expected outcomes, although it is difficult to determine their role in progressing towards systemic change.
The Adult Learning, Literacy and Essential Skills Program and the National Essential Skills Initiative projects have achieved results but delays in funding decisions have tempered results and affected networks.
Performance information (for example, indicators, logic model, data collection tools) exists, but is not systematic or uniform. Given the evolution of the program mandate, performance information could benefit from updating.

Efficiency of the program

The evidence is mixed as to whether the operational transition of literacy and essential skills grants and contributions delivery has had an observable impact on efficiency.

Recommendations

Based on the evidence presented in this evaluation, it is recommended that the Office:

Consider working with provinces, territories and partners to develop formal partnership strategies to support stakeholder network development and the sustainability of effective approaches;
Continue to improve communications with provinces and territories, partners and stakeholders; and
Update the performance measurement information and related tools to reflect recent changes.

Management response

Introduction

Employment and Social Development Canada (ESDC) welcomes the evaluation and its contribution to program policy development. ESDC will give careful consideration and commits to effective implementation of evaluation recommendations and lessons learned. As such, ESDC has started to implement activities related to all three recommendations.

In 2013, the Programme for the International Assessment of Adult Competencies (PIAAC) results found that 50% of the general population tested in PIAAC scored at or below level 2 on a 5 level literacy scale. Given the magnitude of public need, it was acknowledged that efforts needed to be broadened significantly. Accordingly, in 2015 to 2016, the decision was taken to shift the focus of ESDC’s Office of Literacy and Essential Skills (OLES) to support the integration of essential skills into direct delivery programs within ESDC, with other government departments and agencies, with provincial and territorial governments, and other key stakeholders such as post-secondary institutions, employers and labour.

In addition, further research showed that there are particular groups at greater risk of marginalization. Since 2015 to 2016, efforts have placed a greater emphasis on addressing the needs of individuals with low skills and facing multiple barriers to employment such as Indigenous people, youth, newcomers and Official Language Minority Communities.

Given this shift in policy direction was introduced near the end of the timeframe of the evaluation, it is useful to see evaluation evidence has found that recent activities are showing promise. The evaluation recommendations will help OLES, working in close collaboration with Program Operations Branch, to enhance effectiveness and efficiency of Program efforts to deliver on expected results.

Recommendations

Recommendation #1

Consider working with provinces, territories and partners to develop formal partnerships strategies to support stakeholder network development and the sustainability of effective approaches.

Response

ESDC agrees with the recommendation. Respecting that provincial and territorial governments are primarily responsible for the delivery of employment and training systems, the development of partnerships amongst governments is key to making significant progress in terms of integrating essential skills into labour market programming. In addition, working horizontally across the federal family will be central to improve resource strategies to support the sustainability of effective practices.

In line with the shift in program directions introduced in 2015 to 2016, ESDC has established four multilateral tables which align well with the findings and recommendations of the evaluation:

Federal, Provincial and Territorial (FPT) Literacy and Essential Skills Network (Fall 2016)
eLearning Federal, Provincial and Territorial (FPT) Advisory Committee (June 2017)
Essential Skills and Apprenticeship Community of Practice (June 2017)
Government of Canada Federal Network on Essential Skills (Spring 2017)

FPT Literacy and essential skills network

This network was created in Fall 2016
The FPT Literacy and Essential Skills Network, which includes all jurisdictions except Quebec (who chose to abstain from participation), serves as a forum for knowledge sharing on integrating literacy and essential skills into labour market programming.
Members of the network developed terms of reference for the group collaboratively and have identified the sharing of knowledge on how governments are levering labour market transfer agreements to support literacy and essential skills as an area of focus.
Given governments’ mutual interest in improving the evaluation of the effectiveness and outcomes achieved resulting from essential skills programming in the lives of Canadians, performance measurement has also been identified as a key discussion topic.

eLearning FPT advisory committee

The eLearning FPT Advisory Committee was created in June 2017. This volunteer group includes BC, MB, NB, NS, ON and YK.
ESDC is supporting Minister Hajdu’s mandate letter commitments around “improving workers’ access to quality job training that provides Canadians with pathways to good careers” by developing options for eLearning solutions to support Canadians who need to upgrade their skills, particularly essential skills.
This is a new area of focus and, as part of early exploration, ESDC will aim to do so in collaboration with PTs to ensure complementarity of efforts. Ultimately, it will inform the development of an eLearning platform to support Canadians to develop their essential skills.

Essential skills and apprenticeship community of practice

Launched in June 2017, the Community of Practice is a peer assist network amongst federal, provincial and territorial government officials responsible for the integration of essential skills into labour market programming. The purpose of this group is to support government officials responsible for policies and practices to support apprentices to overcome essential skills barriers to complete training and acquire certification. This will support members of the community to draw from efforts across the country to inform their government’s respective efforts and to identify areas of mutual interest where the federal government could facilitate collaborative approaches.

Government of Canada federal network on essential skills

Spring 2017, ESDC formed a Government of Canada Federal Network on Essential Skills which includes departments and agencies who are actively involved in supporting Canadians to improve their skills to support a range of outcomes (for example digital literacy, financial literacy, health literacy, etc.).
Membership to date includes Indigenous and Northern Affairs Canada, Public Health Agency of Canada, Financial Consumer Agency of Canada, Status of Women Canada, Immigration, Refugees and Citizenship Canada, and CANNOR. Membership may be expanded over time, should other government departments express interest.
This network will serve as a forum to share information and learn more about respective activities and identify opportunities to better complement each other’s efforts and, improve capacity to work collaboratively on areas of mutual interest.
It will also provide a means to gather relevant information to be shared with stakeholders and partners to improve awareness of the breadth of federal programming related to essential skills.

Recommendation #2

Continue to improve communications with provinces and territories, partners and stakeholders.

Response

ESDC agrees with the recommendation. As such, ESDC is currently developing strategies and conducting activities that contribute to improving its communications with provinces and territories, partners and stakeholders.

Communication strategy

ESDC is presently developing a communication strategy which will assist in aligning communications vehicles (for example webinars, eBulletins, etc.) to be relevant to different target groups. The primary objectives of this communication strategy will be to share information on ESDC’s current and upcoming activities as well as reinforcing knowledge sharing such as results of projects funded by ESDC. While this strategy is being developed, ESDC will continue to offer webinars, send eBulletins, organize face-to-face meetings and engage in various other communication activities.
As noted above, the FPT Literacy and Essential Skills Network will also contribute significantly to improving communications with PTs. In addition, ESDC will continue to engage with PTs bilaterally to deepen the department’s understanding of each jurisdiction’s unique realities, priorities and gain a greater appreciation of areas of mutual interest to help ensure complementarity of efforts.

Project database

ESDC is currently collecting information and identifying results from projects funded. Starting in Fall 2017 the OLES database website will be updated to include projects from 2017 to 2018 onward. Results will also be identified from a few projects funded over the past few years to share resources, models developed and, lessons learned.

Recommendation #3

Update the performance measurement information and related tools to reflect recent changes.

Response

ESDC agrees with this recommendation. The Performance Information Profile (PIP) is currently being updated and will be completed by June 2018. The updated PIP will:

Be expanded to include the performance measurement of OLES operations and the Grants and Contributions programs;
Include a stronger focus on outcomes measurement (for example skill gain, psycho-social indicators), data collection and reporting (to be developed in collaboration with the Program Operations Branch);
Seek greater consistency in the measurement, tracking and reporting on program results; and,
Account for recent program changes such as testing new training and employment models and working to integrate LES into other employment training programming.

It is expected that these key improvements to OLES’ performance measurement profile will help detect program success factors, strengthen its ability to demonstrate achievement of results, and communicate a more complete and evidence-based performance story.

1.0 Introduction

This evaluation report assesses the relevance and performance of the activities of Employment and Social Development Canada (ESDC) that support efforts to enhance the development of workplace literacy and essential skills, through the Office of Literacy and Essential Skills (hereafter referred to as “the Office”) and the grants and contributions programs funded under the Consolidated Revenue Fund - Adult Learning, Literacy and Essential Skills Program and Employment Insurance Part II - National Essential Skills Initiative.

Multiple lines of evidence were used to inform the evaluation, including: literature and document review, administrative data and file review, key informant interviews and a survey of funding recipient organizations. Appendix A presents the evaluation questions which are linked with the section of the report where those findings can be found. Appendix B provides details about the methodology and limitations. The evaluation was limited by the availability of performance information. Performance measures are being reviewed and should better support the next evaluation. The Office provides support to non-governmental organizations to develop and disseminate tools and supports that contribute to outcomes for beneficiaries and does not provide direct support to the ultimate beneficiaries.

2.0 Program context and description

2.1 Context

Literacy and essential skills contribute to developing and maintaining a strong labour market and healthy economy in Canada. The labour market continues to evolve and the composition of the country’s population is changing. There are increasingly fewer Canadians in the labour force as baby boomers transition into retirement. These factors are giving rise to an increased reliance on immigration, pools of non-traditional sources of labour and the participation of vulnerable populations, as potential sources of skilled labour. At the same time, the skills that adult Canadians need to perform their work today are, in many cases, vastly different than they were a few decades ago. Many workplace tasks that were once manual and routine are now being replaced by non-routine, analytical and interpersonal tasks. New and emerging technologies are also accelerating the pace of change, offering opportunities for working age Canadians to adapt to workplace demands, through continuous learning and skills upgrading. Ensuring that skills training meet the realities of the labour market helps employers find workers with the right skills which in turn improves workplace efficiency, productivity and competitiveness.

Within this context, the barriers faced by vulnerable populations in developing literacy and essential skills are an ongoing social and economic issue in Canada and across the globe. According to the Program for the International Assessment of Adult Competencies (PIAAC) survey^{Footnote 1}, in 2011 to 2012, Canada’s results are average (see Figure 1), with notable variations in scores across provinces and territories.

Figure 1: Canada’s Scores Compared to Other Countries (2011-12) — **Figure 1: Canada’s Scores Compared to Other Countries (2011 to 2012)**

More specifically, in Canada:

Four out of ten adults lack sufficient levels of literacy to be fully competent in most jobs.
Literacy and numeracy scores are highest at ages 25 to 34, and are lower among older age groups.
Individuals aged 16 to 34 are found to be the most proficient, in problem solving in technologically rich environments.
Men have higher numeracy skills than women across all age groups, while both genders display similar proficiencies in literacy and in problem solving in technology rich environments.
Information-processing skills of Indigenous populations, immigrants, and official-language minority populations vary considerably across provinces and territories and across skills being measured.

2.2 Background

Employment and Social Development Canada is committed to the development of a skilled, adaptable and inclusive labour force and an efficient labour market. The Office contributes to the achievement of this strategic outcome by supporting Canadians to improve their literacy and essential skills to help them better prepare for, to get, and to keep a job and to adapt and succeed at work. For these purposes, essential skills include the skills associated with literacy (that is reading, writing, document use and numeracy), but go beyond these to also include thinking skills, oral communication, computer use/digital skills, working with others and the skills associated with continuous learning.

The Office supports the integration of essential skills into direct delivery of employment and training programs funded by the Department, by other government departments and agencies, and by provincial and territorial governments. To this end, it also supports stakeholders such as post-secondary institutions, employers, and labour market groups by providing funding and encouraging the development of networks. Its efforts are focused on programs supporting individuals with low skills and facing multiple barriers to employment such as Indigenous people, youth and official language minority communities. For example, one project identified literacy and essential skills challenges that prevent First Nations and Inuit men from participating in education and employment, as well as built partnerships in order to improve education outcomes and employability of First Nations and Inuit men in Northern Canada.

During the evaluation period, 13 projects were funded. Estimation of the number of Canadians that directly or indirectly benefitted from this support is limited by the nature of the funded projects, which do not deal directly with the ultimate beneficiaries and the absence of requirement for funding recipient organizations to report on impacts on Canadians.

The Office uses a combination of internal resources and grants and contributions to conduct four main activities:

research and analysis;
development and dissemination of tools and supports;
development and maintenance of partnerships; and,
delivery of grants and contributions programming.

These four activities relate to five outputs that support the achievement of its early, intermediate and ultimate outcomes:

evidence-based knowledge and policy options,
literacy and essential skills tools and supports,
partnerships and networks,
calls for proposals, and
funding provided to projects using grants and contributions funding.

The grants and contributions programs are delivered by the Program Operations Branch which works closely with the Office to provide funding to projects that replicate and scale up proven approaches to skills upgrading across Canada, and that develop innovative approaches to improve employment and training supports so they are more responsive to employer and worker needs.

Support is also provided to the Roadmap for Canada’s Official Languages 2013 to 2018, a government-wide commitment of $1.15 billion over five years (2013 to 2014 to 2017 to 2018) to meet the needs of Canada’s official language minority communities.^{Footnote 2}

The evaluation was limited by the availability of performance information. Performance measures are being reviewed and should better support the next evaluation.

2.3 Program budget

In 2011 to 2012, the first year covered by the evaluation, the budget for the Adult Learning, Literacy and Essential Skills Program decreased from $25,009,000 to $21,509,000 (14% decrease) in comparison to the year before and then remained stable throughout the evaluation period. The National Essential Skills Initiative budget decreased from $13,200,000 annually from 2010 to 2011 to 2012 to 2013, to $4,200,000 (68% decrease) in 2013 to 2014. Disbursements for both programs were lower than their budgets and have generally been decreasing over the evaluation period.

The following table lists the budgets for the grants and contribution programs for fiscal years 2011 to 2012 to 2015 to 2016.

**Table 1: Budget for the office’s grants and contributions programs 2011 to 2012 to 2015 to 2016**
Program expenditures^{Footnote 3}	2011 to 2012	2012 to 2013	2013 to 2014	2014 to 2015	2015 to 2016
ALLESP - Budget	$21,509,000	$21,509,000	$21,509,000	$21,509,000	$21,509,000
ALLESP - % of budget spent	80.5%	56.7%	69.1%	56.5%	38.9%
NESI - Budget	$13,200,000	$13,200,000	$4,200,000	$4,200,000	$4,200,000
NESI - % of budget spent	89.8%	69.4%	88.7%	16.1%	Not available

Source: Chief Financial Officer Branch

To select projects for funding the Department held calls for proposals and calls for concepts. It also made efforts to support a continuous intake of proposals by soliciting proposals for projects addressing specific gaps in priority areas. Additionally, the Department accepted unsolicited proposals addressing a labour market issue linked to a literacy and essential skills problem, need or gap. Continuous intake was ongoing throughout the evaluation period.

The Office began to shift its strategic direction during the evaluation period. Funding recipients were informed in 2011 that the non-competitive core funding was not sustainable and were encouraged to work together to find efficiencies and other sources of support. Core funded agreements approved during this time were extended until June 30, 2014.

During this time, two calls for proposals were held. In 2012, 44 applications for project funding were received, and 12 were funded. In 2013, a Call for Proposals was held for the Pan-Canadian Network to replace the non-competitive core funding approach with a more innovative and sustainable approach. It was expected that organizations funded from this call would cover a range of functions that had been identified as integral to an effective pan-Canadian network and an efficient literacy and essential skills system. One project out of 106 applications was approved for funding.

In 2014, when core funding agreements ended, all unsolicited or solicited projects in negotiations at the time were ceased, and proponents were asked to re-submit in the 2015 call for concepts. Beginning with the 2015 call, efforts have focused on the integration of literacy and essential skills into labour market programming, such as employment and training supports to improve labour force participation. The Department is now making strategic investments in transformative projects to improve Canadians’ access to job training. Funding is directed to projects that replicate and scale up proven approaches to improve the quality of employment and training supports that are more responsive to employer and worker needs.

3.0 Relevance findings

3.1 Continued need for program

There is a continued need to support literacy and essential skills to improve labour market outcomes in Canada. While there continues to be a demand from employers and industry for a workforce with improved literacy and essential skills, there is a lack of awareness among them about these issues and the ways to address them. The needs are heterogeneous across groups and regions.

Literacy and essential skills are central to addressing challenges of an ageing labour force and the unprecedented pace of technological change in the labour market. An increasing number of jobs are requiring essential skills^{Footnote 4}. Across Canada, there are skills mismatches and skills shortages in particular occupations and regions^{Footnote 5}. A mismatch implies that the employees occupying given positions are either over- or under-qualified, or that the available pool of labour to fill an open position has different skills than required or lives in a different region. A skills shortage means that there are not enough people with a particular skill or skills to meet demand.

Evaluation findings confirm an ongoing need by industry and employers for better essential skills in the workplace. A survey was completed by 61 funding recipient organizations to collect information on the achievements of funded projects. Approximately 70% of all survey respondents (and a larger proportion (79%) of the sub-group of employers), indicated that there was a large or very large demand among employers and industry for employees with strong workplace essential skills.

There are various perspectives in the literature regarding which skills are most needed by employers. Some authors indicate that transferable, flexible, “soft” essential skills are paramount, while others emphasize the need for improvement in skills such as reasoning, mathematics, numeracy, as well as capacity for critical thinking and problem solving^{Footnote 6}. All of these skills are considered part of the nine essential skills supported by the Department. Through extensive research over approximately 10 years (1992 to 2003), the Government of Canada, along with other national and international agencies, identified and validated nine key literacy and essential skills: reading text; document use; writing; numeracy; oral communication; thinking skills; working with others; computer use; and, continuous learning.

As part of the research conducted for the evaluation, 33 key informant interviews were conducted with ESDC personnel, partners, provincial and territorial representatives and non-governmental experts in adult literacy and essential skills. Most key informants indicated that there is a lack of awareness amongst the general population and employers about the fact that many people have weaknesses in literacy and essential skills, and about the importance and impact of this. According to the interviewees, some employers do not realize the problems they experience in their workforce are often caused by literacy and essential skills issues, and/or some employers do not have the ability or tools to assess the literacy and essential skills capacities of employees or potential employees.

According to some key informants, the extent and nature of the needs for improved literacy and essential skills are not homogenous by province or by group, and as such, service providers working in the field sometimes require adopting a case-by-case approach. According to them, it may be more relevant to talk about needs according to different regions, such as rural and urban, the North, or regions in economic decline. The literature review similarly highlights that there are regional differences in the nature of skill shortages and mismatches.

Some interviewees suggested that further defining areas of common need for essential skills across the country could be helpful in enabling programming to work at the intersection of needs that are common to all partners and stakeholders, and those that are particular to certain contexts or among sub-populations.

3.2 Alignment with government priorities and federal roles and responsibilities

The Office’s main objective is aligned with both Employment and Social Development Canada’s departmental strategic outcomes and federal government priorities.

The Office’s objective, which is “to support Canadians to improve their literacy and essential skills to help them better prepare for, to get and keep a job and to adapt and succeed at work” is aligned with the departmental strategic outcome of “a skilled, adaptable and inclusive labour force and an efficient labour market”.

A large majority of interviewees agreed that there is a link between literacy and essential skills and labour market outcomes and workplace productivity, as lower skill levels present challenges for securing and maintaining employment, and can detract from productivity.

The Office’s priorities for 2015 to 2018 are focused on supporting stakeholders and projects that target individuals with low skills and facing multiple barriers to employment such as Indigenous people, youth, newcomers and official language minority communities. This corresponds with government objectives from the 2015 Speech from the Throne, supporting the economic success of immigrants, cooperation with Indigenous peoples, and the promotion of both official languages. These priorities also align with the federal government’s commitments towards more inclusive growth, as expressed in Budget 2016.

There is a need to further integrate literacy and essential skills development into labour market programming. The federal government has a key role in encouraging co-ordination and communication between provinces and territories, partners, and stakeholders to support the integration of literacy and essential skills into labour market programming. The current efforts do not duplicate the efforts of others.

Need for federal government support

Both interviewees (including partners, province and territories, stakeholders and non-governmental experts) and surveyed funding recipient organizations indicate that there is a need for the federal government to support provinces and territories and stakeholders in the area of literacy and essential skills. Eighty-four percent of all surveyed funding recipient organizations indicated that there is a large or very large need for the federal government to support provincial, territorial and other stakeholder efforts in adult literacy and essential skills in the context of labour market programming.

Some interviewees indicated that although the provinces and territories have the responsibility to implement labour market programming funded through the Labour Market Transfer Agreements and other labour market programming, literacy and essential skills are important determinants of Canada’s labour market outcomes. According to them, it would be desirable if there was further integration of literacy and essential skills elements into more labour market programs. For example, one partner suggested that the federal government work with provinces and territories to develop a common plan of action for integrating literacy and essential skills in Labour Market Transfer Agreements.

Need for conducting and disseminating research and analysis

Most key informants indicated that research and analysis, or the funding of projects that involve research and analysis, is an appropriate role for the Office. This was also reflected in the survey of funding recipient organizations, where 90% of respondents somewhat or completely agreed that there was a need for the federal government to conduct research and analysis on adult literacy and essential skills and labour market outcomes in Canada. A large majority of survey respondents (89%) similarly supported the federal government disseminating this research and analysis.

Need to build knowledge and expertise

Most interviewees and a large majority of survey respondents were in agreement that there exists a need to build knowledge and expertise in literacy and essential skills. However, views were divided on whether the Office has the capacity and/or expertise to fill this role.

Need to develop tools and supports

A large majority of survey respondents (89%) either somewhat or completely agreed that there is a need for tools and other supports related to literacy and essential skills. Some provincial and territorial representatives also indicated that tools were needed and that they should be developed in close coordination with provinces and territories. Non-governmental experts had a different view on the need for tools as they did not express a need for tools, they are likely aware of and informed about other existing tools.

Need for partnerships and networks

The need to support the development of partnerships and networks was expressed by both interviewees and surveyed funding recipient organizations, and could be a key element of the Office’s unique role in the support of literacy and essential skills. The vast majority (92 %) of survey respondents somewhat or completely agreed that there was a need to support partnerships to address Canadians’ literacy and essential skills needs as they relate to the labour market.

Most interviewed experts and provincial and territorial representatives expressed a need for the Office to help develop partnerships and networks and to work across jurisdictions, especially to coordinate across provinces and territories. Some indicated that cut-backs in the core funding of organizations may have affected the ability of some stakeholders to continue being part of existing networks, and there is a need to rebuild.

Complementarity rather than duplication

Interviewees and survey participants stated that there was little duplication of programming between the various actors who provide services or funding for the development of literacy and essential skills. According to interviewees, the needs are great, and overall, more programming in this area would be useful. Evidence from the survey of funding recipient organizations suggests that there is little duplication of federal efforts. Very few survey respondents (less than one in ten) reported that they had received funding from other federal government departments. Among those that did get funding from other federal government departments, most (57%) indicated that the funding was complementary.

Other sources of funding such as provincial/territorial governments, non-profit organizations, and private sector organizations were also described as funding different kind of projects with complementary objectives. Individual donor and municipal government funding sources were identified by a few survey respondents, and this work was described as separate and different than the projects funded by the Department.

Thus the evidence suggests that the federal government plays a distinct and complementary role within the realm of literacy and essential skills, namely through provision of funding and promoting cooperation. However, concerns were raised about the lack of flexibility in federal government programming, such as the requirement to focus on the nine essential skills identified by the federal government, despite the fact that some provinces have different frameworks.

4.0 Performance findings

4.1 Coordination, knowledge sharing and dissemination of information

The communication and coordination between levels of government has been somewhat limited, often focused primarily on projects, and have decreased from earlier years. Provinces and territories value communication from the Office, and there is room for more regular and formalized communication. Recent improvements have been noted.

The Office has facilitated knowledge sharing and networking to proactively disseminate results of research, and information on resources and tools to some extent. Evaluation evidence suggests that these efforts have been modest and that more can be done.

Provinces and Territories

There is evidence that the Office has reached out to provincial and territorial representatives through presentations at conferences and engaged in labour market programming activities and cooperative work with provincial and territorial governments and stakeholders.

Administrative documents provide some evidence of coordinated federal/provincial/territorial initiatives related to the development of literacy and essential skills. For example, the Apprenticeship Initiative by the Canadian Council of Directors of Apprenticeship, supported by the Department, is meant to support the integration of essential skills into apprenticeship systems for the benefit of apprentices, trade qualifiers, and trades people. This initiative was supported by a federal, provincial and territorial working group that was co-chaired by officials from the Office and the Government of Manitoba. Six other provinces were represented on the Working Group including British Columbia, Saskatchewan, New Brunswick, Nova Scotia, Prince Edward Island, and Newfoundland and Labrador.

Most provincial and territorial representatives interviewed indicated that communication and coordination by the Office was valued but was limited and irregular, with slow response times. Furthermore, a number said that information sharing was infrequent, and that the needs of provinces were not always addressed. Some interviewees indicated that meetings had become more ad hoc in recent years, and communications were mostly limited to funded projects. However, others indicated improvements of communication in recent months. While there has been some valuable communication, overall, the evidence does not suggest that the recommendation from the 2012 Evaluation of the Adult Learning, Literacy and Essential Skills Program to “improve communication with the provinces and territories” was fully addressed.

Partners and stakeholders

A list of over 200 instances of collaboration has been extracted from the administrative data. This list includes partnerships and other relationships that have been established or maintained either internally within the Department, with other Government of Canada entities, with external stakeholders, or with provincial and territorial jurisdictions. Through these partnerships and relationships, the Office has engaged organizations representing Indigenous persons, apprentices and trades groups, and organizations servicing job seekers and workers, employers and entrepreneurs, immigrants and newcomers, and official language minority communities, among other actors.

During the evaluation period, activities were conducted to incorporate stakeholder perspectives in its activities and policy development. In 2011, the Office held consultations on the role and use of its Literacy and Essential Skills tools. This included a series of four roundtables across Canada, supplemented by key informant interviews and the development of a report which informed future directions of program activities. In 2013, it conducted a survey of 499 stakeholders across Canada to support the development of a communications strategy.

There is evidence to suggest that the Office has facilitated knowledge sharing and networking among partners and stakeholders. Dissemination of research, tools, and resources among partners and stakeholders has taken many forms, including presentations, conferences, publications, bulletins and newsletters, and meetings. There have been some examples of renewed engagement in partnerships and knowledge-sharing activities with stakeholders, partners, and employers.

Partners and federal government interviewees indicated that the Office actively disseminates information to them. However, other external stakeholders indicated that it has undertaken modest efforts at information sharing and that more could be done. Similarly, 57% of surveyed funding recipient organizations agreed that the Office proactively disseminated funding research, resources and tools. The fact that 43% of surveyed funding recipient organizations disagreed suggests that sharing has been stronger with some stakeholders, such as those involved directly in programming and internal government partners, while external partners and funding recipient organizations reported low levels of information sharing.

The evidence regarding the extent to which awareness and capacity in best practices have been developed also varies. The administrative data review provided examples of awareness-building in a range of activities and for various target groups. In contrast, a few interviewees said there was a lack of awareness as to which programs currently exist in Canada, including a lack of awareness about the Department’s activities. This suggests that communications could be improved with potential partners and stakeholders to raise awareness of the opportunities available to support projects that focus on the integration of literacy and essential skills into labour market programming, and could encourage improved communications between the many stakeholders.

Similarly, more than half (57%) of surveyed funding recipient organizations agreed that the Office has built awareness of proven practices. In terms of whether it has built capacity in proven practices, a similar number of respondents agreed and disagreed with this notion. Again, evidence suggests that awareness and capacity in proven practices have been developed to some extent, but that more can be done.

Likewise, evidence from interviews suggests that the Office has facilitated knowledge-sharing and networking among partners and stakeholders to some extent, but that the degree to which this is occurring can be improved. Two-thirds of survey respondents agreed that it has facilitated knowledge-sharing and networking.

All of the experts interviewed indicated that there has been some effort at knowledge-sharing and the facilitation of networking among partners and stakeholders. Although it was outside of the period covered by the evaluation, the conference of stakeholders hosted by the Department in October 2016 was mentioned as being a very helpful event that brought people together from different jurisdictions. However, other efforts at knowledge sharing were cited as modest or decreasing, for example, the website was mentioned as being out of date.

4.2 Adequacy of performance information

The Office does have a large amount of performance information available regarding the achievement of outputs and outcomes. However, the information available is not always of sufficient quality for accurately measuring the achievement of outcomes, particularly intermediate and ultimate outcomes.

The Adult Learning, Literacy and Essential Skills Program has a performance measurement strategy with a logic model that was developed in 2011, and the Office has a separate logic model that was created in 2008. The documents have not been updated to cover the new areas of work including the integration of literacy and essential skills programming into labour market programming.

The file and administrative data review shows that performance data is collected regularly, most notably through the final close-out reports which provide information on the advancement, outputs and outcomes of funded projects. According to ESDC personnel that were interviewed, the Office ensures that measures and indicators are established for projects, and final reports are checked for results against outcomes. Internal quarterly performance reports on its activities, which are largely based on information from the final close-out reports and activity reports from funding recipient organizations, are also available for fiscal years 2011 to 2012 to 2014 to 2015. However, due to the inconsistency in the quality, quantity and type (for example, sometimes all that is available is a description of project activities, other times there is data on outcomes and/or outputs) of performance data available, the information is often insufficient for accurately assessing the achievement of expected outcomes, especially at the level of intermediate and ultimate outcomes.

4.3 Outcomes: LES in the Labour Market

The Office’s activities have resulted in a number of outcomes contributing to the achievement of literacy and essential skills improvements within a labour market context. These include the creation of partnerships and encouragement of network development, the provision of tools and supports, capacity building, and the support of the integration of literacy and essential skills priorities into policies and programs.

The Office’s activities contribute to change in the labour market system through support for the development of sustainable partnerships, the provision of tools and supports, the support to integrate literacy and essential skills priorities into policies and programs, and organizational capacity-building.

Partnerships

The evidence indicates that the Office contributes to the formation and sustainability of partnerships. For instance, 90% of the 50 internal quarterly reports completed between fiscal years 2011 to 2012 and 2014 to 2015 indicate that the Office has been engaging in collaborative activities and consultation with stakeholders, beneficiaries, associations and other partner groups. The administrative documents provide some examples of long-running collaboration with partners and also provide an overview of collaboration with various sections of the Department, especially Service Canada, leading to increased communication, outreach, promotion, and the development of new approaches with regards to essential skills. Examples include the Office collaborating with Service Canada to embed literacy and essential skills content on their outreach intranet page, developing a blog on Essential Skills for the Youth.gc.ca website, and updating and increasing essential skills content in the Canada Benefits Finder website.

The Office and most partner informants interviewed provided examples of partnerships that were formed and spoke to the resulting benefits (for example, training, expertise and information sharing, etc.). Furthermore, most survey respondents (78%) completely or somewhat agreed that the Office’s activities have led to the development of partnerships to support literacy and essential skills related to the labour market. The majority of respondents (61%) agreed that partnerships developed through these activities have been sustained. The inception and success of these partnerships can be considered a significant contribution to the achievement of literacy and essential skills outcomes within a labour market context.

Provision of tools and supports

There is also evidence that the Office contributes to the provision of tools and supports related to developing and integrating literacy and essential skills. It makes 88 tools and resources available to support job-seekers, workers and employers to be better informed about the importance of essential skill, and to help assess employees’ strengths and support skills development. During the evaluation period, approximately 2 million copies of resources and tools were ordered and viewed online. The Department has received and approved copyright requests to support stakeholders, provinces, and territories, to adapt its tools for their own use.

Between 2010 and 2012, the Department disseminated information through stakeholder meetings, conferences, booths and in-person workshops. Due to the cost and limited reach of such activities, the Office introduced webinars and eBulletins as a means to disseminate information. Although not consistent in terms of frequency, the eBulletins are circulated to over 800 stakeholders and partners, and the webinars have had high participation rates (for example over 100 participants in a webinar).

Capacity-building

Evidence indicates that the Office contributed to organizational capacity-building. Specifically, efforts were undertaken to enhance or increase the capacity of employers, organizations, practitioners, and communities to engage with literacy and essential skills programs and delivery of services. There are also multiple examples of tools and best practices aimed at building capacity. Some key informants mentioned capacity-building as a benefit of their collaboration with the Office (for example, through the implementation of a “train the trainer” approach). Although the definition of capacity-building can be multifaceted and complex, evidence shows that funded projects do assist organizations in securing the means required to achieve their goals related to literacy and essential skills integration within a labour market context.

Integration into policy and program

There is some evidence that the Office’s activities also contribute to the integration of literacy and essential skills as priorities related to training, programs and policies. The document review revealed instances in which literacy and essential skills considerations were embedded into new or existing programs, namely to focus on the needs of target groups (employer training, homeless populations, Indigenous groups, etc.).

Almost all provincial and territorial key informants mentioned at least one instance in which the Office had contributed to the integration of literacy and essential skills into training, programs and/or policies (for example, collaboration for technical training programs, to increase advocacy about literacy and essential skills, etc.). Some interviewees indicated that they agreed that their project contributed to the outcome of developed and improved organizational policies that integrate literacy and essential skills. At the same time, 82% of survey respondents indicated that they agreed that the project they were referring to contributed to the outcome of integrating literacy and essential skills into programs and services.

Issues of project sustainability began to be addressed with the 2015 Call for Concepts in which applicants were asked for a sustainability strategy with the expectation that interventions will be sustained in whole or in part by partners or other interested parties. Non-governmental experts interviewed as part of this evaluation expressed concerns as to the sustainability of pilot projects beyond initial funding. The survey of funding recipient organizations also reflects this uncertainty: just under half (48%) of respondents did not feel that the Office’s activities and funding had led to sustained, systemic change to address adult learning, literacy and essential skills issues related to the labour market, while 35% felt that they had.

In addition, concerns were expressed about the lack of flexibility in federal government programming, such as the requirement to focus on the nine essential skills identified by the federal government, despite the fact that some provinces have different frameworks.

Overall, the evidence suggests that the Office’s activities (especially through project funding, but through its own activities as well) have resulted in concrete positive outcomes contributing to improvements of literacy and essential skills within a labour market context. However, it is too early to assess results related to sustainability.

4.4 Outcomes: Progress toward systemic change

It is too early to determine whether the Office’s shift in focus in 2015 has contributed to progress towards systemic change as it takes time to see these effects. The funding approach currently being pursued and the elimination of core funding to organizations might have influenced the achievement of literacy and essential skills outcomes.

The evidence shows that the Office has achieved to various degrees, the expected outcomes related to the labour market, although it is too early to determine whether or not they have contributed to progress toward systemic change. The documentation available suggests that some projects addressed adult learning, literacy and essential skills issues in a sustainable way, generating lasting change. It was, however, too early to establish firmly whether or not these changes were sustained or systemic over time.

The interviewed key informants suggested that funded projects have generated many positive outcomes for employers and for people needing to improve their literacy and essential skills, and in some cases, have contributed to progress toward systemic change. However, it is worth noting that some interviewees raised questions regarding the funding approach currently being pursued and whether it is the best model to lead to systemic change. The Department funds pilot projects in the area of literacy and essential skills, but some interviewees reported that there are limited sources of funding available to implement the models after a pilot is over.

Based on early findings, results achieved for the following outcomes will support the progress toward systemic changes:

The funded projects present a wide range and variety of knowledge dissemination and transfer training activities across provinces and territories and have contributed to raising awareness and knowledge in the labour market system about challenges and opportunities related to literacy and essential skills. Almost 90% of project reports indicated achieving a degree of knowledge dissemination. According to the survey, there is ample evidence of dissemination, transfer and application of knowledge and information among partners, stakeholders and employers. The most frequently reported type of output developed by organizations as a result of funding is “essential skills tools, resources, materials and supports (for example ‘train the trainer’ materials)”, which was reported by 70.5% of survey respondents. “Awareness raising and/or promotional products” received the second highest response and was identified by 54.1% of respondents. Eighty-seven percent of respondents completely or somewhat agreed that funded projects increased outreach to partners, stakeholders and/or employers.
A vast majority of project summary reports (92%) indicated increased awareness and understanding among target groups and employers of various literacy and essential skills-related needs and existing tools.
A large majority of survey respondents also agreed that funded projects contributed to an increased awareness of the benefits and opportunities for adult learning, literacy and essential skills in the labour market.
Of the project summary reports examined, 90% indicated enhanced or increased capacity to successfully equipping and training individuals to complete particular assignments and/or refers to institutional activities (planning, resource-building) aimed at increasing performance in certain areas. Results have been observed across provinces and territories.

The table below lists some of the most cited ways in which capacity was said to be built.

**Table 2: Types of capacity building for funded projects**
Nature of capacity built by a funded project		Survey results
For the organization involved in the project	Increased organizational capacity to address literacy and essential skills issues	Eighty percent completely or somewhat agreed
	New or improved partnerships	Seventy-nine percent completely or somewhat agreed
	Increased responsiveness to address client needs	Seventy-nine percent completely or somewhat agreed
For other organizations	The products were used to implement training in literacy and/or essential skills	Seventy-three percent indicated that the products were used in this manner
For other organizations	The products were used to better understand the needs of their clients	Sixty-eight percent indicated that the products were used in this manner

LES opportunities and resources created

The file review showed that 72% of funded organizations identified participation in literacy and essential skills activities as a key output. A number of these activities focused on immigrants and immigrant-serving organizations, on job seekers, on adult learners, and on training activities both of beneficiaries and of instructors. In the survey, 79% of respondents agreed that the funded projects contributed to the outcome of creating adult learning, literacy and essential skills resources and 77% agreed that their project contributed to creating and/or enhancing awareness amongst partners and stakeholders of the benefits and opportunities of adult literacy and essential skills.

Seventy-five percent of project reports mentioned the inclusion of literacy and essential skills into new and existing programs, including business and human resources activities, health and safety programs, sustainable development and high capacity workplace strategies, and others. Seventy-three percent of survey respondents indicated that products were used to implement training in literacy and essential skills. Also, 82% of survey respondents agreed that funded projects contributed to the integration of literacy and essential skills into programs and services.

Elimination of core funding to organizations

Between 2007 and 2012, the Office provided core operational funding through a non-competitive process to a network of twenty-two national, provincial and territorial organizations. In 2011 it began to phase out core funding and shifted towards a funding model focused on innovative and sustainable projects that scale up or replicate effective approaches to integrating literacy and essential skills into the labour market. The core funding ended in 2013, although the pan-Canadian network organizations continue to be eligible for project-based funding.

Most interviewees agreed that the elimination of core funding to organizations resulted in the closure of a number of organizations. However, there were divergent views as to the effects of this change on literacy and essential skills outcomes in Canada.

Many interviewees agreed that some organizations were not accomplishing significant results when they were being funded. A similar proportion of interviewees and survey respondents did not identify any negative impacts resulting from the cuts to core funding. Just over half of survey respondents observed positive instead of negative impacts to: their organization’s capacity to address literacy and essential skills issues (42% positive, 38% negative), the integration of literacy and essential skills into organizational policies (53% positive, 24% negative) and into programs and services (42% positive, 36% negative). The exception to this was on the quality of literacy and essential skills opportunities and resources for adult Canadians (where 42% saw a negative impact and 40% saw a positive impact). These were mostly interviewees and survey respondents who did not have direct involvement with those organizations whose core funding was cut. However, some interviewees indicated that organizations which had been effective and doing good work also had their funding discontinued, and in their view impeded literacy and essential skills results.

In contrast, interviewees and survey respondents from organizations which had received core funding, or that worked with these organizations, saw the impacts of the cuts as negative, particularly in relation to: the integration of literacy and essential skills into programs and services (88% indicating a moderate or large negative impact); and the quantity and quality of literacy and essential skills opportunities and resources for adult Canadians (88%).

Outcomes of the roadmap for Canada’s official languages 2013 to 2018

The results of the review of the Official Language Minority Communities Literacy and Essential Skills Initiative that was the Office’s contribution to the Roadmap for Canada’s Official Languages 2013 to 2018, found that projects completed by 2015 to 2016 have contributed to the achievement of Roadmap outcomes through the strengthening of partnerships and the enhancement of knowledge and capacity among stakeholders.

A few interviewees indicated that the elimination of core funding may have had a particular impact on the official language minority communities. Because the community is a minority and thus small, the closure of key organizations after the elimination of core funding were reported to have had a significant impact on the stakeholders working in this area, reducing their cohesion and activities.

4.5 Operational efficiency

The evidence is mixed as to whether the operational transition of literacy and essential skills grants and contributions delivery has had an observable impact on efficiency. Departmental processes have been streamlined, simplified and standardized to an extent, but the administrative burden for recipient organizations remains high, and interactions with the department remain complicated.

In 2012, as a result of the ESDC Grants and Contributions Modernization Agenda, the responsibility for literacy and essential skills operations functions were transferred to the Program Operations Branch. The intent of this was to improve the design, administration, and delivery of grants and contributions programs by, among other things, reducing administrative burden for applicants and recipients and streamlining and simplifying grants and contributions processing.

According to interviewed personnel, most processes have been streamlined and standardized. For example, one single unit (Program Operations Branch) has been assigned the task of reviewing all grants and contributions applications which ensures consistency, approval processes have been streamlined, and forms and templates have been standardized.

Evidence regarding whether there has been a reduced administrative burden for recipient organizations was mixed. Surveyed respondents from funding recipient organizations indicated that the delivery of services is complicated for funding recipient organizations and the majority of respondents found the funding application process somewhat complicated (54%) to very complicated (28%).

Similarly, the majority of respondents assessed the administrative requirements as medium or high, and very few said that it is low. Other interactions with the Department were said to be somewhat complicated (43%) and very complicated (20%), with a small proportion indicating they are somewhat simple (23%).

5.0 Conclusions and recommendations

5.1 Conclusions

There is a continued need for literacy and essential skills programming and activities in Canada

There are many employees who need improved literacy and essential skills to maintain their jobs, to advance, to improve productivity, and to develop the flexibility necessary to effectively navigate the changing labour market. There are also many people without jobs, who are either currently part of the labour force, or who face barriers to joining it, who need improved literacy and essential skills in order to join the labour force and secure employment. Given that an ageing workforce and shrinking labour market is one of the major challenges the Canadian economy is expected to confront in coming years, developing the literacy and essential skills of Canadians is key to a healthy economy.

There is demand from employers for better literacy and essential skills in the workplace. However, evidence also suggests that there is a lack of awareness about literacy and essential skills issues; for instance, some employers may be unaware that low literacy and essential skills are one of the causes of their challenges with their workforce, and of the ways in which those issues impact the workplace. Employers also sometimes are challenged to find ways to assess the literacy and essential skills and needs of their employees.

There is a need and a role for the federal government in literacy and essential skills programming

The evidence confirms that there is a need for the federal government to provide programming to support the literacy and essential skills of Canadians to improve labour market outcomes. Because of the government’s pan-Canadian scope, coordination and communication between stakeholders, including provinces and territories, is an important part of this role. While the Office does some coordination and communicates well with some provinces, territories and stakeholders, especially those with whom they cooperate on projects, this communication is uneven across the country, and there is a desire from some provinces and territories for more.

There is opportunity for the Department to work more closely with the provinces to develop a common plan of action on literacy and essential skills and to increase the integration of literacy and essential skills into the Labour Market Transfer Agreements. Concerns exist about the lack of flexibility in the Department’s programming, such as the requirement for projects to use the “nine essential skills” framework which may, from the point of view of provinces and territories, be considered different than their frameworks.

The Office’s activities do not duplicate those of other funders

The Office’s activities do not duplicate those of other funders, but are complementary. However, there do not seem to be a plethora of other funders in the area of literacy and essential skills and indications are that the needs are greater than the supply of funding. Provinces and territories have limited budgets, and some provide more funding for literacy and essential skills than others.

There is a need for the Office’s activities beyond grants and contributions funding

There is also a need for the Office’s activities that are not related to funding projects, although there were divergent views regarding which activities should be part of their role. For example, most agreed that research and analysis is a needed role for the Office, but there was disagreement as to whether they should have a role in the building of knowledge and expertise and the development of partnerships and networks. In part, that disagreement reflects the needs and situations of the different informants. Therefore, this leads to the conclusion that future work in these veins should consider whom is being targeted as a user, ensuring that the tools being developed or research being undertaken is what is needed by that target user group.

Communication with provinces and territories has been limited, but shows recent signs of improvement

Although interactions do take place between federal, provincial and territorial governments with respect to adult literacy and essential skills activities, results indicate that communication and coordination between levels of government has been somewhat limited, often focused primarily on projects, and had decreased in the earlier period of the evaluation. However, there have been some recent improvements in communication within the last year.

Provinces and territories value communication from the Office, and there is room to make communication more regular and formalized.

Some efforts have been made to facilitate knowledge sharing and networking, but more can be done

Overall, evidence suggests that the Office has facilitated knowledge sharing and networking amongst partners and stakeholders to proactively disseminate results of research, and information on resources and tools to some extent. However, evidence suggests that these efforts have been modest and that more can be done.

The Office’s activities have achieved many of their expected short-term and intermediate outcomes, although it is too early to determine if they have led to systemic change

The Office’s activities have resulted in a number of positive outcomes contributing to the achievement of literacy and essential skills improvements within a labour market context. This has included the development of successful sustained partnerships and integration of literacy and essential skills priorities into policies and program, and organizational capacity-building. However, it is more difficult to assess their contribution to progressing towards sustainable systemic change as the shift in focus to systemic change began at the end of the evaluation period.

Performance information on projects exists, but is not systematic or uniform, and this information and related tools could benefit from updating

The Office does have a large amount of performance information available regarding outputs and direct outcomes, through quarterly reports, provincial profiles and project close-out reports. However, the information available is inconsistent in the quality, quantity and type which makes it challenging to do a comprehensive assessment of progress towards achieving expected outcomes. Additionally, the Adult Learning, Literacy and Essential Skills Program Performance Measurement Strategy and the Office’s logic model could benefit from being updated.

The evidence is mixed as to whether the operational transition of literacy and essential skills grants and contributions delivery has had an observable impact on efficiency

Departmental processes have been streamlined, simplified and standardized to an extent, but the administrative burden for recipient organizations remains high, and interactions with the department remain complicated.

5.2 Recommendations

Recommendations based on the evaluation findings are provided below.

1. Consider working with provinces, territories and partners to develop formal partnership strategies to support stakeholder network development and the sustainability of effective approaches

The Office plays a key role in encouraging the development and co-ordination of partnerships and networks to support the integration of literacy and essential skills into labour market programming. However, formal, clearly defined partnership strategies focusing on the development of stakeholder networks across Canada and the resources required to ensure sustainability would help improve understanding of, and coordination with, the literacy and essential skills needs and integration efforts of provinces, territories, and partners. This could include integration of literacy and essential skills into labour market programming (for example, Labour Market Transfer Agreements).

Additionally, working with provinces, territories and partners to develop a formal partnership strategy that includes approaches to stakeholder network and resource development would indirectly help stakeholders transition from a core-funding approach to one focused on sustainability and scaling up or replicating effective interventions. A few interviewees requested that consideration be given on ways to support the official language minority communities’ networks on literacy and essential skills, saying that they tend to have fewer resources available to them than other organizations.

2. Continue to improve communications with provinces and territories, partners and stakeholders

The evidence shows that the Office has made efforts to improve communication and collaboration with partners and stakeholders since the last evaluation (2012). However, there is room for improvement in the frequency and the structure (that is a formal structure) of communications, particularly with relation to changes in priorities and/or funding, and with information sharing on research, tools and resources.

3. Update the performance measurement information and related tools to reflect recent changes

The performance measurement information (for example, logic model, indicators, data collection) and related tools should be updated to provide more consistent, reliable and relevant measures related to the achievement of expected outcomes and that could increase the likelihood that the next evaluation can assess the Program’s contribution to systemic change. It may also be beneficial to consider having one single logic model and performance measurement strategy for the Office and the grants and contributions programs (including the role of the Program Operations Branch) in order to tell a comprehensive performance story.

The existing logic model for the Office (developed in 2008) and performance measurement strategy for the Adult Learning, Literacy and Essential Skills Program (developed in 2011) should reflect the recent strategic changes. These should be updated to reflect the changes in outputs and outcomes (for example, funding of core activities is no longer a relevant output), as well as to reflect the shift to focus on integrating literacy and essential skills into labour market programming. Indicators and associated data collection tools should also be updated accordingly to provide valid, reliable measures to inform the achievement of outcomes.

Consideration should also be given to revising the internal quarterly performance reports completed by the Office, as well as the close out reports as data collection tools. The type of data provided in these reports is dependent on what recipients choose to report in their activity reports and in Program Operations Branch’s close out reports. The type of information collected in these reports has been inconsistent in terms of providing data for indicators that are specific to the achievement of outcomes.

Appendix A – Evaluation questions

Evaluation issue/Question	Section of this report	Source documents
Relevance – Continued need for the program
1. Is there a continued need to support the literacy and essential skills of Canadians to improve labour market outcomes?	3.1	Literature and Document Review Key Informant Interviews
2. Is there a link between program^{Footnote 7} objectives and federal government priorities and departmental strategic outcomes?	3.2	Literature and Document Review Key Informant Interviews
3. What are the roles and responsibilities of the federal government to support the literacy and essential skills of Canadians to improve labour market outcomes?	3.3	Literature and Document Review Key Informant Interviews
Performance – Effectiveness
4. Is there still a need for the Office to facilitate knowledge sharing and networking amongst partners and stakeholders to disseminate results of research, and information on resources and tools?^{Footnote 8}	4.1	Literature and Document Review Key Informant Interviews
5. Has the Office improved the communication and coordination between the federal, provincial and territorial governments with respect to adult learning, literacy and essential skills activities focused on improving Canadians labour market outcomes?^{Footnote 9}	4.1	Administrative Data and File Review Key Informant Interviews
6. Does the Office have performance information to accurately measure and report on its outputs and outcomes on a regular basis?	4.2	Administrative Data and File Review Key Informant Interviews
7. Have the Office’s activities contributed to the achievement of literacy and essential skills outcomes within a labour market context?	4.3	Administrative Data and File Review Key Informant Interviews Survey of Funding Recipient Organizations
8. Has the Office’s funding components (Adult Learning, Literacy and Essential Skills Program and National Essential Skills Initiative) progressed towards achieving the LES expected outcomes, especially those related to producing systemic or institutional change?^{Footnote 10} ^{Footnote 11}	4.4	Administrative Data and File Review Key Informant Interviews Survey of Funding Recipient Organizations
9. Has the Office facilitated knowledge sharing and networking amongst partners and stakeholders to proactively disseminate results of research, and information on resources and tools?	4.4	Administrative Data and File Review Key Informant Interviews Survey of Funding Recipient Organizations
Performance: Efficiency
10. Has the operational transition of literacy and essential skills grants and contributions delivery had an observable impact on efficiency?	5.1	Administrative Data and File Review Key Informant Interviews Survey of Funding Recipient Organizations
11. Has the Office implemented strategies to reduce resource expenditures and maximize achievement of outcomes?	5.2	Administrative Data and File Review Key Informant Interviews

Appendix B – Methodology

Multiple lines of evidence were used to inform the literacy and essential skills evaluation, including: a literature and document review, administrative data and file review, key informant interviews and a survey of funding recipients.

Evidence is presented in the report using the following scale to depict the frequency of the comment:

“A few” indicates less than 25% respondents (but at least 2)
“Some” indicates between 25% and 49% of respondents
“Half” indicates 50% of respondents
“Most” or “majority” indicates between 51% and 74% of respondents
“A large majority” indicates 75% or more of respondents

One limitation of the evaluation is that no consultations were undertaken directly with employers, or with beneficiaries of the programming (that is the people who received training or support in the development of literacy and essential skills). However, in order to get some indication of an employer perspective, the survey results for respondents who represented employer organizations were analyzed separately for select questions.

Literature and document review

A literature and document review was produced for the 2012 evaluation and was updated to inform the current evaluation.

The literature and documents reviewed included material that is internal to the Government of Canada, including but not limited to the Ministerial Mandate Letters, Employment and Social Development Canada’s Report on Plans and Priorities, the Speech from the Throne, Budget documents, legislation, and program documents. External material, including academic research, information from provincial and territorial websites, and organizational reports, as well as other relevant material were also analyzed. The review included both peer-reviewed and grey literature.

Administrative data and file review

The administrative data and file review addressed particular evaluation questions related to the performance of literacy and essential skills activities, including the achievement of expected outcomes. There were two components of the literacy and essential skills administrative data and file review. The first component examined the administrative data for the Office’s internal activities while the second component analyzed data for the grants and contributions programs.

The Office has produced quarterly performance reports on its internal activities from fiscal year 2011 to 2012 to fiscal year 2014 to 2015. The reports provided updates on indicators for select outputs and outcomes. This data was primarily in the form of textual information contained within word documents.

A review of the administrative data and files from projects funded under the literacy and essential skills grants and contributions programs, the Adult Learning, Literacy and Essential Skills Program and the National Essential Skills Initiative, between fiscal years 2011 to 2012 and 2015 to 2016 was conducted. The Common System for Grants and Contributions, the departmental database for the administration of grants and contributions, was an important source of information on the funding programs. All Close-Out Summary reports of projects completed in fiscal years 2011 to 2012 to 2015 to 2016 were reviewed.

Limitations: There were some cases where the evidence in the review of Close-Out Reports, administrative data, and other files did not agree. As well, these sources also contained differing amounts of detail on projects. However, it was possible to identify trends in each and to determine a similarity or difference among these trends, as well as to use the different sources to complement information where it would be missing in another source. There wasn’t explicit data supporting whether the Office is conducting its activities and funding with greatest efficiency, and whether it is, through its projects, implementing strategies to reduce expenditures and maximize outcomes. Indicators trying to measure the extent of achievement of outcomes were often challenging to establish. There is a lot of variability in the data available, and it is difficult to make generalizations about all of the reports, files and data examined.

Key informant interviews

Key informant interviews were used to answer all of the evaluation questions both on relevance and performance. Thirty-three key informant interviews were conducted with representatives from four groups. The groups, which provided a diversity of views about Employment and Social Development Canada’s literacy and essential skills activities, included:

The Office’s and Program Operations Branch personnel at national headquarters (3);
The Office’s partners (11);
Provincial and territorial representatives (10); and
Non-governmental experts in adult literacy and essential skills (9).

These key informant groups represented a range of views, both internal and external to the Government of Canada. They provided a balanced understanding of the relevance and results of literacy and essential skills activities.

Limitations: The level of familiarity with the Office varied by respondent, with some being familiar with the Office, some with one of the funding programs, and some with little familiarity. Therefore, not all respondents could comment on all of the evaluation questions.

Survey of funding recipient organizations

Recipient organizations of funding from the Adult Learning, Literacy and Essential Skills Program and the National Essential Skills Initiative, whose projects were completed between fiscal years 2009 to 2010 and 2015 to 2016^{Footnote 12} were surveyed to collect information on the achievements of funded projects.

The survey of funding recipient organizations focused on whether program outputs (for example tested models to support the development of literacy and essential skills, resources to support learning, workshops, training sessions, etc.) were achieved and whether these products were sustained and integrated into programs, services, and policies.

In all, 137 survey invitations were sent out. Of those 137, 51 were invalid contacts as their emails bounced back and could not be reached by phone. Additionally, 3 individuals were ineligible as they were not the right person to complete the survey or were not familiar enough with the funding topic. The survey of funding recipient organizations received 61 complete responses online, for a response rate of 73.5%. The largest number of respondents (77%) were from non-profit/charitable organizations and the voluntary sector. Approximately 8% were from educational organizations such as universities and colleges. Respondents were most commonly working to improve adult learning and education, employability skills, literacy and basic skills and essential skills training in the workplace.

The Office is conducting an impact study for projects completed between April 1, 2009 and March 31, 2014 related to the labour market. The survey of funding recipient organizations will be used to provide information for the impact study. The questions asked in the survey needed to meet the needs of both the LES evaluation and the impact study. Consent was sought from survey respondents to share their responses with the Office.

Limitations: The survey received 61 responses, for a response rate of 73.5%. Due to a sizable number of bounce backs (37%), the resultant size of the sample was not large and it is not possible to know if there were important differences between projects for which the email bounced back and those for which the email was successful. Thus, some findings should be interpreted with caution.

Footnotes

Footnote 1

The survey measures the key cognitive and workplace skills, including literacy skills, needed for individuals between the ages of 16 and 65, to participate in society and for economies to prosper. It defines “literacy” as the “ability to engage with written texts (print-based and digital) and thereby participate in society, achieve goals, and develop their knowledge and potential”.

Return to footnote 1 referrer

Footnote 2

Canada’s official language minority communities are Anglophones in Quebec and Francophones outside of Quebec.

Return to footnote 2 referrer

Footnote 3

ALLESP (Adult Learning, Literacy and Essential Skills Program), NESI (National Essential Skills Initiative). Funding provided by these two programs are from distinct sources, but there is no distinction between the types of projects funded.

Return to footnote 3 referrer

Footnote 4

Lane and Murray (2015), p. 14.

Return to footnote 4 referrer

Footnote 5

Lane and Murray (2015), p. 36.

Return to footnote 5 referrer

Footnote 6

Heisz, Notten, and Situa (2016), Cartwright (2011), Stuckey and Munro (2013), Windisch (2015).

Return to footnote 6 referrer

Footnote 7

The program includes the Office of Literacy and Essential Skills, the Adult Learning, Literacy and Essential Skills Program and the Employment Insurance Part II—National Essential Skills Initiative.

Return to footnote 7 referrer

Footnote 8

Question 4 refers to the recommendation from the previous evaluation, “to more fully develop into a Centre of Expertise, the Office should be more proactive in the dissemination of the Adult Learning, Literacy and Essential Skills Program funded research, resources and tools.”

Return to footnote 8 referrer

Footnote 9

Question 5 refers to the recommendation from the previous evaluation that “the Office should continue to improve coordination between the federal government and provincial and territorial government representatives with respect to adult learning, literacy and essential skills, as well as improve transparency regarding its decisions in relation to priority-setting for the program and the funding of individual projects.”

Return to footnote 9 referrer

Footnote 10

Question 8 will include an assessment of the impact of eliminating core funding to LES network organizations on the LES network itself and on service providers.

Return to footnote 10 referrer

Footnote 11

For question 8, “systemic or institutional change” is associated with the integration of literacy and essential skills into policies, programs, practices, training, accreditation, etc.

Return to footnote 11 referrer

Footnote 12

The survey of funding recipient organizations goes beyond the scope of the evaluation by including organizations whose labour market-related projects closed in fiscal year 2009 to 2010. The inclusion of the 2009 to 2010 labour market projects is to support the Office’s “Office of Literacy and Essential Skills Grants and Contributions Performance Assessment”.

Return to footnote 12 referrer

Page details

Date modified:: 2022-04-19

Evaluation of Literacy and Essential Skills

On this page

Alternate formats

Executive summary

Key findings

Relevance of the program

Effectiveness of the program

Efficiency of the program

Recommendations

Management response

Introduction

Recommendations

Recommendation #1

Response

FPT Literacy and essential skills network

eLearning FPT advisory committee

Essential skills and apprenticeship community of practice

Government of Canada federal network on essential skills

Recommendation #2

Response

Communication strategy

Project database

Recommendation #3

Response

1.0 Introduction

2.0 Program context and description

2.1 Context

2.2 Background

2.3 Program budget

3.0 Relevance findings

3.1 Continued need for program

3.2 Alignment with government priorities and federal roles and responsibilities

Need for federal government support

Need for conducting and disseminating research and analysis

Need to build knowledge and expertise

Need to develop tools and supports

Need for partnerships and networks

Complementarity rather than duplication

4.0 Performance findings

4.1 Coordination, knowledge sharing and dissemination of information

Provinces and Territories

Partners and stakeholders

4.2 Adequacy of performance information

4.3 Outcomes: LES in the Labour Market

Partnerships

Provision of tools and supports

Capacity-building

Integration into policy and program

4.4 Outcomes: Progress toward systemic change

LES opportunities and resources created

Elimination of core funding to organizations

Outcomes of the roadmap for Canada’s official languages 2013 to 2018

4.5 Operational efficiency

5.0 Conclusions and recommendations

5.1 Conclusions

There is a continued need for literacy and essential skills programming and activities in Canada

There is a need and a role for the federal government in literacy and essential skills programming

The Office’s activities do not duplicate those of other funders

There is a need for the Office’s activities beyond grants and contributions funding

Communication with provinces and territories has been limited, but shows recent signs of improvement

Some efforts have been made to facilitate knowledge sharing and networking, but more can be done

The Office’s activities have achieved many of their expected short-term and intermediate outcomes, although it is too early to determine if they have led to systemic change

Performance information on projects exists, but is not systematic or uniform, and this information and related tools could benefit from updating

The evidence is mixed as to whether the operational transition of literacy and essential skills grants and contributions delivery has had an observable impact on efficiency

5.2 Recommendations

1. Consider working with provinces, territories and partners to develop formal partnership strategies to support stakeholder network development and the sustainability of effective approaches

2. Continue to improve communications with provinces and territories, partners and stakeholders

3. Update the performance measurement information and related tools to reflect recent changes

Appendix A – Evaluation questions

Appendix B – Methodology

Literature and document review

Administrative data and file review

Key informant interviews

Survey of funding recipient organizations

Page details