Departmental Evaluation Plan 2021 and 2022 to 2025 and 2026
On this page
- List of acronyms
- Deputy Head Departmental Evaluation Plan confirmation note
- Executive summary
- Introduction
- Accomplishments in 2020 to 2021
- Results
- Evaluation coverage in 2020 to 2021
- Challenges and opportunities
- Resource allocation
- Going forward
- Annex 1 – Evaluation governance
- Annex 2 – Planned annual coverage table
Alternate formats
Departmental Evaluation Plan 2021 and 2022 to 2025 and 2026 [PDF - 1.9 Mo]
Large print, braille, MP3 (audio), e-text and DAISY formats are available on demand by ordering online or calling 1 800 O-Canada (1-800-622-6232). If you use a teletypewriter (TTY), call 1-800-926-9105.
List of acronyms
- CPP
- Canada Pension Plan
- EI
- Employment Insurance
- ESDC
- Employment and Social Development Canada
- GBA+
- Gender-Based Analysis Plus
- LMDA
- Labour Market Development Agreements
- OAS
- Old Age Security
- PMEC
- Performance Measurement and Evaluation Committee
- TBS
- Treasury Board Secretariat
Deputy Head Departmental Evaluation Plan confirmation note
I approve the Departmental Evaluation Plan of Employment and Social Development Canada for 2021 to 2026. I submit it to the Treasury Board of Canada Secretariat as required by the Policy on Results.
I confirm that this 5 year rolling Departmental Evaluation Plan:
- plans for evaluation of all ongoing programs of grants and contributions with 5-year average actual expenditures of $5 million or greater per year at least once every 5 years, in fulfillment of the requirements of subsection 42.1 of the Financial Administration Act
- meets the requirements of the Mandatory Procedures for Evaluation; and
- supports the requirements of the expenditure management system including, as applicable, Memoranda to Cabinet, Treasury Board Submissions, and resource alignment reviews
I will ensure that this plan is updated annually. I will provide information about its implementation to the Treasury Board of Canada Secretariat, as required.
Graham Flack
Deputy Minister of Employment and Social Development
Executive summary
In 2020 to 2021, ESDC’s Evaluation Directorate continued to fulfill its mandate to support evidence-based decision-making at the Department. Evaluators adjusted rapidly to the emergency circumstances created by the COVID-19 pandemic, providing strategic and tactical support to decision-makers across the Department. They maintained close partnerships with program officials with a view to developing evaluation evidence and providing trusted advice based on their knowledge of ESDC’s programs, and expertise in the conduct of various assessment methods. At the same time, the Directorate succeeded in consulting broadly with partners on the timing and scope of future evaluations that are targeted for completion between 2021 and 2026.
Evaluation Directorate accomplishments in 2020 to 2021
Supported evidence-based decision making
- Completed 8 evaluation reports and 24 technical studies to assist program officials
- Provided sound advice to support ESDC’s COVID-19 response
- Extended Labour Market Program Data Platform to conduct net impact analysis for the Opportunities Fund for Persons with Disabilities
- Provided evaluation evidence to enable sound decision-making for ESDC’s Labour Market programs
87%: Percentage of evaluation clients who thought the evaluations contained valid, evidence-based findings
Shared knowledge through dissemination & engagement
- Shared best practices with the Organization for Economic Co-operation and Development (OECD)
- Published 8 reports online and delivered 5 virtual presentations on evaluation techniques
- Promoted public awareness of ESDC evaluations through social media platforms
90%: Percentage of evaluation clients who believed that evaluators’ recommendations were realistic and actionable
Broadened the trusted advisor role
- Advised program officials and Head of Performance Measurement on 7 Performance Information Profiles, including regarding quality of performance indicators
- Assisted a project by Service Canada to identify lessons learned and good practices during the time of the Canada Emergency Response Benefit call centre operations
- Informed development of COVID-19 measures with existing body of evaluation evidence and agile assessments to produce early knowledge
- Supported EI Commissioner of Employers’ and Workers’ Forums through evaluation findings and studies
Challenges and opportunities for the evaluation function
1. Human resource capacity
The Directorate is responding by:
- continuing to recruit a diverse and inclusive workforce
- supporting the professional development of staff
- empowering staff to remain healthy, positive and engaged as they work remotely
- using flexible staffing options
2. Quality of ESDC’s data and performance measures
The Directorate is responding by:
- providing advice on the quality of Performance Information Profiles to support evaluations and ESDC’s COVID-19 response
- assessing data and performance indicators in evaluability assessments and strategies
- leveraging machine learning to determine what labour market programs work for which clients
- increasing use of datasets in collaboration with Statistics Canada
Going forward
In the coming years, the Directorate’s evaluations will look at ESDC’s COVID-19 response. This work will encompass a range of programs dealing with labour market development, learning and income security. In addition to these evaluation activities, evaluators will assess the impact of the Employment Insurance Program, the Canada Education Savings Program, and the Canada Pension Plan Disability Benefits, among other programs.
The Directorate will cooperate with program officials to continue to furnish timely evidence for strategic and tactical decision-making and ensure that the scope and timing of evaluations meet the evolving needs of the Department. The plan is to complete 10 evaluations in 2021 to 2022 and 13 in 2022 to 2023.
The Directorate will continue to expand its ability to use innovative data analysis techniques, such as machine learning, to identify program impacts. The Directorate remains committed to ensuring the professional development of its staff. This will help staff to adapt to the increasingly demanding work of program evaluation. Efforts to recruit and retain staff will continue to emphasize maintaining a diverse and inclusive workforce.
Introduction
Purpose of the report
As per the Treasury Board of Canada Secretariat (TBS) Policy on Results, each department must develop and publish an annual 5-year departmental evaluation plan. The purpose of this plan is to assist the Deputy Head to ensure the availability of factual, neutral, and timely information on the department’s programs. This information supports evidence-based decision-making and helps keep Canadians informed on departmental results.
Rigorous planning is important to ensuring the effectiveness of ESDC’s evaluation function. The Directorate determines the timing of evaluations by consulting with program officials and central agencies on the basis of need, risk, and priorities. The Plan also helps ensure transparency in the decision making process for the evaluation function, highlighting priorities and coverage decisions.
Departmental context
Employment and Social Development Canada’s mandate is to build a stronger and more competitive Canada, to support Canadians in:
- helping them live productive and rewarding lives; and
- to improve Canadians’ quality of life
The Department delivers a range of programs and services that affect Canadians throughout their lives.
The Department’s portfolio includes Employment and Social Development, the Labour Program, and Service Canada.
- Employment and Social Development delivers a range of programs and services that assist Canadians throughout their lives. It provides seniors with basic income security, supports unemployed workers, helps students to finance their post-secondary education, and assists parents who are raising young children. The Department also assists persons with disabilities and their families, and supports the homeless-serving sector through Reaching Home: Canada’s Homelessness Strategy
- The Labour Program fosters safe, healthy, fair and inclusive work environments and cooperative workplace relations in the federal jurisdiction
- Service Canada serves as a single point of access for some of the Government’s largest and most well known programs and services. This includes Employment Insurance (EI), Old Age Security (OAS), Canada Pension Plan (CPP), the Social Insurance Number, and the Passport Program
ESDC played a key role in supporting the Government of Canada’s significant and decisive actions to support Canadians and businesses facing hardship due to the COVID-19 pandemic. This included developing an array of programming for individuals and families, including Indigenous people and self-employed individuals. ESDC’s COVID-19 measures also targeted various non-governmental organizations and non-profit sectors helping Canadians.
In 2019 to 2020, the Department spent $144.4 billion on programs and servicesFootnote 1. Most of this spending directly benefited Canadians through statutory transfer payment programs, including EI, CPP, and OAS. The Department also administers employment benefits and support measures, which include billions of dollars in transfers annually to provinces and territories. Finally, in 2020, ESDC delivered a range of emergency benefits to Canadians in collaboration with the Canada Revenue Agency. These benefits, which included the Canada Emergency Response Benefit, helped those struggling with the impact of COVID-19.
The Department interacts with Canadians across the country. These interactions occur through online service channels, over the telephone, and through a regional network of in-person sites. The Evaluation Directorate systematically collects and analyzes evidence in a neutral fashion to support the Department’s decision-making and inform Canadians. Evaluators look at the relevance, effectiveness and efficiency of departmental programs and services.
ESDC programs touch the lives of Canadians across the country. The Department works to ensure that these programs and services meet the diverse needs of all Canadians. Evaluators use a GBA+ analytical framework when carrying out their work. The most recent Mandate Letters for the 4 Ministers of the Department call on them to apply GBA+ when making decisions.
Evaluation Directorate mandate
Evaluation Directorate collaborates and shares knowledge with program officials across the Department to support evidence-based decision-making. The Directorate seeks input from program officials and senior management on the evaluation process through an inclusive approach to governance (Annex 1).
Evaluation Directorate supports program officials at all stages of the policy life cycle. At the initial stage, officials use evaluations to help inform the development of new programs and policies. During implementation, officials use evaluation recommendations to guide adjustments or program renewal. As a program or policy matures or winds down, officials can use evaluations to inform lessons learned for future programs or policies.
Evaluation Directorate occasionally advises program officials that are developing responses to the recommendations found in final reports. Evaluators can assist officials with developing action plans for recommendations. The Directorate monitors progress on the implementation of action plans. This promotes a system of continuous improvement throughout the evaluation cycle.
Accomplishments in 2020 to 2021
The Evaluation Directorate’s products focused on key knowledge gaps and on the relevance, effectiveness and efficiency of programs. Most evaluations included recommendations for program improvement.
Supporting evidence-based decisions
Evaluation influenced and supported evidence-based decision making at ESDC by:
- completing 8 evaluation reports and 24 technical studies to support program officials, and the EI Monitoring and Assessment Report
- advising on 23 Treasury Board Submissions and 9 Memoranda to Cabinet to inform policies and programs
- providing information and advice to ESDC so that it can make sound decisions on labour market and social development programs. These include programs focusing on the unemployed, Indigenous peoples, persons with disabilities and vulnerable youth; and
- expanding its use of innovative methodologies for impact evaluation
Disseminating and engaging
Despite disruptions caused by the COVID-19 pandemic, the Evaluation Directorate continued to disseminate knowledge and engage with partners inside and outside of the federal government in 2020 to 2021. The Directorate produced and published 8 evaluation reports online. It also generated technical reports, one-page summaries, lessons learned syntheses, conference presentations, and supplemental studies. These activities spanned 3 key areas:
- 1. Publishing, presentations and engagement
- Collaborated with the Organization for Economic Co-operation and Development (OECD) to share best practices for leveraging administrative data and using state-of-the-art methodologies to support impact evaluations. This will assist countries wishing to strengthen their capacity for labour market policy evaluation
- Delivered a virtual presentation to the Canadian Evaluation Society’s webinar on Technology and Evaluation. The presentation covered the efficient use of administrative data and machine learning for net impact evaluations
- Delivered a nationwide presentation to ESDC staff on approaches for developing qualitative information to support the evaluation of the Foreign Credential Recognition Program
- Published 3 technical reports related to seniors benefits. This included reports on the take-up of the Guaranteed Income Supplement, the labour market impact of a work earnings exemption on benefits for low-income seniors, and participation in the Old Age Security Program
- Collaborated with the Chief Data Office to share more data insights across the federal government. Evaluators and officials did this through the ESDC Data Community of Practice and the Government of Canada Data Conference
- Shared results through social media, including on LinkedIn
- 2. Supporting policy analysis
- Shared technical studies and evaluations with policy developers across the Department. These documents helped to inform policy analysis and program improvements, including in relation to student access to post-secondary education and persistence in education
- Shared evaluation results on labour market programs with decision makers to inform policy and program development, and the Government’s pandemic response and recovery measures
- 3. Advancing federal evaluation function
- Collaborated with TBS and other federal Heads of Evaluation via a steering committee to assess and advance the influence of the Evaluation function
- Presented 3 webinars to the Canadian Evaluation Society
The trusted advisor role
Over the course of 2020 to 2021, the Evaluation Directorate provided trusted advice to program officials and senior management. The advice included information to support ESDC’s development of recovery measures in response to COVID-19. The Directorate did this by:
- providing evaluation evidence to support labour market policy and program decisions by providing reliable and timely impact analysis
- assisting Service Canada with a project that identified lessons learned and good practices during the time of the Canada Emergency Response Benefit call centre operations. The Directorate provided advice on how to conduct focus groups and key informant interviews, how to analyze the data and synthesize results, and how to report the findings from them. This information might be used to improve Service Canada’s project management practices and to respond to a commitment to TBS. The lessons learned may also be used to help Service Canada to modernize its services in future
- providing evaluation evidence to support the development of certain measures announced in the Speech from the Throne, as well as the Budget
- supporting the EI Commissioners for Employers and Workers in examining labour market issues. Evaluators prepared 3 supplemental studies from the 2019 to 2020 work plan. They also supported the work of the Monitoring Report and Advisory Committee by aiding the coordination of the call for study proposals and their prioritization for the 2020 to 2021 work plan
- collaborating with Learning Branch on the assessment of the Canada Student Loans Program Skills Boost Measures. Evaluators and program officials examined the impact of the Skills Boost Measures on participants’ decision to return to school and their studies
- obtaining thorough peer reviews for all evaluation results and evaluation methodologies used by the Directorate. This approach represents an international best practice for ensuring evaluation rigour
- contributing to a new Memorandum of Understanding with Immigration, Refugees and Citizenship Canada on the Passport Program. The memorandum clarifies responsibilities between the 2 departments for providing and evaluating the program. It specifies that IRCC plays the lead role and ESDC is focused on work related to service delivery
- assessing the Migrant Worker Support Network pilot administered by the Temporary Foreign Worker Program in British Columbia. The goal of this work was to help inform decision-making related to the protection of migrant workers. The Directorate conducted interviews and focus groups, as well as an analysis of the pilot’s strengths, weaknesses, opportunities and challenges. These activities helped to assess whether the pilot achieved its expected outcomes
In response to COVID-19, Evaluation worked with partners across the Department to efficiently generate the knowledge required to inform program decisions and avoid duplication of work. Evaluation leveraged core evaluation products and decades of evidence gathered through robust evaluations to inform policy development and departmental priorities. This established body of evaluation evidence coupled with tactical support helped senior officials integrate the lessons from the past into their policy and service delivery advice and proposed options. Going forward, Evaluation will continue to provide real-time, agile, and tactical advice.
The Evaluation Directorate collaborates with the Employment Insurance Policy Directorate to provide input and analyze data for the annual Employment Insurance Monitoring and Assessment Report. This close collaboration fosters the timely coordination and production of quality supplemental studies to inform the Report. Both Directorates work together with members of the Monitoring Report and Advisory Committee to prepare an annual work plan. The plan includes supplemental studies and internal analysis with input and direction from both the EI Commissioner for Employers and the EI Commissioner for Workers. The list of studies and analyses are available in the annual Monitoring and Assessment Report. Given the scale and unique objectives of each EI benefit, the EI program is also subject to a number of evaluations.
Each year, the Evaluation Directorate administers a questionnaire to program officials. Between 2017 to 2018 and 2020 to 2021, the percentage of respondents indicating that the evaluation report for their respective program contained “valid, evidence-based findings and conclusions” increased by 12 percentage points, from 75% to 87%. The Directorate scored over 90% from respondents in 2020 to 2021 on professionalism and responsiveness. Evaluators are working to address areas where the Directorate rated lower. This includes providing officials with sufficient time to prepare documentation required for the evaluation.
Innovative approaches
As a central component of ESDC’s program innovation landscape, Evaluation uses advanced methods to develop an evidence base for program development. In 2020 to 2021, the Directorate pursued several innovative practices. The goal of this was to better estimate the impact of programs, disseminate results to inform decisions, and experiment with new approaches. These included:
- further developing the Labour Market Program Data Platform. Domestic and international observers have recognized the Platform as an innovative best practice for labour market performance measurement and net impact evaluation of labour market programming. Evaluation Directorate recently transferred the Platform to ESDC’s Chief Data Office to enable enhanced access
- leveraging the Labour Market Program Data Platform to examine the employment outcomes for participants of the Opportunities Fund for Persons with Disabilities. Evaluators used the Data Platform to analyze the net impact of the program’s numerous active labour market interventions
- leveraging a new database that integrates a number of internal and external data sources to support of the recent Canada Apprentice Loan evaluation. The Directorate used the integrated data to better estimate the impact of the loans when apprenticeships were completed. Going forward, evaluators can use the same database to assess the impacts of Canada Apprentice Loans on the labour market outcomes of recipients
- collaborating with Statistics Canada and Immigration, Refugees and Citizenship Canada to integrate data sources related to the Temporary Foreign Worker Program. This included integrating program data into the Canadian Employer-Employee Dynamics Database for the first time. Project partners also consolidated program data and Immigration, Refugees and Citizenship Canada’s Temporary Residents file. The data integration will enhance ESDC’s capacity to measure the impact of the program on the Canadian labour market
- connecting data sources so that ESDC could survey seniors who do not file income taxes. This activity was part of the Phase 2 evaluation of the Guaranteed Income Supplement. Non-tax filer information is not available to the program. By connecting administrative data and CRA income tax data, evaluators created a sampling frame of persons who may be eligible for the GIS but had not applied. The Directorate surveyed those non tax-filing seniors regarding their reasons for not applying for the program. Insights from the survey not only informed the evaluation, but will also support decisions on delivering seniors programming in the future
- summarizing results of labour market program evaluations for ease of reference and use by decision makers. In the coming year, similar broad results mapping will be applied to recent evaluations of Labour Market and Persons with Disabilities programs
- working with ESDC’s Innovation, Information and Technology Branch to use a pop-up survey to inform the Foreign Credential Recognition Program evaluation
- using social media platforms to contact students during the COVID-19 pandemic to gather data for the evaluation of the Canada Student Loans Program
- advising counterparts in other departments on streamlining processes for Management Action Plans
Results
Through 8 evaluations completed in 2020 to 2021, Evaluation Directorate measured results across all 5 core departmental responsibilities. Results from these evaluations include:
Simplifying and harmonizing foreign credential recognition processes to support internationally-trained individuals in Canada
The evaluation reported that the Foreign Credential Recognition Program improved the fairness, consistency, timeliness, and transparency of credential recognition processes. The evaluation assessed early results from the program’s Canadian Work Experience Pilot Project. The project’s goal is to help newcomers to overcome barriers due to lack of Canadian professional work experience. Evaluators found that the project helped participants to acquire Canadian experience and job search skills. Participants reported that work placements and coaching supports offered through the program were particularly helpful to them.
The evaluation looked at early results from the program’s Foreign Credential Recognition Loans project. This initiative provides money and support services to newcomers so that they can address barriers to employment. Many internationally-trained individuals continue to face multiple barriers as they seek to work in positions that are commensurate with their qualifications. Evaluators found low loan default rates from the initiative and positive employment outcomes. Moreover, two-thirds of recipients would have taken more time to complete their credential plan in the absence of the project’s supports.
Increasing the labour market attachment of persons with disabilities
The Opportunities Fund for Persons with Disabilities assists persons with disabilities to prepare for, obtain and maintain employment. The Directorate’s evaluation found the program was responsible for producing a 38% annual earnings increase for program participants over the 5 years after participation. This translates to about $1,100 more in earnings for participants than for non-participants with similar characteristics. In addition, a cost-benefit analysis found that a $1 investment in the Opportunities Fund yields $1.70 in return over a 10-year period following the intervention. This was particularly evident with the Skills for Employment component of the Fund. The Fund enables participants to undertake short-duration training and develop basic to advanced skills. The evaluation found that investing $1 in this type of intervention yields $4.40 in return over the 10 years following the intervention. A survey of employers partly informed the evaluation report and provided complementary evidence to support the impact analysis.
Helping apprentices to complete their training
The evaluation of the Canada Apprentice Loan shows that the program helps recipients complete training. An above-average proportion of recipients who were women, Indigenous, persons with disabilities or older in age indicated that the loans were important or very important in covering their expenses while pursuing their apprenticeships. Between 2015 and 2017, approximately one-third of the eligible population used the loan. Among those people, the gender balance was roughly even between women and men. Over that same period of time, only 4% of Canada Apprentice Loan recipients used the loan as their only source of financial support from the Government of Canada.
Removing barriers to seniors receiving needed financial support
This evaluation examined take-up and awareness of the Guaranteed Income Supplement among non-tax-paying seniors. There is a lack of recent income tax information for this segment of the population, which makes it difficult to assess their needs. The evaluation found that only approximately 15% of non-tax-paying seniors who were eligible for the Supplement actually applied for it. The evaluation found that barriers to take-up among non-tax-filing seniors related to difficulties accessing information about the Supplement. Seniors from all socio-economic groups experience these barriers.
Increasing the well-being of seniors
The evaluation found that, between 2015 and 2019, the New Horizons for Seniors Program contributed to the well-being of seniors and their communities. For example, 57% of Community-based organizations that ESDC funded through the program believed that their projects increased seniors’ socialization or reduced their social isolation. Nearly half of organizations that ESDC supported through the Pan-Canadian stream of funding indicated that their projects created social and community connections among seniors. This reduced barriers to inclusion. The evaluation also found that the program’s Collective Impact approach showed promise in addressing social isolation. Collective Impact projects bring together organizations from different sectors to produce significant changes in their community. The evaluation found that 92% of recipients engaged in Collective Impact projects indicated that this approach effectively solved community challenges. The Program also contributed to increasing the capacity of recipient organizations to support seniors’ initiatives.
Identifying the leading causes of sickness benefit claims
The recent evaluation of EI sickness benefits provides a practical example on how a supplementary survey can complement administrative data. With EI sickness benefits, ESDC does not require an individual to indicate the type of illness or injury before providing them benefits. Rather, the individual must obtain a medical certificate from a certified practitioner that indicates the number of weeks that they are unable to work. Evaluation Directorate conducted a supplementary survey to better understand the types of illnesses or injuries that are keeping claimants from working. The survey asked EI sickness claimants questions on the illness or injury that caused their separation from work. The survey found that, in 2018, 38% of cases related to chronic conditions. 37% of cases related to acute traumas, while 25% of cases related to episodic conditions. This information will help inform policy decisions on Employment Insurance in the future.
Evaluation coverage in 2020 to 2021
The Evaluation Directorate uses a risk-based approach when deciding the timing and order of future evaluations. This approach prioritizes mandatory evaluations. It also prioritizes evaluations that are most likely to inform major policy and program decisions.
In 2020 to 2021, the Evaluation Directorate completed all evaluations that were mandatory according to the Financial Administration Act and the Policy on Results.
The need to respond rapidly to the COVID-19 pandemic placed exceptional demands on the Department in 2020 to 2021. As a result, the Directorate carried over 2 evaluations scheduled for 2020 to 2021 to the following fiscal years. These were the Evaluation of the Canada Pension Plan Disability Benefits (reassessment) and the Evaluation of the Employment Insurance Seasonal Claimant Pilot.
The Directorate developed 12 evaluability assessments and 1 evaluation strategy in 2020 to 2021. These documents outline options for evaluating a given program or service. They contain information on the scope, methodology and timing of an evaluation project. The Performance Measurement and Evaluation Committee (PMEC) is responsible for reviewing and approving these documents. Evaluators always engage with program officials while developing evaluability assessments and strategies to obtain their input.
Planned Evaluation Reports | Completed (Deputy Minister Approval) |
---|---|
Foreign Credential Recognition Program | Completed: May 2020 |
Employment Insurance Sickness Benefits (Carried Over) | Completed: June 2020 |
Guaranteed Income Supplement Take-up – Phase 2 | Completed: September 2020 |
Canada Apprentice Loan – Phase 1 (Carried Over) |
Completed: September 2020 |
New Horizons for Seniors Program | Completed: December 2020 |
Opportunities Fund for Persons with Disabilities – Phase 2 (Carried Over) | Completed: January 2021 |
Job Bank (Carried Over) | Completed: March 2021 |
Canada Student Loans Program | Completed: April 2021 |
Consultations
ESDC rapidly developed a range of measures to respond to the COVID-19 pandemic in spring and summer 2020. During this time, evaluators engaged officials on key considerations for evaluating the department’s response.
Officials with specialized expertise participated in the discussions. This included internal researchers and representatives from the Chief Data Office, Integrity Services, and Internal Audit Services.
In September 2020, PMEC discussed broad evaluation considerations for COVID-19 measures. The considerations included the need to focus on critical evaluations given capacity constraints, avoid duplication, and ensure complementarity of efforts amongst players. Going forward, the Directorate will present evaluation strategies and evaluability assessments relating to COVID-19 measures to PMEC for approval. The proposed evaluation approaches will include a GBA+ lens, and analysis of marginalized groups (for example, racialized communities) when possible.
The Evaluation Directorate conducted its annual consultation process for this plan in fall 2020. As a result of these discussions, the Directorate has adjusted the priorities, scope, and timing of evaluations to:
- re-scope ongoing and planned evaluations to encompass COVID-19 related programming changes (for example, Work-Sharing Program, Reaching Home: Canada’s Homelessness Strategy)
- better measure program impacts on marginalized groups, such as racialized communities
- add an interim evaluation of the Youth Employment and Skills Strategy to provide early evidence on the impact of program adjustments. This will inform policy and program development
- minimize the burden placed on officials when the Directorate evaluates a program.
- promote complementarity of effort across the portfolio (for example, Internal Audit Services, Head of Performance Measurement, Chief Data Office, and research), ensuring the efficient generation of knowledge the department needed, while avoiding duplication
The evaluation planning process also included consultations with TBS officials in January 2021. Topics included possible thematic evaluations and GBA+ analysis, including evaluation and analysis of program impacts on persons with disabilities.
Risk assessment
When assessing the risks associated with a program, the Evaluation Directorate looks at several factors:
- program characteristics
- departmental risks identified in the Corporate Risk Profile, and
- risks specific to the Evaluation Directorate
This multi-level approach to risk assessment ensures that evaluators consider the perspectives of key program partners and internal risks.
Program risks considered in early stages of evaluation planning include:
- size of the population affected or targeted by the program
- program materiality (for example, value)
- program complexity
- time elapsed since the previous evaluation
- magnitude of change to the program since the previous evaluation
- knowledge gaps with respect to the program
The Directorate assesses risks specific to individual evaluations when developing evaluability assessments.
The Directorate considers key corporate risks for the Department, including the Risk-Based Audit Plan. In situations where audit and evaluation schedules overlap for the same program, Evaluation Directorate may partner with Internal Audit Services to conduct a joint audit/evaluation. This provides a means of efficiently examining performance and outcomes. The Evaluation Directorate is also working with departmental researchers to coordinate activities and identify opportunities for collaboration.
Challenges and opportunities
Human resources capacity
The nature of the Evaluation Directorate’s work requires it to recruit and develop its highly-skilled staff on a continuous basis. The Directorate also works to retain its employees over an extended period of time. The Directorate tasks its evaluators to develop complex lines of evidence. Evaluators need specialized skills to provide quality advice to program officials. These skills are important because the Directorate supports the Department’s ambitious COVID-19 response.
Despite disruptions caused by the pandemic, the Directorate continues to pursue improvements to recruitment and staffing processes. The Directorate uses flexible staffing tools to meet short-term capacity demands. These methods include hiring students, casual employees and consultants. The Directorate also hired a number of retired public servants with expertise in 2020 to 2021 on a temporary basis to fill specific capacity gaps.
In 2021 to 2022, Evaluation staff will work with colleagues from other branches to support the development of a special project to promote the hiring and retention of persons with disabilities in ESDC. This pilot project will test approaches and help determine what human resources and enabling activities work best in this area.
The Evaluation Directorate promotes the professional development of its staff. The Directorate’s approach in this area aligns with the Government of Canada’s Beyond 2020 initiative, which envisions an agile, inclusive and equipped workforce. The Directorate enables evaluators to build their professional networks within the broader evaluation community. The Directorate supports the Canadian Evaluation Society’s Emerging Evaluators Network. This is an informal group of evaluators from across government who discuss issues of importance to the evaluation community. The Directorate sponsors membership in the Canadian Evaluation Society for 40 evaluators and professional accreditation for 14 of those evaluators.
The Directorate also supports training opportunities for staff. Evaluators regularly undertake training sessions organized within the Directorate. This equips them to meet the challenges of their evolving profession. After the initial outbreak of COVID-19, the Directorate moved the sessions online so that evaluators could continue to learn while working remotely. The Directorate organizes weekly second language practice sessions geared towards improving the communication skills of staff in both official languages. This is part of an overall Directorate strategy to support bilingual capacity and advance experienced staff.
Despite COVID-19, the Directorate maintains an enriching and positive work environment. The Directorate responded rapidly to pandemic-related office closures in March 2020. It leveraged technology so that staff could move forward on their projects collaboratively with peers while working remotely. The Directorate also supported the mental and physical well-being of employees. It organized regular online sessions so that staff could share wellness best practices with respect to teleworking and fitness and provided flexibility to staff that faced challenges working from home, particularly parents and caregivers. Managers also maintained close contact with their team members online to keep them engaged.
In addition, an employee-led team designed and piloted the Health, Wellness and Greening of the Workplace Initiative in January 2020. The Initiative’s objective is to engage employees to promote health, well-being and greening of the workplace. In the context of COVID-19 and the new telework environment, the Directorate is re-focusing the Initiative on team-based social events and activities.
The Directorate has long supported the mental well-being of its staff. As part of these efforts, it is helping to develop the Department’s mental health framework. A continued focus on the mental health of employees is central to encouraging open discussion about issues that can affect employees and teams.
Quality of data and performance measurement to support evaluations
Directorate evaluations depend on timely, quality data to produce useful evidence. Evaluators use performance measurement information to determine the relevance of programs and services, and whether they deliver value for money.
There is a need in the Department to improve access in keeping with the objectives of Open Data. Better harnessing the administrative data of the department also enables improved policy and service delivery.
The Evaluation Directorate supports robust data collection and performance measurement practices across the Department. Evaluators advise PMEC and program officials on the quality of performance indicators in the Departmental Results Framework. The Directorate collaborates with ESDC’s Head of Performance Measurement when providing this advice. The Framework is a key means to track and communicate the Department’s results and financial information to the public.
Evaluators also offer advice on Performance Information Profiles. These documents outline the performance measurement approach for each ESDC program. Program officials develop and regularly update them. Evaluators collaborate with program officials and the Head of Performance Measurement to ensure that each Profile contains appropriate indicators and will be useful in supporting future evaluations.
The Evaluation Directorate is working closely with departmental partners to address data gaps, particularly in relation to ESDC’s COVID-19 response. This will include, in part, the Evaluation Directorate playing an advisory role to departmental monitoring and assessment activities on the Canada Emergency Response Benefit. Ongoing collaboration in this area will permit the Department to develop a better understanding of how several COVID-19 program interventions benefitted Canadians.
Evaluators also carefully address data issues when developing plans for program evaluations. These evaluability assessments or evaluation strategies include a review of a program’s performance information and the quality of data supporting the performance indicators. When evaluators identify data gaps that could hamper an evaluation, they will work with officials to develop approaches to mitigate those challenges.
The Directorate is developing the data and methodologies to conduct more refined GBA+ analysis of how program impacts might be different for specific socio-economic groups. Some administrative data contains information on various identity factors, for example:
- gender
- age
- level of education
- rural/urban
- families with children
- visible minorities
- Indigenous peoples, and
- persons with disabilities
The incorporation of GBA+ analysis into evaluations can help identify barriers to access, systemic inequality experienced by various groups and data gaps. It can also make evaluation findings more comprehensive by considering all relevant population groups.
Resource allocation
The COVID-19 pandemic was a strain on existing resources. The Directorate is working to incorporate plans for new and modified programs into the existing allocations, while carrying on previously planned evaluations. During the pandemic, some resources were redirected to more timely, on-the-ground analysis of evaluation evidence to inform program and policy development.
Non-salary expenditures remain low relative to historic standards. However, future evaluations of COVID-19 measures may require the Directorate to spend more on Operations and Management expenses over the coming fiscal years.
In 2021 to 2022, Evaluation’s budget remains largely unchanged from 2020 to 2021. The forecasted amount for salary expenditures for fiscal year 2021 to 2022 is $6.27M, and forecasted operations and maintenance is $1.62M. The Directorate will continue to focus on providing advice and planning evaluations of new and modified programs in addition to undertaking previously planned evaluations. It will manage its resources prudently in order to ensure it is capable of handling an expected workload increase over the near term. Having the capacity to address unexpected events in the future will necessitate the adoption of more flexible and judicious approaches while determining evaluation scope and approaches.
Annual directorate spending in millions ($)

-
Text description of figure 1
In Millions ($) 2010 to 2011 2011 to 2012 2012 to 2013 2013 to 2014 2014 to 2015 2015 to 2016 2016 to 2017 2017 to 2018 2018 to 2019 2019 to 2020 2020 to 2021 2021 to 2022 Salaries $5.65 $5.16 $5.17 $4.72 $5.51 $5.23 $4.58 $5.32 $5.13 $5.70 $6.46 $6.27 O&M $3.47 $4.06 $2.87 $2.55 $2.01 $1.02 $1.00 $1.13 $0.91 $1.30 $1.50 $1.62 Total Budget $9.12 $9.22 $8.04 $7.27 $7.52 $6.25 $5.58 $6.45 $6.04 $7.00 $7.96 $7.89 - Note: Figures rounded to the nearest tenth; 2021 to 2022 is a forecast.
Going forward
Budget 2021 included numerous ESDC initiatives that will be assessed and reflected in the next Departmental Evaluation Plan. The Directorate plans to complete 10 evaluations in 2021 to 2022 and 13 in 2022 to 2023. For 2022 to 2023, several evaluations will encompass COVID-19 measures. These include the Work-Sharing Program and Reaching Home: Canada’s Homelessness Strategy. The Directorate will complete an interim evaluation of the Youth Employment and Skills Strategy. The purpose will be to develop early evidence on the impact of new measures implemented by the program in response to COVID-19. The Directorate will evaluate the incremental impacts of participation in labour market programs, such as the LMDA, using net impact analysis and machine learning techniques.
The Directorate will assess the impact of several programs targeted towards students. These include the Student Work Placement Program and the Canada Education Savings Program. Evaluators will also look at the Canada Pension Plan Disability Benefit, as well as the Employment Insurance Program’s Seasonal Claimant Pilot. The findings of these and many other evaluations will inform improvements to ESDC’s programming for the benefit of clients.
During this period, Evaluation demonstrated how its body of evaluation evidence as well as its expertise in conducting various assessment methods can be strategically relied upon in support of the Department’s mandate. In the years ahead, the Directorate will continue this role by collaborating with policy and program officials to furnish timely evaluation evidence to inform decision makers.
The Directorate will continue to incorporate innovative methods and approaches into its operations and the services it provides to the Department and other stakeholders. It will consider pursuing more horizontal evaluations that investigate themes common to multiple programs or priorities. These evaluations could also look at specific client groups that the Department serves across a range of programs. Innovation will also be used for the dissemination of results, to enhance ease of reference and use.
The Directorate will engage experts in the academic community on how best to conduct evaluations. This will ensure that ESDC’s evaluators adopt best practices from other jurisdictions, departments and disciplines to achieve results for Canadians. The Directorate will engage with researchers to increase the use of administrative data. Furthermore, the Directorate will collaborate with federal counterparts to strengthen the evaluation function across government.
This document provides an evaluation coverage table in the attached annex (Annex 2 – Planned Annual Coverage Table). The table demonstrates the Directorate will meet all coverage requirements mandated by the Treasury Board of Canada Secretariat, the Policy on Results, and the Financial Administration Act. This is in addition to completing discretionary evaluations the Department has designated as priorities. The Evaluation Directorate will use flexibilities provided in the Policy on Results to prioritize evaluations of programs.
Programs of grants and contributions that do not have a 5-year average actual expenditure of $5 million or greater per year are not required to be evaluated every 5 years. These programs include:
- Canadian Benefit for Parents of Young Victims of Crime
- Sustainable Development Goals
- Skilled Trades and Apprenticeship Grants (Red Seal Program)
- Accessible Canada Initiative and,
- Strategic Engagement and Research Program
The Evaluation Directorate will continue to support the Department in making informed decisions. Evaluators will continue to adopt innovative approaches to better measure the impacts of ESDC programs.
As indicated in the 2018 Neutral Assessment of the Evaluation Function, the Directorate’s work in these areas is valuable in supporting the mission of ESDC and reporting on results. The Evaluation Directorate will build on the priorities that this plan outlines in order to deliver positive results for Canadians.
Annex 1: Evaluation governance
Performance Measurement and Evaluation Committee
Composed of partners from senior ranks of ESDC, including the Head of Evaluation.
- Approves the departmental evaluation plan
- Reviews evaluation reports, including management responses
- Approves Evaluability Assessments
- Discusses follow-up on action plans
Evaluation Advisory Committee
Composed of Stakeholders from across ESDC and/or other departments and agencies such as TBS Program Sector and Finance Canada (Director General level and below).
- Identifies options for evaluation scope
- Discusses preliminary findings
- Provides input for final report
Evaluation working group
Composed of partners from across ESDC (Director level and below).
- Focuses on technical aspects
- Performs data collection and analysis
- Provides input for preliminary report
Annex 2: Planned annual coverage table
Program | Last evaluation of the Program | Planned evaluation in the next 5 years | Planned fiscal year of approval | Reason for evaluation |
---|---|---|---|---|
Temporary Foreign Worker Program (TFWP) | Evaluation of the labour market opinion streams of the Temporary Foreign Worker Program (2010 to 2012) | Temporary Foreign Worker Program Evaluation | 2021 to 2022 First Quarter |
Commitment in TB Submission (Adjusted to 2021 to 2022 completion) |
Federal Mediation and Conciliation Service | Evaluation of the Federal Mediation and Conciliation Service - Phase 1 (2018) | Phase 2 Evaluation of Federal Mediation and Conciliation Service | 2021 to 2022 Second Quarter |
Departmental Priority |
Student Work Placement Program | Not Applicable. New Program. | Evaluation of Student Work Placement Program – Phase 1 | 2021 to 2022 Third Quarter |
Commitment in TB submission (2021 to 2022 completion) |
Canada Education Savings Program | Canada Education Savings Program: Summative Evaluation (2015) | Evaluation of the Canada Education Savings Program (Canada Learning Bond-focused) |
2021 to 2022 Third Quarter |
Deputy Minister Request |
Canada Pension Plan Disability | Summative Evaluation of the Canada Pension Plan Disability Benefit (2011) | Canada Pension Plan Disability (CPPD) Benefits (Reassessment) (Carried Over) |
2021 to 2022 Third Quarter | Departmental Priority |
EI Program – Regular Benefits | Evaluation of Initiatives to Extend EI Regular Benefits (2016) | Seasonal Claimant Pilot (Carried Over) |
2021 to 2022 Third Quarter | Commitment in TB Submission |
Union Training and Innovation Program | Not Applicable. New Program. | Union Training and Innovation Program Evaluation | 2021 to 2022 Fourth Quarter | Required by the FAA |
Workforce Development Agreements | Previous evaluations conducted involving various programs (such as, Labour Market Agreements (LMAs), Targeted Initiative for Older workers (TIOWs), Canada-Manitoba Labour Market Agreement for Persons with Disabilities) | Evaluation of the Workforce Development Agreements (2018 to 2023) | 2021 to 2022 Fourth Quarter | Commitment in TB Submission |
Enabling Fund for Official Language Minority Communities | Enabling Fund for Official Language Minority Communities (OLMC) (2017) | Evaluation of Enabling Fund for Official Language Minority Communities 2022 | 2021 to 2022 Fourth Quarter | Required by the FAA |
Wage Earner Protection Program | Evaluation of Wage Earner Protection Program (2014) | Evaluation of Wage Earner Protection Program | 2021 to 2022 Fourth Quarter | Commitment in TB Submission |
Work-Sharing Program | Evaluation of the Work-Sharing Program (2016) | Evaluation of the Work-Sharing Program | 2022 to 2023 | Deputy Minister Request |
Enabling Accessibility Fund | Enabling Accessibility Fund | Evaluation of Enabling Accessibility Fund | 2022 to 2023 | Required by the FAA |
Skilled Trades Awareness and Readiness (STAR) | Not Applicable. New Program. | Skilled Trades Awareness and Readiness (STAR) Evaluation | 2022 to 2023 | Required by the FAA |
Reaching Home: Canada’s Homelessness Strategy | Homelessness Partnering Strategy (HPS) 2018 | Evaluation of the Reaching Home Strategy | 2022 to 2023 | Required by the FAA |
Literacy and Essential Skills | Literacy and Essential Skills 2017 | Evaluation of Literacy and Essential Skills | 022 to 2023 | Required by the FAA |
Future Skills Program | Not Applicable. New Program | Evaluation of Future Skills Program | 2022 to 2023 | Required by the FAA |
Labour Market Development Agreements | Second Cycle Evaluation of the Labour Market Development Agreements (2012 to 2017) | Evaluation of Labour Market Development Agreements | 2022 to 2023 | Commitment in TB Submission |
Canada Education Savings Program | Canada Education Savings Program: Summative Evaluation (2015) | Canada Education Savings Program Impact Evaluation | 2022 to 2023 | Deputy Minister Request in response to an OAG report |
Supports for Student Learning Programs | Pathways to Education 2018 Evaluation | Pathways to Education Canada - Evaluation | 2022 to 2023 | Required by the FAA |
EI Program – Special Benefits | Summative Evaluation of EI Parental Benefits (2005) |
Evaluation of EI Parental and Maternity Benefits | 2022 to 2023 | Commitment in TB Submission |
Early Learning and Childcare | Not Applicable | Evaluation of Early Learning and Childcare | 2022 to 2023 | Commitment in TB Submission |
Youth Employment and Skills Strategy | Summative Evaluation of the Horizontal Youth Employment Strategy, 2020 | Mid-Cycle Evaluation of ESDC Youth Employment and Skills Strategy | 2022 to 2023 | Departmental Needs/Risks |
Labour Market Development Agreements & Opportunities Fund for Persons with Disabilities | Not Applicable. GBA+ Evaluation not previously conducted. | GBA+ Evaluation Study | 2022 to 2023 | Departmental Needs/Risks |
Sectoral Initiatives Program | Evaluation of Sectoral Initiatives Program 2018 | Evaluation of Sectoral Initiatives Program | 2023 to 2024 | Required by the FAA |
Canada Pension Plan Service Improvement Strategy | Not Applicable. Initiative Not Previously Evaluated. | Evaluation of Canada Pension Plan Service Improvement Strategy | 2023 to 2024 | Commitment in TB Submission |
Canada Student Loans Program | Canada Student Loans Program | Canada Student Loans Program: Loan Repayment | 2023 to 2024 | Commitment in TB Submission |
Canada Service Corps | Not Applicable. New Program | Canada Service Corps Evaluation | 2023 to 2024 | Required by the FAA |
Federal Workers Compensation Service | Evaluation of the Federal Workers' Compensation Service (2018) | Federal Workers’ Compensation Service | 2023 to 2024 | Departmental Priority |
Canada Disability Savings Program | Evaluation of Canada Disability Savings Program 2018 | Evaluation of the Canada Disability Savings Program | 2023 to 2024 | Required by the FAA |
Opportunities Fund for Persons with Disabilities | Opportunities Fund for Persons with Disabilities – Phase 2 Evaluation (2021) | Evaluation of the Opportunities Fund for People with Disabilities | 2023 to 2024 | Required by the FAA |
Indigenous Early Learning and Child Care | Not Applicable. New Program. | Horizontal Indigenous Early Learning and Child Care Evaluation | 2023 to 2024 | Required by the FAA |
Canada Student Loans Program – Forgiveness for Doctors and Nurses | Not Applicable. New Initiative. | Evaluation of the Canada Student Loans Program – Forgiveness for Doctors and Nurses | 2023 to 2024 | Deputy Minister Request |
Employment Equity Program | Evaluation of the Employment Equity Program March 2019 | Evaluation of Employment Equity Program | 2023 to 2024 | Required by the FAA |
Old Age Security Service Improvement Strategy | Not Applicable. Strategy Not Previously Evaluated. | Old Age Security Service Improvement Strategy Review | 2023 to 2024 | Commitment in TB Submission |
Occupational Health and Safety (OHS) | Occupational Health and Safety (OHS) (2019) | Occupational Health and Safety (OHS) | 2023 to 2024 | Departmental Priority |
International Labour Affairs | Multilateral Labour Affairs (2018) | Evaluation of International Labour Affairs | 2023 to 2024 | Departmental Needs/Risks |
Labour Standards | Labour Standards (2019) | Labour Standards | 2023 to 2024 | Departmental Priority |
Canada Student Loans Program and Canada Apprentice Loans | Evaluation of the Canada Apprenticeship Loan (2020) | Evaluation of the Canada Apprentice Loan | 2024 to 2025 | Commitment in TB Submission |
Skills and Partnership Fund | Aboriginal Skills and Employment Training Strategy and the Skills and Partnership Fund, 2020 | Evaluation of Skills and Partnership Fund | 2024 to 2025 | Required by the FAA |
Indigenous Skills and Employment Training Program | Aboriginal Skills and Employment Training Strategy and Skills and Partnership Fund, 2020 | Evaluation of Indigenous Skills and Employment Training Program | 2024 to 2025 | Required by the FAA |
Youth Employment and Skills Strategy | Summative Evaluation of the Horizontal Youth Employment Strategy, 2020 | Horizontal Evaluation of Youth Employment and Skills Strategy | 2024 to 2025 | Required by the FAA |
Social Development Partnerships Program | Evaluation of Social Development Partnerships Program 2019 | Evaluation of the Social Development Partnerships Program | 2024 to 2025 | Required by the FAA |
EI Program – Regular Benefits | Not applicable | Evaluation of EI Program – Seasonal Claimant Pilot – Phase 2 | 2024 to 2025 | Departmental Needs/Risks |
EI Program – Special Benefits | Not applicable | EI Extended Parental Benefits | 2024 to 2025 | Departmental Needs/Risks |
Support for Student Learning Programs – Outbound Student Mobility | Not applicable. New Program | Evaluation of Outbound Student Mobility Pilot – International Education Strategy | 2024 to 2025 | Required by FAA |
Foreign Credential Recognition Program | Evaluation of the Foreign Credential Recognition Program (2020) | Evaluation of the Foreign Credential Recognition Program | 2024 to 2025 | Required by the FAA |
Apprenticeship Grants | Evaluation of the Apprenticeship Grants (2019) | Evaluation of the Apprenticeship Grants | 2024 to 2025 | Required by the FAA |
Service Delivery Partnerships | Not applicable. New program. | Evaluation of the Service Delivery Partnerships program | 2024 to 2025 | Departmental Needs/Risks |
EI Program – Special Benefits | Evaluation of Compassionate Care Benefit (2006) | Evaluation of EI Program - Caregiving Benefits | 2024 to 2025 | Departmental Needs/Risks |
Student Work Placement Program | Evaluation of Student Work Placement Program – Phase 1 (2021) | Evaluation of Student Work Placement Program – Phase 2 | 2024 to 2025 | Departmental Needs/Risks |
Old Age Security | Guaranteed Income Supplement Evaluation (2021) | Evaluation of the Guaranteed Income Supplement | 2025 to 2026 | Departmental Needs/Risks |
EI Program – Fishing | 2006 Summative Evaluation of EI Part I | Evaluation of EI Program - Fishing Benefits | 2025 to 2026 | Departmental Priority |
Passport Program | Evaluation of the Passport Program (2020) | Evaluation of the Passport Program | 2025 to 2026 | Departmental Priority |
New Horizons for Seniors Program | Evaluation of the New Horizons for Seniors Program (2020) | Evaluation of the New Horizons for Seniors Program | 2025 to 2026 | Required by the FAA |
Social Innovation and Social Finance Strategy | Not Applicable. New Program | Evaluation of the Social Finance Fund | 2025 to 2026 | Commitment in TB Submission |
Note:
As an evaluation for the Job Bank program was approved in March 2021, it is not listed in the table above. Consultations with program officials, scheduled for Fall 2021 will need to occur before it is included in the coverage table.
The Government of Canada Telephone General Enquiries, In-Person Points of Service, and Government of Canada Internet Presence programs will be included in a proposed evaluation of Providing Service and Information to Canadians in 2029 to 2030.
Page details
- Date modified: