Departmental evaluation plan 2020 and 2021 to 2024 and 2025

Key achievements

In 2019 to 2020, as a contributor to ESDC’s program innovation landscape, the Directorate continued to develop innovative approaches and methodologies in designing and carrying out its evaluations. The Directorate also informed decision making, shared knowledge with program officials and increased its role as a trusted advisor. Several key achievements were realized over the course of the year:

Challenges

Evaluation ranked human resource capacity and availability and quality of data as the two top challenges for the Directorate. To meet human resource capacity challenges, Evaluation is recruiting highly-qualified employees, facilitating the development of employees’ skills and competencies, and improving employee recognition programs.

The Directorate will respond to anticipated demands through ongoing investment in its workforce and capacity, and adoption of innovative tools and methodologies.

The ability to collect, produce and analyze data effectively is fundamental to program evaluation. To mitigate the risks associated with the availability and quality of data and performance measurement information, the Directorate will continue to support efforts of the Head of Performance Measurement and the Chief Data Officer by advising program officials on data quality. The Directorate will also leverage the Labour Market Data Platform to support evaluations of labour market programming.

Another challenge relates to the availability of data necessary to conduct a GBA+ analysis for specific socio-economic groups. While administrative data contains information on a number of identity factors (for example, gender, age, level of education, rural/urban, families with children, persons with disabilities) that help facilitate this level of analysis, it generally does not contain the necessary information to examine identity factors such as membership in racialized and LGTBQ2+ communities. Therefore, the Directorate will continue to support the Department in exploring new and innovative methods for gaining information on program performance to permit broader GBA+ analysis.

Following the outbreak of the COVID-19 pandemic in spring 2020, the Department’s role in supporting Canadians expanded through the delivery of a range of emergency benefits and measures, such as the Canada Emergency Response Benefit, the Canada Emergency Student Benefit, and an investment of $350 million to support the non-profit sector in delivering essential services to those in need. In this context, the Directorate is making several adaptations to evaluation work, such as:

• Mitigating the risk of virus transmission for evaluators, consultants, key informants, and partners by conducting surveys and interviews virtually as appropriate
• Recognizing potential limitations on the quality of virtual interview and survey data, such as lower response rates, and the higher risk of recall bias due to the pandemic crisis and its aftermath

• Reaching out to evaluation networks to identify and adopt innovative approaches in conducting evaluations, including utilizing new communication platforms such as Teams and WebEx
• Leveraging existing lines of evidence that have not been affected by the crisis, including rich integrated administrative data
• Taking into consideration how data collected following the COVID-19 outbreak will have contextual differences compared to data collected in the pre-pandemic era

Notwithstanding adaptations to evaluation work in the COVID-19 context, further changes to evaluation planning will be required in consultation with partners in the coming months, including:

• Re-scoping some ongoing and planned evaluations in light of limitations encountered in some lines of evidence and changes to existing programs in response to the pandemic
• Advancing work for the evaluation of significant new COVID-19 initiatives
• Recalibrating resource allocation and the plan based on new priorities responding to the new environment

Going forward

Over the next five years, the ESDC Evaluation function will continue to seek ways to better support evidence-based decision-making, and fulfill its obligations under the Financial Administration Act and the TBS Policy on Results. There are several priority evaluation areas in the coming fiscal year. Reviews of labour market programming will focus on incremental impacts of participation to the program (using machine learning). Evaluations of learning programs will explore programs’ effect on education savings, as well as access to and affordability of education. Efforts on income security programs will focus on ongoing service improvement strategies.

Building on recent evaluations, the Directorate will place greater emphasis on analyzing program impacts from a GBA+ lens, with a focus on marginalized groups (for example, racialized communities) when information allows. Evaluation Directorate will continue to support the Department’s efforts to modernize the way it collects and uses data to make informed evidence-based decisions and improve programs and services for all Canadians.

In particular, over the next two years, ESDC will be prioritizing the analysis of Labor Market Programs for the unemployed, marginalized youth, Indigenous Peoples, students, and persons with disabilities through a GBA+ lens. This analysis will integrate a diversity lens when information allows.

In addition, plans will be developed for evaluating new ESDC initiatives created in response to COVID-19. Evaluators will place special emphasis on identifying, gathering and analyzing available data on how ESDC’s COVID-19 related initiatives are impacting Indigenous Peoples, racialized communities and other marginalized groups. The Directorate will continue to work horizontally with other government departments to share best practices in terms of planning and conducting evaluations during and after the pandemic.

Purpose of the report

To comply with the TBS Policy on Results, ESDC must ensure that a five-year departmental evaluation plan is developed and released annually.^{Footnote 3} This plan aims to assist the Deputy Head in ensuring that factual, neutral, and timely information on the Department’s programs is available to support evidence-based decision-making, and inform Canadians on results.

Increased flexibilities provided under the Policy have increased the necessity for sound planning. The timing and scope of evaluations are determined in consultation with program officials and Central Agencies on the basis of need, risk, and priorities. The Plan also ensures transparency in the decision-making process within the evaluation function, highlighting program priorities and coverage decisions.

Departmental context

Employment and Social Development Canada’s mandate is to build a stronger and more competitive Canada, to support Canadians in helping them live productive and rewarding lives and to improve Canadians’ quality of life. The Department’s portfolio includes Employment and Social Development, the Labour Program, and Service Canada.

• Employment and Social Development delivers a range of programs and services that assist Canadians throughout their lives, providing seniors with basic income security, supporting unemployed workers, helping students finance their post-secondary education, and assisting parents who are raising young children.
• The Labour Program fosters safe, healthy, fair and inclusive work environments and cooperative workplace relations in the federal jurisdiction.
• Service Canada provides a single point of access for some of the Government’s largest and most well-known programs and services, such as Employment Insurance (EI), Old Age Security (OAS), Canada Pension Plan (CPP), the Social Insurance Number, and Passports.

The Department spends over $130 billion annually^{Footnote 4} on programs and services, with approximately 94% of this amount directly benefiting Canadians through statutory transfer payment programs, including EI, CPP, and OAS. In addition, the Department administers employment benefits and support measures, which includes significant transfers to provinces and territories. The Department’s services are delivered through online service channels, over the telephone, and through a regional network of over 590 in-person service delivery sites.

The most recent mandate letters for ESDC’s four ministers, which were released in December 2019, provide specific direction on the manner in which the Department’s objectives should be advanced. The letters emphasize the importance of evidence-based decision-making that takes into account a GBA+ perspective, which addresses the intersection between gender and other socio-economic factors. The Prime Minister also directs three ministers to lead ESDC in a way that enhances “openness, effectiveness and transparency in government,” and improves the unique and critical relationship between the Government of Canada and Indigenous Peoples.

The COVID-19 global pandemic led the Government of Canada to rapidly develop and adjust a wide range of programs to support Canadians during the crisis. ESDC was heavily implicated in these developments, including by creating entirely new programs, such as the Canada Emergency Response Benefit and the Canada Emergency Student Benefit. The Department expanded numerous existing programs to make them more responsive to the changed circumstances of Canadians, such as relaxing debt repayment rules for clients of the Canada Student Loans Program, investing additional resources to support those experiencing or at risk of homelessness, and waiving the waiting period for EI sickness and expediting the issuance of money for eligible applicants.

The Evaluation Directorate is adapting the way it works to conduct evaluations in a complex and evolving context. This includes evaluators discussing changes in evaluation designs and supporting the Head of Performance Measurement and program officials in developing their performance measurement strategies for new initiatives.

ESDC’s Evaluation function supports the Department’s goals, including in the context of the federal pandemic response, by conducting the neutral collection and analysis of evidence to examine the relevance, effectiveness and efficiency of departmental programs and services. The results of this work are published in evaluation reports to inform Canadians, and to support decision-making by ESDC program officials and senior management. The Department’s evaluation function also supports the broader Government’s objectives to “instill a culture of evaluation, measurement and evidence-based decisions in program and policy design and delivery.”^{Footnote 5}

ESDC programs touch the lives of Canadians across the country, and must be designed to ensure that these programs and services meet the diverse needs of all Canadians. By developing a more advanced capacity to analyze policy, program, and evaluation design and delivery through a GBA+ lens, the Department is working to build a stronger and more inclusive Canada, and ensure that all segments of the population have equitable access to programs and services.

Mandate

The Evaluation function collaborates with program officials across the Department to support evidence-based decision-making and ensures program officials and senior management are involved in the design and execution of evaluations. The Directorate encourages program partners to provide input into the evaluation process through an inclusive approach to governance (Annex A2). This approach helps provide relevant information to program officials in a timely fashion, ensuring that Evaluation continues to be a trusted advisor in the Department.

Evaluation’s involvement with program officials occurs throughout the program and policy life cycle. During the initial stages, evaluations are able to inform the design and development of new programs and policies. During implementation, interim reports and phased evaluations can provide recommendations, supported by available data and information that guide adjustments and program renewal. As a program or policy matures, sunsets, or further evolves to meet needs, evaluations provide lessons learned, which can inform the development of future programs or policies.

Evaluation advises program officials on the development of their management response and action plans addressing evaluation recommendations from several perspectives, including strategic, corporate, and program design and implementation. By monitoring progress on action plans, Evaluation helps to ensure a system of continuous improvement throughout the evaluation cycle.

Supported evidence-based decision making

• Completed 9 evaluation reports, 52 technical studies and 1 assessment to support program officials and senior decision makers.
• Advised on 31 Treasury Board Submissions and 3 Memoranda to Cabinet to ensure that evaluation evidence is considered in the Government’s policy-making process.
• Extended the Labour Market Program Data Platform to conduct net impact analysis for evaluations using administrative data from youth and Indigenous programs.

Shared knowledge through dissemination & engagement

• Disseminated evidence and innovations through more than 15 presentations at domestic and international conferences and workshops.
• Published 9 reports and increased their mobile discoverability by extending online retention to 10 years.
• Revamped the Evaluation webpage and promoted evaluations publicly on social media.

Broadened the trusted advisor role

• Advised the Head of Performance Measurement on performance measures in 19 Performance Information Profiles to improve their quality.
• Supported the EI Commissioners for Employers and Workers by examining labour market issues through evaluations, supplemental studies, and internal analysis.

Supporting evidence-based decision making

The evaluation function influenced and supported sound decision-making at ESDC by:

• Producing and submitting 38 products to the Performance Measurement and Evaluation Committee (PMEC), which is chaired by the Deputy Minister to review and approve performance and evaluation activities. The products included 9 evaluation reports, 4 horizontal evaluations, 1 assessment, 1 technical study, and 8 information items
• Advising on 31 Treasury Board Submissions and 3 Memoranda to Cabinet to inform policies and programs
• Since its launch in fall 2018, the Evaluation Directorate`s Labour Market Program Data Platform has become an essential source of information for the Department. It provides new and valuable insights on the long-term impact of ESDC’s labour market programming, enabling program officials to address opportunities for improvement and expand on key successes. The platform has enabled key insights into how Canadians from various socio-economic groups can be impacted differently by the Department’s policies, programs and initiatives. In 2019 to 2020, Evaluation made a series of improvements to the Data Platform, including:
  • Adding new indicators for measuring ultimate outcomes, and integrating data from other labour market programs targeted to youth, Indigenous Peoples and persons with disabilities
  • Providing program officials access to the Platform to conduct policy analysis that informs the annual EI Monitoring and Assessment Report
  • Increasing the Platform’s usefulness in implementing data-driven techniques and algorithms such as machine learning

Disseminating and engaging

In addition to producing and publishing 9 evaluation reports online in 2019 to 2020, the Evaluation Directorate generated a number of other products, including technical reports, one-page summaries, lessons learned syntheses, conference presentations, and supplemental studies. In the spirit of Open Government, Evaluation Directorate improved public awareness and discoverability of Directorate products and expanded our audience using various dissemination channels. These activities spanned five key areas:

The trusted advisor role

Over the course of 2019 to 2020, the Evaluation Directorate continued to act as a trusted advisor to program officials and senior management by:

• Supporting the Head of Performance Measurement, who leads the ongoing review of the Department’s Performance Information Profiles, by advising program officials on the performance measurement of 19 ESDC programs.
• Supporting the EI Commissioners for Employers and Workers in examining labour market issues by developing eight supplemental studies from the 2018 to 2019 workplan and supporting the work of the Monitoring Report and Advisory Committee.
• Collaborating with Learning Branch on the assessment of the Canada Student Loans Program Skills Boost Measures so that appropriate advice on next steps could be formulated. The assessment examined the impact of the Skills Boost Measures on participants’ decision to return to school and its influence on their studies.
• Obtaining thorough peer reviews for all evaluation results, as well as the evaluation methodologies employed by the Directorate. This approach represents an international best practice with respect to ensuring evaluation rigour.
• Continuing involvement in the Forum of Labour Market Ministers’ Performance Measurement Working Group. This includes providing advice regarding performance measurement strategies and data sharing between the federal, provincial and territorial governments in support of evaluation activities.
• Coordinating the completion of eight supplemental studies from 2018 to 2019 and disseminated findings through multiple channels in cases where findings were used as a line of evidence for a program evaluation. Looking forward, the Evaluation Directorate will lead the completion of three supplemental studies related to post- claim outcomes.

The Directorate’s work on COVID-19 related measures is part of an overall effort by the Department’s Strategic and Service Policy Branch to assist program officials in responding to the crisis. In particular, evaluators are providing advice on departmental initiatives aimed at assisting the non-profit sector in weathering the global health crisis. Evaluators are also supporting the Department’s efforts to maintain integrity measures by providing advice on the effectiveness and efficiency of new systems put in place in response to COVID-19.

The Directorate will continue supporting program and policy areas in their efforts to collect data that is necessary for informing more detailed analysis of program impacts on racialized Canadians and other marginalized populations using the GBA+ lens.

The Evaluation Directorate collaborates with the Employment Insurance Policy Directorate to provide input and data analysis for the preparation of the annual Monitoring Assessment Report. Employment Insurance programs are evaluated more frequently than other statutory programs due to the scale and unique objectives of each EI benefit (for example, regular benefit, seasonal, sickness, maternity or parental, fishing), and EI pilots (targeted pilots for seasonal industries in 13 regions throughout Canada).

Both Directorates work together with members of the Monitoring Report and Advisory Committee to prepare an annual workplan that includes supplemental studies and internal analysis with input and direction from both the EI Commissioner for Employers and the EI Commissioner for Workers. The list of studies and analysis can be found in the annual Monitoring Assessment Report (https://www.canada.ca/en/employment-social-development/programs/ei/ei-list/reports/monitoring2018.html).

Evaluation administers a Post-Evaluation Questionnaire following Performance Measurement and Evaluation Committee approval of reports. The questionnaire provides program officials involved with an evaluation the opportunity to rate and comment on the process, the level of collaboration, and the quality, utility, and timeliness of the evaluation. Feedback gathered through this process enables the Evaluation Directorate to identify strengths and areas for improvement.

Innovative approaches

As a central component of ESDC’s program innovation landscape, Evaluation employs advanced tools and methods to develop an evidence base for policy and program development. During 2019 to 2020, the Directorate pursued several innovative practices to better estimate the impact of programs, disseminate results to inform decisions, and experiment with new approaches. These included:

• Engaging program officials to participate actively in the evaluation process by assisting the development of lines of evidence, and embedding program officials within the Evaluation Directorate to enable knowledge transfer
• Convening two Expert Panels as a line of evidence for an evaluation of the Foreign Credential Recognition Program, which promotes inclusion and helps immigrants and refugees to integrate in the Canadian labour market. The Expert Panels consisted of experts (two academics and one representative from a non-governmental organization) involved in the study of Foreign Credential Recognition issues or in addressing the issues “on the ground,” to achieve balanced utilization of information and expertise about the program
• Collaborating with ESDC’s Innovation Lab, which applies design thinking approaches to improving departmental programs and services, to identify barriers to Guaranteed Income Supplement take-up among eligible Canadians
• Working with several federal departments to contribute to horizontal evaluation reports, such as the evaluation of the Passport Program, which involved collaboration between ESDC, Global Affairs Canada and Immigration, Refugees and Citizenship Canada
• Developing a joint audit/evaluation of the Old Age Security (OAS) Program’s Service Improvement Strategy in cooperation with ESDC’s Internal Audit Branch. This partnership will result in a robust evaluation while reducing resource demands for the program area
• Continuing development of the Labour Market Program Data Platform, which is recognized domestically and internationally as an innovative best practice in terms of labour market performance measurement and net impact evaluation of youth and Indigenous labour market programming
• Enhancing the capacity of ESDC to measure the impact of Apprenticeship Grants, as well as the Temporary Foreign Worker Program, on the Canadian labour market by producing datasets that combine several individual administrative data sources. The datasets have been made available to Statistics Canada, enabling provincial and territorial governments to make more informed decisions with respect to labour market programming and research. This work was undertaken in collaboration with Statistics Canada, ESDC’s Chief Data Office, and program officials
• Developing a more efficient process to streamline management action plan reporting, which was shared by the Treasury Board of Canada Secretariat (TBS) as a model practice

Results

Through 9 evaluations completed in 2019 to 2020, Evaluation measured results across all five core departmental responsibilities. A sample of results extracted from these evaluations include:

Increased flexibility

Under the Policy on Results, ongoing grants and contributions programs that have a five-year average actual expenditure of less than $5 million per year are not subject to the Financial Administration Act requirement for a review of the relevance and effectiveness of the program. This exemption is applicable to two programs (that is, the Sustainable Development Goals Funding Program, and the Canadian Benefit for Parents of Young Victims of Crime).

Consultations

In carrying out evaluation planning, the Evaluation Directorate engaged in strategic discussions with corporate, program, and policy officials to ensure that evidentiary and operational needs were considered.

Between October 2019 and February 2020, the Directorate held consultations with program officials to discuss evaluation needs, scope and timing over the next five years. Several common themes and concerns emerged over the course of the consultation sessions, including: the desire for recent changes to program policy to be captured in forthcoming evaluations, the possibility of grouping together programs or sub-programs under one evaluation as a way of increasing evaluation efficiency, and the difficulties of measuring the capacity of program funding recipients for grants and contributions programs. Other topics discussed related to challenges with program data collection, improving program performance measurement, and effectively applying GBA+ perspectives to evaluations.

The consultation period also allowed for the Directorate and other ESDC units to identify new opportunities for collaboration, including a joint audit/evaluation of Old Age Security that will be undertaken in conjunction with Internal Audit Branch and completed in 2022 to 2023.

The evaluation planning process also included consultations with TBS, during which advice from central agencies was provided on priorities and needs related to the evaluation function. Consultations held in January 2020 identified the need to incorporate evaluation evidence into decision-making documents, including Treasury Board Submissions and Memoranda to Cabinet. This evidence can be a combination of research, technical studies, or evaluation results to inform the development or review of programs.

Reflecting the desire to strengthen evidence-based decision-making, program officials continue to improve the quality of the Performance Information Profiles with advice from Evaluation. As well, the Evaluation Directorate continues to work with departmental researchers to coordinate activities and identify opportunities for collaboration. Evaluation will continue to develop means through which early knowledge from ongoing evaluations can inform and be integrated into program and policy decision-making, and provide input to Treasury Board Submissions and Memoranda to Cabinet.

The Directorate has contacted program officials across the Department to discuss potential changes to ESDC’s evaluation schedule necessitated by the outbreak of the pandemic.

Upcoming engagement with officials will focus on adjustments to the timing, scope and methodologies of various evaluations in order to take into account changes in priorities and needs of program areas.

Risk assessment

The risk assessment exercise examines program-level characteristics that contribute to the overall risk of a program, departmental risks identified in the Corporate Risk Profile, and risks specific to the Evaluation Directorate. By incorporating a multi-level approach to the determination of risk, Evaluation ensures that perspectives of key program partners and internal risks are considered.

Program characteristics also act as a proxy for risk exposure during the evaluation planning exercise. Characteristics considered in early stages of evaluation planning include:

• size of the population affected or targeted by the program
• program materiality (for example, value)
• program complexity
• time elapsed since the previous evaluation
• magnitude of change to the program since the previous evaluation
• knowledge gaps with respect to the program

Risks specific to individual evaluations are further assessed in evaluability assessments to ensure that they are identified prior to the evaluation. Evaluability assessments consider limited risks, such as those associated with program oversight, availability of data, and the application of earlier evaluation findings. Evaluability assessments are an early-stage evaluation phase where the essential components of an evaluation are examined. They highlight timing and scope based on resource limitations, data quality and availability, and core evaluation issues of interest to program partners. The assessments are developed in collaboration with program officials and presented to PMEC for approval.

Key corporate risks for the department are also considered when planning discretionary evaluations, and include a review of the risk-based corporate audit plan. In carrying out this work, the Directorate ensures audit and evaluation schedules are aligned to reduce duplication or burden placed on programs. In situations where audit and evaluation schedules overlap for the same program, Evaluation Directorate may partner with Internal Audit Branch to conduct a joint audit/evaluation as a means of efficiently examining performance and outcomes. In 2020 to 2021, the Directorate will begin work on a joint audit/evaluation of Old Age Security Service Improvement in tandem with departmental auditors. Collaboration ensures an overall assessment of the adequacy and effectiveness of a department's control processes is conducted (audit coverage), and that relevance and cost-effectiveness of a program are also assessed (evaluation coverage). This approach also reduces some administrative burden for the program.

Challenge 1: Human resources capacity

The Directorate is responding through several measures:

• Leading efforts to improve promotional material and recruitment for the evaluation function
• Engaging with staff to support a positive and healthy workplace
• Supporting professional development of staff
  • Canadian Evaluation Society certifications
  • Data Skills Training
  • Conference presentations
• Using flexible staffing options
  • Students/Post-Secondary Recruitment
  • Collective staffing
  • Casuals
  • Consultants

Challenge 2: Quality of ESDC’s data and performance measures to support evaluations

The Directorate is responding through several measures:

• Working with the Head of Performance Measurement, advising responsible program officials and the Performance Measurement and Evaluation Committee on the quality and utility of Performance Information Profiles in support of evaluation activities
• Assessing data and performance indicators in evaluability assessments
• Expanding the Labour Market Data Platform to improve data, analysis, and performance measures for related programs
• Conducting surveys to support metrics

Human resources capacity

The nature of the work undertaken in the Evaluation Directorate requires the continuous recruitment, development and retention of highly skilled employees. The Directorate requires specialized skills to analyze, synthesize, and communicate complex lines of quantitative and qualitative evidence in order to provide the quality advice that the Department relies on to effectively build policy and programs to best serve Canadians.

Evaluation is taking several steps to improve recruitment and employee retention and is leading government-wide efforts to improve promotional material and recruitment. The Directorate supports collective staffing processes across government to ensure there is a continuous intake of new candidates with the right skills through a streamlined hiring process. In addition, the Evaluation Directorate is supporting the establishment of the Emerging Evaluators Network, an informal group of evaluators from across government that discuss issues of importance to the evaluation community.

By aligning the Directorate with the Government of Canada’s Beyond2020 vision, Evaluation is ensuring an agile, inclusive and equipped workforce that is enabled to meet departmental challenges. To ensure that capacity demands are met in the short term, the Directorate uses flexible staffing tools, such as hiring students, casual employees and consultants. In addition, a number of retired public servants were hired on a temporary basis to fill specific capacity gaps.

Past surveys have found that Evaluation employees think they get the training they need. Evaluation is implementing additional talent management programs, including:

• continuing to use Performance Agreements, Learning Plans and Talent Management Plans to support staff development
• ensuring that team members understand the competencies required for their current job and those they may aspire to
• implementing strategies to improve second language skills to support bilingual capacity and advancement of experienced staff

To further promote the professional development of employees, Evaluation supports Canadian Evaluation Society membership for 26 employees, 11 of which were accredited as a “Credential Evaluator” by the Society. Evaluation further assists in employees’ professional development by holding regular internal training sessions, equipping them to meet the challenges of an evolving profession and workforce.

Past surveys indicate that Evaluation Directorate employees get a sense of satisfaction from their work, and feel that they receive meaningful recognition for work well done. In addition to leveraging flexibilities provided by the Policy on Results to address these concerns, Evaluation will continue to focus on developing a healthy workplace for its team members by providing training, support and avenues for open dialogue.

In 2019 to 2020, management engaged with staff to strengthen and improve the workplace by:

• continuing to celebrate and recognize achievements
• managing and prioritizing workflows
• encouraging staff to expand skills by presenting their work to internal and external stakeholders and partners
• inviting observers to senior officials’ meetings
• ensuring analysts are present when their files are discussed

In 2019, the Directorate launched its Health, Wellness and Greening of the Workplace Initiative, which is an employee-led campaign encouraging evaluators and staff to organize and participate in more social activities that promote teamwork and well-being. The initiative also promotes approaches to work that are beneficial for the environment, such as green commuting, more flexible work arrangements, and energy conservation in the workplace.

Evaluation fully supports workplace mental health and has contributed significantly to the development of the Department’s mental health framework. A continued focus on the mental health of employees is central to not only encouraging open discussion about issues that affect individuals on a daily basis, but also ensuring that management is aware of emerging issues and is able to support employees in an effective manner.

In response to the pandemic and associated office shutdowns, the Evaluation Directorate is leveraging innovative tools to ensure its staff is able to adapt to a rapidly changing environment. An online virtual learning series is enabling continuous engagement and collaboration with team members. Several members of the Directorate also put their name forward to assist with essential activities in the Department.

Quality of data and performance measurement to support evaluations

Evaluations undertaken by the Directorate are highly dependent on the availability of quality data. Performance measurement information used is used by evaluators to determine relevance and value for money, and occasionally to assess how a program is being implemented.

In collaboration with the Head of Performance Measurement, and consistent with the Policy on Results, Evaluation advises PMEC and program officials on the availability, quality, validity, and reliability of indicators in the Departmental Results Framework. This is also done for each program Performance Information Profile, including with respect to their usefulness for supporting evaluations. Program officials across the Department develop, update and implement Performance Information Profiles for all existing and new programs in the supporting evaluations. They also update and implement Performance Information Profiles for all existing and new programs in the Department’s program inventory.

Evaluation has significantly improved data and analysis for labour market programs, enhancing the attribution of outcomes to program activities. The Labor Market Program Data Platform supported numerous evaluation studies related to the Labour Market Development Agreements. These studies addressed program and policy questions of importance to ESDC and the provinces and territories that are responsible for delivering these interventions under the Agreements. In 2019 to 2020, Evaluation leveraged the Data Platform, expanding net incremental analysis to youth and Indigenous labour market programming.

Evaluability assessments and evaluation strategies include a review of a program’s performance information. Most recent evaluations have provided evidence of program activities’ contribution to desired outcomes, enabled through access to rich administrative data. This is an improvement over past years when ESDC evaluations were often limited to documenting inputs and outputs. Evaluation continues its collaboration with partners to support the continued development of data and the maturity of the performance measurement regime. While the pandemic may render the collection of new data to support evaluations more challenging, the Directorate will pursue mitigation steps in collaboration with program officials.

Another challenge relates to the availability of data necessary to conduct a GBA+ analysis for specific socio-economic groups. While administrative data contains information on a number of various identity factors (for example, gender, age, level of education, rural/urban, families with children, and persons with disabilities) that help facilitate this level of analysis, it generally does not contain the necessary information to examine identity factors such as racialized and LGTBQ2+ communities. Therefore, the Directorate will continue to support the Department in exploring new and innovative methods for gaining information on program performance to permit broader GBA+ analysis.

Performance measurement and evaluation committee

Composed of partners from senior ranks of ESDC, including the Head of Evaluation.

Approves the departmental evaluation plan

Reviews evaluation reports, including management responses

Approves Evaluability Assessments Discusses follow-up on action plans

Evaluation advisory committee

Composed of Stakeholders from across ESDC and/or other departments and agencies such as TBS Program Sector and Finance Canada (Director General level and below).

Identifies options for evaluation scope Discusses preliminary findings

Provides input for final report

Evaluation working group

Composed of partners from across ESDC (Director level and below).

Focuses on technical aspects Performs data collection and analysis Provides input for preliminary report

Results governance: roles & responsibilities

The Chief Results and Delivery Officer is the key contact for the Results and Delivery Unit of the Privy Council Office, and identifies top departmental priorities in support of the results agenda. This role is the primary change agent for the Department and ensures that a delivery plan, measurement strategy and reporting structures are in place to support the delivery of these priorities. The Head of Performance Measurement supports the Chief Results and Delivery Officer by assisting with performance measurement and reporting by establishing, implementing and maintaining the Departmental Results Framework and overseeing Performance Information Profiles.

The key responsibilities of the Head of Evaluation are to assess departmental evaluation needs, undertake departmental evaluation planning, directing and reporting, support the use of performance information, and advise the Deputy Head on evaluation findings and issues.

The Chief Results and Delivery Officer is also supported by the Director General, Strategic Horizontal Policy, with an emphasis on measuring and reporting on government activities and their impact on Canadians. This Director General leads on elements of horizontal priorities for the government and the tracking of mandate letter commitments, including their progress, challenges and barriers to completion.

ESDC’s Chief Data Officer leads the implementation of the Department’s enterprise Data Strategy, including the development of data foundations, such as data governance and stewardship. These foundations are essential to ensuring that data users have the knowledge and support needed to identify, locate, and fully leverage data in their daily work, and in developing indicators to support performance measurement, including reports of the Chief Results and Delivery Officer. All of these positions are located in the Strategic and Service Policy Branch of ESDC.