Language selection

Government of Canada / Gouvernement du Canada

Search

Departmental Evaluation Plan 2018-19 to 2022-23

Alternate formats

Departmental Evaluation Plan 2018-19 to 2022-23 [PDF - 1.0 MB]

On this page

List of Acronyms

EI
Employment Insurance
ESDC
Employment and Social Development Canada
LMDA
Labour Market Development Agreement
PMEC
Performance Measurement and Evaluation Committee

Deputy Head Departmental Evaluation Plan Confirmation Note

I approve the Departmental Evaluation Plan of Employment and Social Development Canada for the fiscal years 2018 to 2019 to fiscal year 2022 to 2023, which I submit to the Treasury Board of Canada Secretariat as required by the Policy on Results.

I confirm that this five-year rolling Departmental Evaluation Plan:

I will ensure that this plan is updated annually, and I will provide information about its implementation to the Treasury Board of Canada Secretariat as required.

Louise Levonian
Deputy Minister of Employment and Social Development

Executive Summary

The Department of Employment and Social Development Canada (ESDC) delivers over $120 billionFootnote 1 annually in programs and services such as Employment Insurance, Canada Pension Plan, Old Age Security, Canada Child Benefit, and other programs that profoundly impact the lives of Canadians. The Evaluation Directorate collaborates widely with ESDC decision makers, providing evidence to inform program renewal and policy and program development.

Collaborative engagement with central agencies and departmental stakeholders on the timing and scope of evaluations was the essence of this responsive five-year evaluation plan. Beyond ensuring that evaluations are conducted on all programs that fall under the Financial Administration Act and the Policy on Results, the Directorate aims to ensure that evaluations are responsive and timely, and meet the needs of decision makers.

Innovative approaches to conducting evaluations are increasing the efficiency and impact of evaluation work. For example, the development of a labour market program data platform in collaboration with ESDC’s Skills and Employment Branch will enable analysis of medium-term program impacts and trends. The Directorate increasingly employs different types of evaluations, and is exploring ways to increase the use of thematic and horizontal evaluations.

Key achievements

During the last fiscal year, the Evaluation Directorate enhanced its ability to support decision making, increased its dissemination and engagement efforts, and provided trusted, neutral advice on performance measurement and program improvements.

1. Influenced decision making

In fiscal year 2017 to 2018, the Evaluation Directorate generated evaluation reports and the supporting technical reports and preliminary findings which provided decision makers with timely, evidence-based information to inform policy and program improvements. In summary, the Directorate:

2. Shared knowledge and increased engagement

In fiscal year 2017 to 2018, the Evaluation Directorate advanced efforts to disseminate knowledge and share promising practices among internal and external stakeholders. Specific initiatives included:

3. Broadened the trusted advisor role

The Evaluation Directorate is increasingly regarded as a source of trusted and neutral advice to ESDC’s departmental portfolio. During fiscal year 2017 to 2018, the Evaluation Directorate:

4. Managing challenges

The Directorate encounters two key challenges when conducting its evaluation work: the analytic capacity to address the workload requirements, and the availability and quality of data.

To mitigate challenges and in support of human resource management, the Directorate has developed a Strategic Human Resource Plan that identifies current needs, priorities, and career development opportunities. Initiatives include: actively recruiting new talent from the academic community, professional organizations, and other federal agencies; contracting retired evaluators; and providing relevant training opportunities for employees.

To improve the availability and quality of data, the Directorate has identified specific priorities in the areas of grants and contributions and labour market program data. The Department’s Chief Data Officer ensures that data is efficiently integrated and securely stored, and that analytics programs are in place to maximize its business value.

5. Increasing emphasis on outcomes and impacts

The Evaluation Directorate has made significant effort to evolve from generating output-focused evaluation reports to ones that are more outcome-oriented. Previously, in the absence of solid data, evaluations tended to focus on analyzing output, performance measures and qualitative outcomes. Recently, with the investment in data and performance measurement, evaluation work has focused on program impacts and the provision of more timely, policy-relevant evaluations. The Directorate also increasingly employs different types of evaluations; considers and tests innovative approaches; and has become more efficient in producing evaluation work.

Introduction

Purpose of the report

The primary purpose of the Departmental Evaluation Plan (the Plan) is to assist the Deputy Head in ensuring that factual, timely and neutral information on the department’s programs, including grants and contributions and transfer payments, is available to support evidence-based decision making and inform results for Canadians. To foster the integration and streamlining of departmental plans, this report uses standardized language and table formats found in other management plans. The Plan also:

Departmental context

The mission of Employment and Social Development Canada (ESDC) is to build a stronger and more competitive Canada, to support Canadians in making choices that help them live productive and rewarding lives, and to improve their quality of life. This large and complex Department is responsible for developing, managing, and delivering social, labour market, and economic programs and services that support Canadians via three business lines: Employment and Social Development, the Labour Program, and Service Canada.

The Department delivered over $120 billionFootnote 2 in benefits each year directly to individuals and organizations through major programs and services such as Employment Insurance, Old Age Security, Canada Pension Plan, and the Canada Student Loans Program. In addition, ESDC provides $1.9 billionFootnote 3in grants and contributions funding each year to other orders of government, such as those made to provincial and territorial governments through Workforce Development and Labour Market Development Agreements, as well as to charitable and community organizations, private sector companies, and educators.

Employment and Social Development programs help Canadians at all stages in their lives. The broad range of programming includes assisting parents who are raising young children, helping students finance their post-secondary education, supporting unemployed workers, and providing seniors with basic income security. The programs also provide targeted support to vulnerable populations such as seniors, people with disabilities, Indigenous people, homeless people, and recent immigrants.

The Labour Program is responsible for overseeing federal labour regulatory responsibilities, including facilitating compliance with occupational health and safety, labour standards, and employment equity legislation. The Labour Program assists trade unions and employers in the negotiation and renewal of collective agreements in federally regulated workplaces. It also represents Canada in international labour organizations, and negotiates and implements labour provisions in the context of trade liberalization initiatives.

Through Service Canada, Canadians can access ESDC programs as well as other Government of Canada programs and services at 590 points of service across the country: 320 Service Canada Centres, 238 Outreach Sites, and 32 Stand-alone Passport Offices. In addition to in-person services, Canadians can access services online and by phone at 1–800–O–Canada and specialized call centres. ESDC is currently transforming its service delivery approach to enhance the service experience for clients, employers, communities and organizations. The Transformation Plan outlines short-, medium-, and long-term opportunities and key goals of its initial phase include increased access, experience, timeliness and quality in service delivery.

ESDC Evaluation Directorate Accomplishments

The Evaluation Directorate systematically gathers and neutrally analyzes evidence to assess whether, why, and how programs work. This evidence informs decision making, improvements, innovations, and accountability. It also supports the Government’s Results Agenda by building the knowledge base on performance measurement, data, and research in partnership with program officials and researchers.

The Evaluation Directorate delivers evaluations that focus on key knowledge gaps and the relevance, effectiveness, and efficiency of programs for Canadians, while making recommendations to improve performance information. All evaluations are scoped in partnership with program areas and key stakeholders to optimize their utility and practical application.

During the last fiscal year, the Directorate has furthered efforts to influence decision making, increased its dissemination and engagement efforts, and provided neutral and trusted advice on performance measurement and program development.

Influenced decision making

Evaluation reports and the supporting technical studies, preliminary findings, and methodological strategies all provide decision makers with timely, evidence-based information used to improve program design and performance, and strengthen accountability. During fiscal year 2017 to 2018, the Evaluation Directorate:

Shared knowledge and increased engagement

During fiscal year 2017 to 2018, the Evaluation Directorate made a concerted effort to disseminate knowledge and share promising practices among internal and external stakeholders through learning series, various scientific and public policy conferences, and published articles. The Directorate continues to modernize the way it communicates results to a wider audience, using social media platforms such as Twitter and LinkedIn to advance the dissemination and use of evaluation findings. The Directorate staff also:

Broadened the trusted advisor role for Evaluation

The Evaluation Directorate is becoming a source of trusted advice to ESDC’s departmental portfolio, and is engaged early in the policy and program design phase to ensure the right program information and data is collected to enable relevant evaluations. The Directorate also supports ongoing policy dialogue by providing strategic guidance on how performance information can strengthen results reporting and drive program improvements. During fiscal year 2017 to 2018, the Evaluation Directorate:

Moreover, the Evaluation Directorate continues to coordinate, in collaboration with the Skills and Employment Branch, the Monitoring Report Advisory Committee. This Committee - whose members include the Canada Employment Insurance Commissioners for Employers and for Workers as well as an observer from the Canadian Research Data Centre Network - develops and advances the analytical agenda that supports the annual EI Monitoring and Assessment Report. The research agenda is approved by the Canada Employment Insurance Commission chaired by the Deputy Minister of Employment and Social Development Canada. The supplemental studies proposed by and prepared for the committee are not connected to formal evaluations of the EI program. The Evaluation Directorate will undertake 4 supplemental studies on the EI Program in fiscal year 2018 to 2019.

By increasing efforts to influence decision making, disseminate knowledge and broaden its trusted advisor role, the Directorate will continue to enhance program integrity, help programs achieve better performance and results, and improve service delivery to Canadians.

Results

In 2016, the Government of Canada began implementing the Policy on Results to foster a culture of measurement and evaluation needed to strengthen the provision of evidence-based results to Canadians. The Policy on Results has enabled ESDC to consider and test innovative approaches, and increase the efficiency of evaluation work.

Complementing the Policy on Results, the Government of Canada’s Results Agenda focuses on identifying, delivering, and communicating to Canadians what government has achieved on their behalf and reporting on the resources used to do so.

To ensure that governmental and departmental results priorities are achieved, ESDC is transforming its business activities and departmental structures. For example, the appointment of a Chief Results and Delivery Officer provides central agencies and other departments with a key contact on departmental results. The Chief Results and Delivery Officer is supported by the Head of Evaluation, the Head of Performance Measurement, the Chief Data Officer, and the Director General Supporting Departmental Charters. These officials collaborate and coordinate to ensure that evidence-based results influence decision making.

The Evaluation Directorate, guided by the Head of Evaluation reporting to the Deputy Minister, supports the Results Agenda by providing timely, focused evidence needed to improve programs and strengthen accountability. Below are some of the results highlights of published evaluations undertaken in the past five years:

More Canadians are utilizing educational support programsFootnote 5

Canada Student grants and bursaries increased from $421 million in fiscal year 2008 to 2009 to $715 million in fiscal year 2013 to 2014. In addition, 47% of Canadians opened a Registered Education Saving Plan (RESP) in 2013 as compared to 9.8% in 1998. Although gains have been made, the evaluation recommended that programs explore further outreach options and take-up incentives to reach families with the greatest need for assistance.

Greater stability for homeless CanadiansFootnote 6

In fiscal year 2014 to 2015, the Homelessness Partnering Strategy program emphasized the implementation of a Housing First approach that focuses on stable housing as a priority for Canadians experiencing chronic or episodic homelessness. By fiscal year 2015 to 2016 the program surpassed its target of placing 3,000 people in longer-term stable housing. Over 6,900 people were placed in stable housing and approximately three-quarters of clients were placed within 90 days. With these early indications of positive outcomes, the program is anticipated to contribute to the ultimate goal of preventing and reducing homelessness in Canada.

Meeting the financial needs of retired CanadiansFootnote 7

The Canada Pension Plan reaches 100% of eligible Canadians who are over 70 years old. This take-up rate provides a strong financial base for retired Canadians, representing about 25% of their retirement income. The evaluation revealed that Canada’s retirement income system compared well with other countries with similar living standards in reducing low income among seniors.

Providing financial security to persons with disabilitiesFootnote 8

From its introduction in 2008 to 2010, the number of participants in the Canada Disability Savings Program has grown steadily, reaching 15% of people with disabilities. However, the program still remains beyond the reach of some segments of the population. To increase take-up of the program, the evaluation recommended improvement to the communications approach and outreach to the target population.

Participating in youth programs increases earningsFootnote 9

The evaluation revealed that in the six years following participation in a Youth Employment Strategy (YES) program, participants in the Career Focus stream earned a cumulative $40,500 more than those who did not participate in the program. In the same time period, 80% of youth participating in the Skills Link stream program found employment and were able to increase their annual earnings from $3,651 to $14,009. The evaluation recommended that ESDC and YES partner departments/agencies work to improve the distribution of relevant labour market and career information to youth in all three YES streams.

Consistent service across Canada for Employment Insurance recipientsFootnote 10

Automated claims processing has contributed to greater consistency in the speed of payments to employment insurance recipients. Service Canada will continue to enhance and modernize the service delivery model and supporting information technology with the ultimate goal of offering the same standard of service to Canadians across the country.

Labour disputes are settled without work stoppagesFootnote 11

During the five-year evaluation time frame, the Workplace Dispute Prevention and Resolution Program contributed to settling collective agreement disputes on 95% of disputes without a work stoppage.

Job Bank is top of mind for many employersFootnote 12

Almost 60% of employers report that Job Bank is the primary place where they post job vacancies. At the same time, 70% of job seekers report that they are satisfied with job postings on Job Bank. The evaluation report recommended that Job Bank could improve its services by providing information reflecting local realities for better-informed decision making.

Evaluation planning and coverage

Table 1 shows the status of evaluations planned for fiscal year 2017 to 2018. All ESDC evaluations that are mandatory under the Financial Administration Act and the Policy on Results were completed by the end of fiscal year 2017 to 2018 with the exception of the Pathways to Education Canada evaluation (however it is only required in fiscal year 2018 to 19). The Directorate has completed and received Deputy Minister Approval for 13 evaluation reports, and two reports are in the final approval stage following Performance Measurement and Evaluation Committee (PMEC) endorsement. In addition, six non-mandatory evaluations have been carried over to the following fiscal year for completion by the fall of 2018.

Table 1: Status of evaluations planned for fiscal year 2017 to 2018

Evaluation Report Status
1. International Trade and Labour and Occupational Health and Safety Evaluation CompletedFootnote 13: April 2017
2. Enabling Fund for Official Language Minority Communities – Evaluation Completed: June 2017
3. Evaluation of the Labour Market Development Agreements Synthesis Report Completed: June 2017
3.1 Evaluation of the Canada – New Brunswick Labour Market Development Agreement Synthesis Report Completed: December 2017
3.2 Evaluation of the Canada – Nunavut Labour Market Development Agreement Synthesis Report Completed: December 2017
3.3 Evaluation of the Canada – Northwest Territories Labour Market Development Agreement Synthesis Report Completed: December 2017
3.4 Evaluation of the Canada - Ontario Labour Market Development Agreement Synthesis Report Completed: February 2018
3.5 Evaluation of the Canada – Prince Edward Island Labour Market Development Agreement Synthesis Report Completed: February 2018
3.6 Evaluation of the Canada – British Columbia Labour Market Development Agreement Synthesis Report Completed: February 2018
3.7 Evaluation of the Canada - Alberta Labour Market Development Agreement Synthesis Report Completed: February 2018
3.8 Evaluation of the Canada - Manitoba Labour Market Development Agreement Synthesis Report Completed: March 2018
3.9 Evaluation of the Canada - Saskatchewan Labour Market Development Agreement Synthesis Report Completed: March 2018
3.10 Evaluation of the Canada – Newfoundland and Labrador Labour Market Development Agreement Synthesis Report Completed: March 2018
3.11 Evaluation of the Canada – Nova Scotia Labour Market Development Agreement Synthesis Report Completed: March 2018
3.12 Evaluation of the Canada - Yukon Labour Market Development Agreement Completed: March 2018
4. Learning and Labour Market Information as disseminated by ESDC using a web-based consolidated approach – Evaluation Completed: July 2017
5. Literacy and Essential Skills Completed: October 2017
6. Federal Income Support for Parents of Murdered or Missing Children - Evaluation Completed: December 2017
7. Enabling Accessibility Fund - Summative Evaluation Completed: December 2017
8. National Action Plan to Combat Human Trafficking Completed: December 2017
9. Evaluation of the Old Age Security Program: Phase 1 Completed: February 2018
10. Multilateral Labour Affairs - Evaluation Completed: April 2018
11. Bilateral and Regional Labour Affairs – Evaluation Completed: April 2018
12. Homelessness Partnering Strategy - Summative Evaluation Completed: April 2018
13. Sectoral Initiatives Program PMEC Endorsed :Footnote 14 April 2018
14. Building Aboriginal Support for West Coast energy projects initiative–SPF Evaluation (technical report for Horizontal Initiative led by NR-CAN) PMEC Endorsed: January 2018
15. Opportunities Fund for Persons with Disabilities
Phase 1 - Summative Evaluation PMEC Endorsed: June 2018
16. Employment Equity Programs - Evaluation Carried Over:Footnote 15 November 2018
17. Connecting Canadians with Available Jobs – Evaluation Carried Over: June 2018
18. Working While on Claim Pilot Projects - Evaluation Carried Over: June 2018
19. Pathways to Education Canada - Evaluation Carried Over: September 2018
20. Labour Standards Carried Over: September 2018
21. Occupational Health and Safety Carried Over: November 2018
22. Providing Services and Information to Canadians Interim Report Carried Over: September 2018

Given the surge in reports required for completion in fiscal year 2017 to 2018 under the old Policy on Evaluation and planned for in 2013 (five years ago), the Evaluation Directorate has prioritized mandatory evaluations (under the Financial Administration Act and Policy on Results requirements), as well as work that informs pressing policy and program decision making. Some evaluation work has been accelerated to inform policy development, including the Youth Employment Strategy. Other planned non-mandatory and lower risk evaluations are all well advanced and are planned for completion for fall 2018.

Evaluation reports include recommendations to address areas for improvement; from these, program officials develop Management Responses. These responses are integrated into practices, policies, and initiatives leading to the improvement of program design, implementation, and/or delivery – all documented in Management Action Plans, which are tracked by the Evaluation Directorate. Management Action Plans are reviewed annually until they are completed, and the extent of implementation is assessed and reported annually to the Performance Measurement and Evaluation Committee. The majority of Management Action Plan commitments outlined in past evaluations are well underway. In fiscal year 2016 to 2017, a total of 133 actions from 15 evaluation reports were monitored by the Evaluation Directorate. Of these, the majority have been completed.

The Departmental Evaluation Plan is shaped and influenced by a variety of considerations. Beyond legislative requirements and departmental considerations, the Evaluation Directorate also considers governmental priorities and initiatives such as the Results Agenda and its emphasis on outcome-based evidence and gender-based analysis when planning and executing its evaluations. Further, the Directorate also supports the Government’s Open Data and Open Government Initiative by proactively disseminating evaluation results via conferences, workshops and improving online access. The Directorate is also developing a labour market data platform to conduct approved research through a request for proposals process.

Risk assessment in evaluation planning

The Financial Administration Act requires departments to evaluate ongoing grants and contributions programs with a five-year annual allocation of $5 million or greater on a five-year cycle. Other mandatory requirements such as commitments in Treasury Board Submissions or centrally-led evaluations requested by Treasury Board Secretariat are captured under the Policy on Results. For non-mandatory evaluations, risk and other factors (e.g. data availability, program renewal or sunset and timing) are also considered to identify high-priority evaluations and the most effective use of resources.

After first ensuring compliance with the Financial Administration Act and the Policy on Results, the Directorate assesses programs’ risk exposure during the planning exercise by reviewing program characteristics including:

The risk assessments allow the Directorate to make informed decisions and promote meaningful discussion at management tables in planning for evaluations such as horizontal and thematic projects. Evaluations of complex or higher-risk programs can be allocated additional resources, phased to accommodate program stages of development, or conducted prior to evaluations of lower-risk programs where appropriate.

Programs benefit from evaluation risk assessments by the timely planning and development of interim reports or analysis for Memoranda to Cabinet and Treasury Board Submissions based on risks and opportunities. For example, programs that support departmental priorities or those undergoing renewals are prioritized. This approach ensures that the Directorate is responsive and able to adjust to the changing environment, and the needs of programs requiring timely evidence for decision making.

Performance information and evaluations

In July 2016, the Policy on Results replaced the Policy on Evaluation, the Policy on Management Resources and Results Structures, and the Policy on Reporting of Federal Institutions and Corporate Interests to Treasury Board of Canada Secretariat. To further emphasize and promote transparent, results-oriented and evidence-based approaches to federal policy development and program delivery, the new Policy introduced a number of new requirements. One such requirement was the development of a Departmental Results Framework to replace the strategic outcomes and programs of the current Program Alignment Architecture as well as the creation of Performance Information Profiles for each program listed in the Program Inventory.

Sound performance information can be used to guide and influence evaluation planning in several ways. Creating Profiles over the past year to meet Treasury Board Secretariat requirements was particularly significant, and engaged Evaluation as a source of trusted advice to programs throughout the process. In the initial phase, completed in November 2017, the Directorate advised on Profile design for every ESDC program to include all the required elements, such as a logic model, indicators, and data sources. Subsequently, programs will focus on improving the quality of Profiles so that they can become useful tools for monitoring performance. For example, ESDC’s Skills and Employment Branch is building its performance capacity by creating a dedicated results unit for its programs and initiatives that promote skills development, labour market participation, inclusiveness, and labour market efficiency.

The Directorate periodically reviews and advises programs improving their Profile information and program performance indicators as part of evaluability assessmentsFootnote 16. The availability, quality, validity, and reliability of the indicators and information in the Profile, including their utility for evaluation, are also factored into the assessment of the program’s state of readiness to be evaluated, and the scope of future program evaluations. Going forward, the Directorate will continue to work with program areas to advice on the development of performance information.

Increased flexibility

The department is required to evaluate specific programs based on two authorities:

The Policy, in alignment with the Financial Administration Act, requires that every five years, departments evaluate ongoing grants and contributions programs with five-year average annual allocations of $5 million or greater. It requires only periodic evaluations for programs with five-year average actual expenditures of less than $5 million per year. ESDC has only two grants and contributions programs with a five-year annual average actual expenditures of $5 million or less:

The Department aims to maintain a five-year evaluation cycle for all grants and contributions programs including the two smaller ones. Programs, for which evaluations are governed by the Policy on Results, are evaluated every 5 years by default, while alternative timing may be negotiable with a solid rationale. The timing for some programs is sometimes specifically set in Treasury Board Submissions.

The following discretionary evaluations are not required and therefore are not included in this five-year evaluation plan:

The Evaluation Directorate will continue to engage with program officials and relevant stakeholders to ensure its coverage meets the evolving needs and priorities of the Department.

Consultations

In the fall of 2017, the Evaluation Directorate began consultations with officials regarding the timing and scope of evaluations outlined in the Plan, and on the importance of collecting performance information for the purpose of evaluation. These are the key findings:

The Directorate also coordinates with ESDC’s Internal Audit Services Branch to ensure there is no duplication or undue burden for programs, and to explore joint projects. As prescribed by the Policy on Results, the Directorate informs the Head of Communications on survey research included in evaluations through evaluation planning discussions at the Performance Measurement and Evaluation Committee.

The Directorate also engaged in a series of consultations with Treasury Board Secretariat to ensure that the current evaluation schedule reflects the information needs and considerations of central agencies. The Directorate was advised to:

Central agencies also recognized that ESDC has an ambitious evaluation agenda over the next two years, which will require significant resources.

In accordance with central agency expectations, the Evaluation Directorate has demonstrated collaboration with program areas to ensure that relevant evaluation findings and results are shared in time to inform policy and program decision making. The Directorate continues to work with program officials to ensure that evaluations are timed and scoped to effectively inform policy and program development. In partnership with program areas, Evaluation will continue to balance summative evaluations highlighting program efficacy and lessons learned with reports focussing on more recent program information.

The Directorate is also collaborating with Immigration, Refugees and Citizenship Canada to examine the Passport program. The Directorate is also using a thematic approach to examine ESDC’s apprenticeship programs. To increase the use of horizontal and thematic evaluations going forward, the Directorate is developing a labour market program data platform to enable evaluation of all labour market programs, including youth and Aboriginal programming.

Evaluation will continue collaboration with departmental stakeholders to systematically provide evaluation findings for inclusion in the analysis of program-led Memoranda to Cabinet and Treasury Board Submissions. Lastly, central agencies have noted interest in examining the effectiveness of ESDC’s service delivery activities. The Evaluation Directorate agrees that this is an area to strengthen moving forward, and is currently undertaking an evaluation to examine the effectiveness and efficiency of Service Canada’s provision of services and information via three main service delivery channels (in-person, telephone, and internet). The Directorate will continue to further explore opportunities to examine service delivery activities.

In the past two years, the Directorate has systematically collected information on client satisfaction with its evaluation function following the completion of all evaluation projects. A total of 90 per cent of respondents surveyed indicated that reports over the last fiscal year were delivered in a timely manner, contained relevant findings, and included accurate data and information. This compares to only 75 per cent in the previous year. Respondents indicated that planned use of evaluation findings has grown from 22% in fiscal year 2016 to 2017 to 77% in fiscal year 2017 to 2018.

As the Departmental Evaluation Plan is an annual report, the Directorate will continue to engage program officials and other key stakeholders in ongoing discussions and consultations. This ensures that planned evaluations and their findings continue to meet the needs of programs, the Department, and the Government of Canada Results Agenda.

Challenges and opportunities

The Directorate encounters two key challenges when conducting its evaluation work: the human resources and analytic capacity to address the workload requirements, and the availability and quality of data.

Human resources

As evaluation demands continue to increase and the role of evaluators expands, the Directorate acquired four new positions in fiscal year 2017 to 2018 and increased the number of employees. However, more employees with strong analytical and strategic competencies are needed to meet the increasing demands

Given the complexity of the work involved, as well as the significant government-wide demand for these skills, the Directorate is facing challenges in the recruitment and retention of high-calibre talent, particularly mid-level evaluators.

In the recent past, the Evaluation Directorate has in effect been acting as a feeder of analysts and evaluators who have gone on to contribute to the public service in other ESDC branches or other departments. To mitigate some of these challenges, the Directorate has developed a Strategic Human Resource Plan that identifies current needs, priorities, and career development opportunities. These include promoting top-performing junior evaluators and providing relevant training, such as a two-day workshop on the Essentials in Evaluation, and other learning activities for employees to expand their current skills and acquire new knowledge in the evaluation field.

The Directorate is also addressing human resource capacity needs by recruiting talent from the academic community, professional organizations, other federal agencies and retired evaluators. Related activities include participating in post-secondary recruitment campaigns and posting career opportunities on internal web platforms. The Evaluation Directorate, in collaboration with the Heads of Evaluation, is promoting evaluation as an attractive career choice by producing a series of posters profiling Government of Canada evaluators to highlight and promote employment opportunities.

Resource allocation table

The Evaluation Directorate currently maintains 71 positions, 65 of which are staffed across three divisions overseen by the Office of the Director General: Statutory and Grants and Contributions Evaluation, Partnership Evaluation, and Strategic Planning and Methodology.

To alleviate short-term capacity pressures and workload demands, the Directorate uses flexible staffing mechanisms to engage external experts, consultants, and temporary staff (as reflected in the increase of full-time equivalents from 52 in fiscal year 2016 to 2017 to 65 by the end of fiscal year 2017 to 2018). The Directorate is also investing in the professional development of junior analysts to enable medium-term succession planning.

The Departmental Evaluation Plan is based on its budget of $6.52M as summarized in

Table 2. The Directorate meets its commitment to seek cost-effective ways of conducting evaluations, while ensuring the relevance and impact of its evaluations. Going forward, additional pressures for horizontal and thematic evaluations; and further examining the effectiveness of ESDC’s service delivery will impact budget considerations.

Table 2: Summary of evaluation spending and number of full time equivalents

  Actual spending Forecast
2011 to 2012 2012 to 2013 2013 to 2014 2014 to 2015 2015 to 2016 2016 to 2017 2017 to 2018 2018 to 2019
Salaries $5.06M $5.17M $4.72M $5.51M $5.23M $4.58M $5.32M $5.32M
O & M $3.81M $2.87M $2.55M $2.01M $1.02M $998,975 $1.13M $1.20M
Total $8.87M $8.04M $7.27M $7.52M $6.25M $5.57M $6.44M $6.52M
FTEs 69 61 56 62 57 52 65 65

Notes:

Actual salary spending includes all employees except students; forecast spending includes further human resources staffing actions planned for fiscal year 2018 to 2019. Operations and Maintenance expenses include students, training, travel, printing, utilities, materials and supplies, special services, construction/acquisition machinery and equipment, other business services, and tenant services.

Data availability and quality

The availability and quality of data remains a challenge that evaluators are collaborating with program areas and other stakeholders to overcome. The Directorate has identified two data priorities for the coming years: better quality information for grants and contributions programs, including having data considerations taken into account at the design stage of programs; and the development of integrated labour market program data. The Evaluation Directorate is also working with partners in the Department to streamline access to program data for efficient and timely evaluation.

An area of successful data development is in extending net impact analysis to all labour market program data. The Directorate is currently developing and implementing a rich, high-quality labour market program data platform that will allow evaluation of all labour market program interventions. This initiative is designed to assist decision making in the context of the Government of Canada Policy on Results, supporting evidence-based evaluation analysis of the major labour market programs.

This Data Platform of administrative program information enables analysis of long-term program impacts and trends over time. Ultimately, the Data Platform could enable innovative analysis on what programs work best for which ESDC clients. It can also provide the foundation for horizontal evaluations that could include impact analysis across multiple programs. As the Data Platform develops, it will enable horizontal analysis on net impacts and address broad evaluation questions.

These efforts also respond to the Government of Canada Open Government Agenda, which aims to strengthen openness, transparency, and accountability, as well as to drive innovation.

ESDC’s Chief Data Officer is responsible for working with stakeholders across ESDC to implement a department-wide data strategy. The ESDC Data Strategy is establishing foundations and supports needed to provide timely and secure access to data. This includes instituting a system of data governance and stewardship to ensure that data is available, consistent, and of good quality. Further, the strategy aims to ensure that existing governance structures are streamlined to improve data access while strengthening security. Finally, the Chief Data Office is working with key stakeholders to establish secure and auditable environments for data and analysis, and establishing a departmental analytics program to maximize the business value of data.

Leading the way

The Evaluation Directorate continues to move forward on the new agenda to deliver results for Canadians while building on the strong foundation of evaluation findings in program and policy development. Evaluation will collaborate closely with its key stakeholders which include program authorities and central agencies. The Directorate will also work to strengthen the evaluation function at ESDC by investing in innovation and learning, making use of the increased flexibility under the Policy on Results, sharing expertise of trained analysts and evaluators with other ESDC branches, and continuing to expand its dissemination efforts.

Building capacity in the Directorate will enable evaluators to conduct horizontal and thematic evaluations on a broader scope, such as clustering apprenticeship program evaluations and interdepartmental evaluation of the Passport program. For example, when the Indigenous Early Learning and Childcare program is fully established, a horizontal approach for the evaluation of the program will be considered.

Innovation and learning

The Directorate will continue to innovate and better meet its mandate by:

Use of flexibility

To further support the ongoing implementation of the Policy on Results, the Directorate will continue to:

Dissemination

The Evaluation Directorate is committed to modernizing its communication approach to increase awareness and use of evaluation findings. In 2017, the Directorate developed a dissemination strategy which aims to:

Dissemination activities encompass four target audiences: departmental employees, Government of Canada employees, the academic community, and Canadians at large. Evaluators will continue to implement its dissemination strategy by:

The dissemination strategy is an evolving initiative that continues to adapt as opportunities arise to use new communication platforms and tools.

Conclusion

In recent years, the Evaluation Directorate has evolved its work from output-focused evaluations to more outcome-oriented analysis and reports. Previously, in the absence of solid administrative data, evaluations tended to focus on analyzing output, performance measures, and qualitative outcome measures. Recently, with the investment in data and performance measurement, evaluation work has increasingly focused on impact and on ensuring the provision of more timely, policy-relevant evaluations.

The Directorate has also strengthened its internal capacity, planning processes, and collaboration with program areas; developed innovative approaches to produce evidence; and increased dissemination efforts. Going forward, as performance measurement systems and data strategy further mature, evaluations will increasingly emphasize more policy-relevant evidence and utilize horizontal, thematic approaches to deliver on the Government’s Policy on Results.

Annex 1: Five Year Evaluation Schedule

Programs planned to be evaluated in the next five years

Program Last evaluation Evaluations planned in the next 5 years Fiscal year of approval 2018 to 19 Program spending covered by the planned evaluationFootnote 18
(dollars)		 2018 to 19 Program spending covered by all planned evaluations
(dollars)		 2018 to 19 Total program spending
(dollars)		 2018 to 19 Total gross program spendingFootnote 19
(dollars)		 Rationale for not evaluating Program or spending
Homelessness Partnering StrategyFootnote 22 Evaluation of the Homelessness Partnering Strategy 2018 Homelessness Partnering Strategy – 2021 2021 to 22 179,417,053 179,417,053 179,417,053 179,417,053  
Social Development Partnerships ProgramFootnote 22 Summative Evaluation of the Social Development Partnerships Program Social Development Partnerships Program (SDPP) - Summative Evaluation 2019 to 20 38,895,693 38,895,693 38,895,693 38,895,693  
  2014              
New Horizons for Seniors ProgramFootnote 22 Evaluation of the New Horizons for Seniors Program New Horizons for Seniors Program – Evaluation 2019 to 20 49,309,210 49,309,210 49,309,210 49,309,210  
  2015              
Enabling Accessibility FundFootnote 22 Evaluation of the Enabling Accessibility Fund Enabling Accessibility Fund 2022 Evaluation 2022 to 23 22,807,476 22,807,476 22,807,476 22,807,476  
  2017              
Early Learning and Childcare N/A Early Learning and Childcare 2022 to 23 416,916,802 416,916,802 416,916,802 416,916,802  
Federal Income Support for Parents of Murdered or Missing Children Evaluation of the Federal Income Support for Parents of Murdered or Missing Children Grant 2022 Evaluation of the Federal Income Support for Parents of Murdered or Missing Children 2022 to 23 11,252,381 11,252,381 11,252,381 11,252,381  
  2018              
Old Age Security Evaluation of the Old Age Security Program: Phase 1 Old Age Security (OAS) Phase 2 2018 to 19 54,145,595,510 54,145,595,510 54,145,595,510 54,145,595,510  
  2017 OAS/GIS Service Improvement – Evaluation 2021 to 22 see above        
                 
    Guaranteed Income Supplement Take-up Evaluation Phase I (2018 to 19) 2019 to 20 see above        
Canada Disability Savings Program Formative Evaluation of the Canada Disability Savings Plan: Canada Disability Savings Bonds & Canada Disability Savings Grant Canada Disability Savings Program - Summative Evaluation, Phase II 2018 to 19 671,810,903 671,810,903 671,810,903 671,810,903  
  2014              
Canada Pension Plan Canada Pension Plan Disability - Summative Evaluation Canada Pension Plan Disability Benefits Summative Evaluation 2020 to 21 29,735,268 29,735,268 29,735,268 47,283,133,523  
  2011              
Employment Insurance Evaluation of Initiatives to Extend EI Regular Benefits Working While on Claim Pilot Projects 2018 to 19 see below 69,717,337 69,717,337 19,247,913,320  
  2016 Connecting Canadians with Available Jobs 2018 to 19 see below        
    EI Sickness Benefits – Evaluation 2018 to 19 see below        
    Benefit for Parents of Critically Ill Children – Evaluation 2018 to 19 see below        
    EI Benefits and Post-Claim Outcomes – Evaluation 2020 to 21 69,717,337        
Workforce Development Agreements N/A Workforce Development Agreements 2021 to 22 799,810,093 799,810,093 799,810,093 799,810,093  
Labour Market Development Agreements Second Cycle of Labour Market Development Agreements Labour Market Development Agreements 2023 2022 to 23 2,410,531,390 2,410,531,390 2,410,531,390 2,410,531,390  
  2018              
Opportunities Fund for Persons with DisabilitiesFootnote 22 Summative Evaluation of the Opportunities Fund for Persons with Disabilities Opportunities Fund for Persons with Disabilities Phase I 2018 to 19 41,075,478 41,075,478 41,075,478 41,075,478  
  2014 Opportunities Fund for Persons with Disabilities (OFPD) Phase 2 - Summative Evaluation 2019 to 20          
Job Bank Job Bank - Evaluation 2011 Job Bank - Evaluation 2018 to 19 4,687,802 4,687,802 4,687,802 21,586,223 The timing of the evaluation is currently under discussion and may be subject to change. If necessary, the new timeline would be reflected in the next DEP.
Youth Employment StrategyFootnote 22 Summative Evaluation of the Youth Employment Strategy Youth Employment Strategy (YES) - Summative Evaluation 2019 to 20 422,524,755 422,524,755 422,524,755 422,524,755  
  2015              
Youth Service Initiative N/A Canada Service Corps 2021 to 22 31,898,337 31,898,337 31,898,337 31,898,337  
Literacy and Essential SkillsFootnote 22 Evaluation of Literacy and Essential Skills Workplace Literacy and Essential Skills 2022 to 23 22,358,803 22,358,803 22,358,803 27,915,532  
Aboriginal Skills and Employment Training StrategyFootnote 22 and Skills and Partnership FundFootnote 22 Evaluation of the Aboriginal Skills and Employment Training Strategy and the Skills and Partnership Fund Aboriginal Labour Market Programming – Evaluation 2019 to 20 267,538,221 267,538,221 267,538,221 366,237,186  
  2015              
Student Work-Integrated Learning ProgramFootnote 22 N/A Student Work-Integrated Learning Program 2020 to 21 17,970,718 17,970,718 17,970,718 17,970,718  
Union Training and Innovation N/A Union Training and Innovation 2020 to 21 27,512,059 27,512,059 27,512,059 27,512,059  
Sectoral Initiatives ProgramFootnote 22 N/A Sectoral Initiatives Program 2018 to 19 7,178,976 7,178,976 7,178,976 27,866,456  
Temporary Foreign Worker Program Evaluation of the Labour Market Opinion Streams of the Temporary Foreign Worker Program Temporary Foreign Worker Program (TFWP) 2020 to 21 49,214,714 49,214,714 49,214,714 49,214,714  
  2013              
Foreign Credential Recognition ProgramFootnote 22 Evaluation of the Foreign Credential Recognition Program and the Interprovincial Labour Mobility Initiative Foreign Credential Recognition Loans Program (2019) - Summative Evaluation 2019 to 20 25,883,667 25,883,667 25,883,667 26,442,522  
  2015              
Enabling Fund for Official Language Minority CommunitiesFootnote 22 Enabling Fund for Official Language Minority Communities (OLMC) Enabling Fund for Official Language Minority Communities 2022 2021 to 22 13,583,353 13,583,353 13,583,353 13,783,353  
  2017              
Canada Student Loans and Grants and Canada Apprentice Loans Program Canada Apprentice Loan - Phase 1 Pathways to Education Canada - Evaluation 2018 to 19 9,500,000 2,089,142,258 2,089,142,258 2,089,142,258  
  2017 Canada Apprentice Loan – Phase 2 2019 to 20 5,278,857        
    Canada Student Loans Program 2019 to 20 2,074,363,401        
    Canada Student Loans Program - Phase 2 2021 to 22          
Canada Education Savings Program Summative Evaluation of Canada Education Savings Program Canada Education Savings Program - Evaluation 2020 to 21 1,078,827,536 1,078,827,536 1,078,827,536 1,078,827,536  
  2015              
Skilled Trades and Apprenticeship (Red Seal Program) N/A     N/A N/A 643,906 17,033,059 Red Seal Program is funded with EI Part II funds converted to O&M. There is no associated evaluation requirement.
Apprenticeship GrantsFootnote 22 Apprenticeship Grants and Red Seal Phase 1 - Summative Evaluation Apprenticeship Grants (AG) Phase I 2018 to 19 120,777,849 120,777,849 120,777,849 120,777,849  
  2014              
Labour Relations Summative Evaluation of the Workplace Dispute Prevention and Resolution Evaluation of the Federal Mediation and Conciliation Service (FMCS) 2018 to 19 16,334,732 16,334,732 16,334,732 16,334,732  
  2014              
Federal Workers’ Compensation Workers Compensation Cost Recovery Program – Administration Evaluation Federal Workers’ Compensation Service (FWCS) Phase I 2018 to 19 48,931,661 48,931,661 48,931,661 174,166,661  
  2007 Federal Workers’ Compensation Service - Evaluation 2022 to 23          
Occupational Health and Safety Evaluation of Labour Program Supplementary Funding with Limited Scope Occupational Health and Safety - Evaluation 2018 to 19 23,570,624 23,570,624 23,570,624 23,570,624  
  2005              
Workplace Equity Employment Equity Program Employment Equity Programs – Evaluation 2018 to 19 1,618,926 1,618,926 1,618,926 1,618,926  
  2013              
Labour Standards Labour Program Supplementary Funding Labour Standards - Evaluation 2018 to 19 19,290,230 19,290,230 19,290,230 19,290,230  
  2005              
    Occupational Health and Safety – Evaluation 2018 to 19 see above        
Wage Earner Protection Program Summative Evaluation of the Wage Earner Protection Program Wage Earner Protection Program (WEPP) - Evaluation 2021 to 22 51,329,234 51,329,234 51,329,234 51,329,234  
  2014              
International Labour Affairs Bilateral and Regional Labour Agreements and Multilateral Labour Affairs International Labour Affairs 2022 to 23 5,882,080 5,882,080 5,882,080 5,882,080  
  2018              
Government of Canada Telephone General Enquiries Services N/A Providing Services and Information to Canadians 2019 to 20 15,413,377 15,413,377 15,413,377 20,413,377  
Government of Canada Internet Presence N/A Providing Services and Information to Canadians 2019 to 20 10,952,300 10,952,300 10,952,300 10,952,300  
In-Person Points of Service N/A Providing Services and Information to Canadians 2019 to 20 22,545,249 22,545,249 22,545,249 22,545,249  
Passport Evaluation of Passport Receiving Agent Service Offering Passport Program DeliveryFootnote 20 2018 to 19 21,349,179 21,349,179 21,349,179 194,647,083  
  2014              
Other Government Department Programs N/A N/A N/A N/A N/A 115,194 1,677,833 No applicable government programs.
Internal Services N/A N/A N/A N/A N/A 248,131,493 824,994,686 No applicable government programs.
Other CostsFootnote 21 N/A N/A N/A N/A N/A N/A 1,384,577,942 No applicable government programs.
Total       60,950,621,788 60,950,621,788 61,199,512,381 132,466,700,481  

Annex 2 – Departmental Results Framework - Program Inventory

CR1: Social Development CR2: Pension and Benefits CR3: Learning Development and Employment Programs CR4: Working Conditions and Workplace Relations CR5: Information delivery and services for other departments
  • Homelessness Partnering Strategy
  • Social Development Partnerships Program
  • New Horizons for Seniors Program
  • Enabling Accessibility Fund
  • Early Learning and Child care
  • Federal Income Support for Parents of Murdered or Missing Children
  • Old Age Security
  • Canada Disability Savings Program
  • Canada Pension Plan
  • Employment Insurance
  • Labour Market Development Agreements
  • Workforce Development Agreements
  • Opportunities Fund for Persons with Disabilities
  • Job Bank
  • Youth Employment Strategy
  • Skills and Partnership Fund
  • Literacy and Essential Skills
  • Aboriginal Skills and Employment Training Strategy
  • Student Work-Integrated Learning Program
  • Union Training and Innovation
  • Sectoral Initiatives Program
  • Temporary Foreign Worker Program
  • Foreign Credential Recognition Program
  • Enabling Fund for Official Languages Minority Communities
  • Canada Student Loans and Grants and Canada Apprentice Loans Program
  • Canada Education Savings Program
  • Apprenticeship Grants
  • Skilled Trades and Apprenticeship (Red Seal Program)
  • Canada Service Corps (formerly Youth Service Initiative)
  • Labour Relations
  • Federal Workers’ Compensation
  • Occupational Health and Safety
  • Workplace Equity
  • Labour Standards
  • Wage Earner Protection Program
  • International Labour Affairs
  • Government of Canada Telephone General Enquiries Services
  • Government of Canada Internet Presence
  • In-Person Points of Service
  • Passport
  • Other Government Department Programs

Page details

Date modified: