Audit of Program Performance Measurement
From: Employment and Social Development Canada
Official title: Audit of Program Performance Measurement
On this page
Alternate formats
Audit of Program Performance Measurement [PDF - 510 KB]
Large print, braille, MP3 (audio), e-text and DAISY formats are available on demand by ordering online or calling 1 800 O-Canada (1-800-622-6232). If you use a teletypewriter (TTY), call 1-800-926-9105.
List of abbreviations
- CPMD
- Corporate Planning and Management Directorate
- ESDC
- Employment and Social Development Canada — The Department
- PIP
- Performance Information Profile
- PM
- Performance Measurement
- PMD
- Performance Measurement Division
- PMEC
- Performance Measurement and Evaluation Committee
- PMIP
- Performance Measurement Improvement Plan
- SSPB
- Strategic and Service Policy Branch
- TB
- Treasury Board
1. Background
Employment and Social Development Canada (ESDC — the Department) delivers a range of programs and services. These programs and services have an impact on Canadians throughout their lives. Establishing and maintaining a robust performance measurement within the Department is key. This is essential for delivering programs and improving service and results to Canadians.
The Treasury Board's (TB) Policy (and Directive) on Results sets out the basic requirements for performance information.
1.1 Audit Objectives
The goal of this audit was to offer assurance to senior managers on the extent that program performance measurement:
- effectively delivers information to management and other stakeholders to make informed decisions
- delivers processes according to Treasury Board policy
1.2 Scope
The scope of this audit included a range of programs and their related performance measurement processes and products.
1.3 Methodology
The audit utilized the following methodologies:
- documentation review and analysis
- sampling
- data analysis and
- interviews with management and staff
2. Audit findings
2.1 PMD's Performance Measurement Improvement Plan targets many issues, but opportunities exist to enhance the plan
The Performance Measurement Division (PMD) recognizes the challenges and deficiencies in managing performance measurement (PM) within the department. As a result, PMD developed an extensive Performance Measurement Improvement Plan (PMIP).
The PMIP includes the following improvements:
- updates to the Departmental Results Framework
- development of thematic Program Information Profiles (PIPs)
- creation of a performance measurement database
- development of training materials and
- creation of a governance body to oversee implementation of the plan
The PMIP identifies and addresses many of the deficiencies identified during the engagement. To further strengthen the plan's comprehensiveness, Internal Audit noted the following enhancement opportunities. The opportunities noted included: i) clarifying objectives; ii) identifying risks, dependencies and mitigation strategies, and; iii) identifying progress success factors.
2.2 Enhancing and reinforcing the Departmental Program Performance Measurement Framework and Oversight is essential
Overall, the existing performance measurement guidance aligns with the requirements of the TB Policy (and Directive) on Results. Yet, a need exists to further strengthen performance measurement governance and oversight.
The Performance Measurement and Evaluation Committee (PMEC) allocates very limited time to performance measurement. Many meetings do not include performance measurement as a formal agenda item. Also, the PMEC does not play a significant role in oversight and approval of PIPs, nor their elements. This can lead to inadequate strategic direction and inconsistency in the program performance measurement approach. It can then impact the ability to give effective and coherent performance information to senior decision-makers. There exist instances where the PMD's recommendations go unaddressed within the PIPs. Internal Audit observed that this resulted in inadequate and inconsistent departmental performance measurement practices.
Moreover, stakeholders in operational and service delivery do not have a significant opportunity for input on PIP elements. A large number of program stakeholders responsible for PIP development were not aware of existing resources. Some also indicated a need for improving resources supporting performance measurement development. The PMIP includes notable initiatives aimed at improving program stakeholder training.
Recommendation
1. The Chief Results and Delivery Officer, Strategic and Service Policy Branch (SSPB), should strengthen and reinforce governance oversight through periodic tabling of PIPs at PMEC for their scrutiny of performance measurement utility and results reporting.
2. The Chief Results and Delivery Officer, SSPB, in collaboration with Program Assistant Deputy Ministers should inform PMEC regularly of PIP utility and any obstacles to results reporting, as PIPs are developed and/or updated.
Management response
1. The Chief Results and Delivery Officer agrees with this recommendation.
In a department as large as ESDC, it has been difficult for PMEC to oversee all 51 program PIPs. To facilitate PMEC oversight on PIPs, the PMIP proposes to move to thematic PIPs. This will reduce the total number of PIPs, making oversight more meaningful and manageable for both the programs and for PMEC.
In addition, the Corporate Planning and Management Directorate (CPMD) has created a new Director-level committee, the Performance Measurement Implementation Committee (PMIC), to align performance measurement work of the programs and corporate functions and serve as a feeder committee to the PMEC.
2. While the Chief Results and Delivery Officer agrees with this recommendation, there is a risk that a lack of performance measurement resources, systems, and processes may limit the Department's ability to fully respond to this recommendation.
CPMD reports regularly to PMEC, through the Head of Performance Measurement, and recently (2022) provided a full report of its investigation of the quality, use and utility of ESDC's current suite of PIPs.
Based on findings from this investigation, CPMD created the PM Improvement Plan, which aims to improve the quality, use, and utility of PIPs across the department. This will be done by creating thematic PIPs to which each program will contribute a few indicators. evolution will make reporting on the PIPs more focused on functional indicators for which the programs have collected data, and directly link PIP reporting to ESDC's Core Responsibilities.
Progress on thematic PIPs, and later on results under the thematic PIPs, will be discussed regularly at the PMEC table.
2.3 Program performance information lacks overall use and utility
The Department reports a subset of indicators from the PIPs to the public. Many program performance indicators that the department reports do not have data or are not meeting their goals. From 2018-2022, ESDC listed around 16% of its public program-level indicators as "data not available". When Internal Audit requested indicator results for PIPs from prior years, few were available. Among those results, a few were organized and consistent.
Many programs lack an organized and consistent approach to collecting and reporting data for indicators outlined in their PIPs. Moreover, a majority of program stakeholders stated they did not view their PIPs as having significant utility. This is in terms of providing prompt and useful performance information to enable effective decision-making.
PMD did not undertake a fulsome periodic review of PIPs, which could show data collection and reporting deficiencies. The PMIP includes a step to develop an annual internal process related to updating the PIPs.
Recommendation
3. The Head of Performance Measurement, SSPB, in collaboration with program stakeholders, should develop a centralized database of departmental indicators and results to be able to monitor departmental results and draw trends to support the decision-making process.
Management response
3. The Head of Performance Measurement agrees with this recommendation.
CPMD has developed a database of indicators and results that are publicly reported via GC InfoBase both at the department and program levels. This database has proved instrumental in identifying gaps and supporting revisions to the Program Inventory, as well as plans to move to thematic PIPs.
After the thematic PIPs are created, CPMD plans to expand the database to non-public facing indicators. This would expand results monitoring and reporting to senior management.
3. Conclusion
The audit found the current performance measurement approach and guidance complied with Treasury Board's Policy (and Directive) on Results.
The following practices can further strengthen the effectiveness of the performance measurement:
- Increasing the level of departmental governance and oversight of program performance measurement to support effective decision-making
- Strengthening PIP completeness, relevancy and readiness as well as adopting a more consistent approach to performance measurement
4. Statement of assurance
In our professional judgment, the audit team gathered and analyzed sufficient and appropriate evidence to support the accuracy of conclusions found in this report. The audit team based these conclusions on observations and analyses at the time of the audit. These conclusions are applicable only for the Audit of Program Performance Measurement. The audit team followed the Treasury Board Policy on Internal Audit and the International Standards for the Professional Practice of Internal Auditing.
Appendix A: Audit criteria assessment
- Audit criteria: Framework and oversight
- Rating: Controlled, but should be strengthened; medium-risk exposure
- Audit criteria: Use and Utility of Performance Information
- Rating: Controlled, but should be strengthened; medium-risk exposure
Page details
- Date modified: