Audit of the Implementation of the Policy on Results

Office of the Chief Audit Executive

March 2019

This publication is available upon request in alternative formats.

This publication is available on the Internet at:

Canada.ca/Canadian-Heritage

©Her Majesty the Queen in Right of Canada, 2019
Catalogue number: CH6-60/2019E-PDF
ISBN: 978-0-660-30908-8

Table of contents

List of appendix

Appendix A – Summary of governance requirements

Appendix B – Assessment scale and results summary

Appendix C – Management action plan

List of acronyms and abbreviations

CIO
Chief Information Officer
DM
Deputy Minister
DRF
Departmental Results Framework
ESD
Evaluation Service Directorate
EXCOM
Executive Committee
PCH
Department of Canadian Heritage
PIP
Performance Information Profile
PMEC
Performance Measurement and Evaluation Committee
RIPEC
Results, Integrated Planning and Evaluation Committee
SPPCA
Strategic Policy, Planning and Corporate Affairs
SPD
Strategic Planning Directorate

Executive summary

With the 2016 Policy on Results, the Government of Canada has an expenditure management system that seeks to direct resources towards achieving results. The policy introduced a number of new components, and revised existing approaches to performance measurement and evaluation. The policy innovates in how these two functions are formally expected to collaborate in order to support departments towards increased achievement of results. The policy also reinforces the accountability of Deputy Heads and outlines important responsibilities for Heads of Performance Measurement, Heads of Evaluation, Program Officials, Chief Financial Officers and Chief Information Officers.

Audit opinion and conclusion

Based on the audit findings, my opinion is that the Department of Canadian Heritage (PCH) has put in place the main requirements of the Policy on Results. PCH has also put in place a management control framework that will allow to maintain and improve these requirements as the Department begins to report on results under the policy. Elements of the framework were tested based on the level of risk and it was found that:

Statement of conformance

In my professional judgment as Chief Audit Executive, this audit was conducted in conformance with the Institute of Internal Auditors' International Standards for the Professional Practice of Internal Auditing and with the Policy and Directive on Internal Audit of the Government of Canada, as supported by the results of the quality assurance and improvement program. Sufficient and appropriate audit procedures were conducted, and evidence gathered, to support the accuracy of the findings and conclusion in this report. The findings and conclusion are based on a comparison of the conditions, as they existed at the time, against pre-established audit criteria that were agreed with management and are only applicable to the entity examined and for the scope and time period covered by the audit.

Original signed by

Bimal Sandhu
Chief Audit Executive
Department of Canadian Heritage

Audit Team Members
Sophie Frenette, Audit Manager
Chrystianne Pilon, Senior Auditor
Carolann David, Senior Auditor
Houssein Ndiaye, Auditor

With the support of external resources.

1.0 Background

The management of results seeks to enhance the achievement of results and understanding of the resources used to achieve them. With the 2016 Policy and Directive on Results, the Government of Canada expenditure management system was refined to place greater emphasis on directing government resources towards commitments that have an impact on Canadians. Canadian Heritage (PCH) and other departments are expected to:

Figure A – Summary of the Policy on Results

Figure A: Summary of the Policy on Results
Figure A: Summary of the Policy on Results – text version

Summary of the Policy on Results requirements: The Deputy Head is required to establish a Departmental Results Framework and to identify and chair a Performance Measurement and Evaluation Committee to oversee departmental performance measurement and evaluation. Performance measurement function at PCH is the responsibility of the Head of Performance Measurement who is also responsible for establishing, implementing and maintaining a Program Inventory and overseeing Performance Information Profiles. The PCH evaluation function is the responsibility of the Head of Evaluation .

To achieve these expected policy outcomes, deputy heads are required to put in place governance in support of managing for results (Policy s.4.3.3), to establish a Departmental Results Framework (DRF) (s.4.3.1), to develop a Program Inventory and associated Performance Information Profiles (PIP) (s.4.3.5 and 4.3.6), and to implement associated responsibilities and functions (s.4.3.4, 4.3.5, 4.3.6, 4.3.11 and 4.3.12). A Program is understood as individual or group of services and activities managed together to focus on a specific set of outputs, outcomes or service levels (Policy Appendix A). A summary of these requirements is reflected in Figure A. Departments are expected to carry out activities such as preparing documentation, reporting on results (internally and externally), establishing governance, and providing advice to Deputy Heads and to the Performance Measurement and Evaluation Committee.

PCH has deployed efforts to implement the new policy requirements. A DRF was approved in November 2017 and inaugural PIPs have been developed, and went through one round of update in 2018. The first Departmental Plan to use the DRF was published for 2018-2019. The DRF, PI and PIPs establish the framework of what the Department wishes to accomplish, and the indicators it will use to determine if results are being attained. To fully realize the objectives of the Policy on Results, it is expected that management use the resulting information to manage and improve programs, policies, and services (Policy s. 3.2.2), and to support evaluations.

It was recognized in the planning phase of this audit that PCH was in the early stages of implementing the Policy on Results. The first Departmental Results Report using the DRF is scheduled for 2019, and internal use of the framework is just beginning. In addition, a number of draft documents, such as competency profiles (Policy s.2.3.2), need to be finalized by the Treasury Board of Canada Secretariat to allow PCH to respond to and implement related requirements (Policy s.4.3.5 and s.4.3.12).

It is important to acknowledge efforts made by the Department to integrate requirements for results and delivery (an initiative led by the Privy Council Office that supports ministerial accountability) and for the Policy on Results (an internal tool from the Treasury Board of Canada Secretariat pursuant to the Financial Administration Act). While it falls outside of the scope of this audit, these efforts have greatly reduced the reporting burden on programs within PCH.

2.0 About the audit

2.1 Project authority

The Office of the Chief Audit Executive completed the Audit of the Implementation of the Policy on Results, in accordance with the Risk-based Audit Plan for 2018-2019 to 2020-2021.

2.2 Objective and scope

The objective of this audit was to provide assurance regarding the management control framework that PCH has put in place to achieve the expected results of the Policy on Results.

The scope of this audit initially covered the period from July 1, 2016 to October 25, 2018; however, it was extended to October 31, 2018, to include an update of PIPs.

The scope excluded requirements related to Treasury Board Submissions and Memoranda to Cabinet. It was not feasible to look at both these processes within the context of an audit of the Policy on Results, as they respond to other requirements outside of results management. A stand-alone audit of these processes, which would include all requirements, would be more value-added.

2.3 Approach and methodology

This audit was conducted in accordance with the Treasury Board Policy on Internal Audit, its affiliated directive, and the International Standards for the Professional Practice of Internal Auditing of the Institute of Internal Auditors. This included a risk analysis to focus the audit on those areas of the Policy on Results that were determined to be of greatest risk or significance to the Department.

The methodology for this audit included:

3.0 Findings and recommendations

Findings are based on the evidence gathered through the combined results of interviews, analyses and documentation reviewed, for each audit criteria. A summary of these findings, aligned with the audit criteria, is reflected in Appendix B. Findings of lesser materiality, risk or impact will be communicated with the auditee either verbally or in management letters.

3.1 Oversight of results

Most of the requirements regarding oversight of results are met. There is an opportunity to either clarify or simplify the two-tier governance structure that PCH has put in place, for greater efficiency and alignment with policy requirements

The Policy on Results sets out the requirements for a departmental Performance Measurement and Evaluation Committee (Policy s.4.3.3). The Committee should be chaired by the Deputy Minister of PCH (henceforth referred to as the Deputy Minister). However, there is some latitude in how departments structure their governance, as long as the Deputy Minister approves the required documents and the advice of key stakeholders on the various requirements is obtained.

At PCH, it was decided to designate two existing committees responsible for the Policy on Results when it came into force in 2016: the Results, Integrated Planning and Evaluation Committee (RIPEC), a level 2 committee chaired by the Assistant Deputy Minister of Strategic Policy, Planning and Corporate Affairs; and the Executive Committee (EXCOM), a level 1 committee composed of senior members and chaired by the Deputy Minister. The two-tier structure is seen as efficient by departmental officials as it allows for the refinement of documents at RIPEC (such as annual reports and evaluation reports), and more focused discussions at EXCOM.

A survey with EXCOM and RIPEC members (26% response rate) provided insight into the governance structure:

  • The time available for reviewing documentation in advance of meetings was noted as a barrier for fulsome discussions by these committees.
  • Members assessed their efficiency as higher when reviewing, advising and making decision on matters related to evaluation (60% for RIPEC and 43% for EXCOM) than performance information (30% for RIPEC versus 13% for EXCOM).

The PCH governance structure was not fully documented in committee terms of reference, and the audit observed instances of an unclear understanding of the two-tier structure. For example, a presentation to RIPEC for the update of PIPs noted that the responsibilities of a Performance Measurement and Evaluation Committee were performed by RIPEC at PCH. However, to be in compliance with the Policy and Directive on Results, a Performance Measurement and Evaluation Committee should be chaired by the Deputy Minister; at PCH, that is EXCOM. Without clearly outlining the relationship between RIPEC and EXCOM, there is a risk that functions which should be performed at the committee chaired by the Deputy Minister are carried out exclusively at RIPEC.

As a result, there is an opportunity to review the balance between efficient senior management discussions, and the requirements for specific documents and advice to be tabled at a committee chaired by the Deputy Minister. Appendix A provides a summary of governance requirements from the Policy and Directive on Results and observations from the audit regarding potential improvements, including responsibilities to provide advice to the Deputy Minister which are exclusively assigned to RIPEC.

Recommendation:

  1. The Assistant Deputy Minister, Strategic Policy, Planning and Corporate Affairs, should lead a review of the governance structure for results to ensure it meets all the requirements of a Performance Measurement and Evaluation Committee, either by:
    1. Updating the terms of reference for the Results, Integrated Planning and Evaluation Committee and for the Executive Committee, for a more complete description of roles; or
    2. Having a single committee chaired by the Deputy Minister perform all functions outlined in the Policy and Directive on Results.

3.2 Performance measurement roles, responsibilities and accountabilities

Roles, responsibilities, and accountabilities for performance measurement, including those of corporate functions and program officials, are for the most part well understood. There is an opportunity to reinforce the role of Program Officials.

The Policy on Results includes a number of roles for performance measurement that were mostly in place prior to the new policy. However, they are now a government-wide obligation; the Deputy Minister’s accountability has also been reinforced. At PCH, the Assistant Deputy Minister, Strategic Policy, Planning and Corporate Affairs is the Head of Performance Measurement, and Head of Evaluation; both functions are in that Sector, under the leadership of a Director General, Strategic Planning, Evaluation and Research Branch and two Directors (one for Evaluation Services and one for Strategic Planning). The Assistant Deputy Minister is also the chair of RIPEC and was delegated the authority for approving updates to PIPs by the Deputy Minister.

Within the sectors of PCH, nine Program Officials are responsible for developing, implementing and maintaining PIPs for 13 Programs, and ensuring that valid, reliable, useful performance data is collected and available; there is one Director, and eight Directors General. Sectors planners support Program Officials and participate in a planning network led by the Strategic Planning Directorate. It is in large part due to the efforts deployed by the Strategic Planning Directorate to document roles and responsibilities, offer training and guidance, that roles and responsibilities are well understood.

Two opportunities for improvements have been identified, with regard to the role of Program Officials. Now that PCH has developed its first set of PIPs, and they have been through one round of update, it is important to more formally and systematically use them. The value-added is not the development of indicators, but rather when the reports on results are used to manage and improve programs, policies and services. Reporting on results will enable Program Officials to observe the level of effort required to collect, report and monitor data linked to indicators and to adjust indicators in the PIPs based on experience and usefulness for decision-making. Inviting Program Officials to present results at senior management committees would also reinforce their responsibility for the collection and availability of data (Directive s.4.3.1), and allow committees to integrate results in their oversight of programs.

It will also be important for the Department to recognize and reflect the responsibilities of Program Officials for any work related to program data, which impacts results management, in the future. For example, a temporary working group is in place – the PCH Grants and Contributions Data Management Working Group – which oversees the development, collection and use of new grants and contributions data, in the context of the Grants and Contributions Modernization Project. While this working group is not a requirement of the Policy on Results, representatives from four “early adopters” grants and contributions programs participate in the working group; these four grants and contribution programs are covered in three PIPs. There is an opportunity to review the terms of reference for the working group to reflect Program Officials responsibilities for data collection, as per the Directive on Results. This will help ensure that this important PCH modernization initiative, which includes reviewing data, is in line with the requirements of Directive on Results (s.4.3.1). Consideration should also be given to include Program Officials in any future data work the Department undertakes.

Recommendation:

  1. The Assistant Deputy Minister, Strategic Policy, Planning and Corporate Affairs should reinforce the role of Program Officials by:
    1. Inviting Program Officials to present the results of PIPs to relevant committees to provide oversight of results; and
    2. Ensuring that the terms of reference for the Grants and Contributions Data Management Working Group is in line with the responsibilities for data collection and availability in the Directive on Results.

3.3 The Departmental Results Framework

PCH developed a Departmental Results Framework in compliance with the Deputy Minister’s responsibilities outlined in the Policy on Results; alignment with the other responsibilities in the Directive on Results could be improved.

The DRF consists of Core Responsibilities, Departmental Results and Departmental Results Indicators (as depicted in Figure A). It is the basis of what PCH is trying to achieve and how it measures progress. The DRF is also the framework for external reporting to parliamentarians (via the Departmental Plan which is part of the Government Estimates process) and to the public (via InfoBase, a data-visualization tool which provides financial, people and results information).

The DRF was the first key deliverable that PCH developed for the Policy on Results. For the most part, this involved adapting the former “Program Alignment Architecture” to the new policy requirements, and updating it for recent changes such as the return of the Multiculturalism Program to PCH. The financial coding structure in the departmental financial system (SAP) was also adapted to report financial information by Core Responsibilities, and Internal Services. The first PCH DRF was approved in November 2017 and formed the basis of the 2018-2019 Departmental Plan; it was completed in compliance with the Deputy Minister’s responsibilities outlined in the Policy on Results (Policy s.4.3.1 and 4.3.2).

The Directive on Results outlines responsibilities related to the DRF for RIPEC, EXCOM (Directive s.4.1.1), the Head of Performance Measurement (s.4.2.5), the Head of Evaluation (s.4.4.2), and the Chief Information Officer (s.4.6.3). RIPEC, EXCOM and the Chief Financial Officer were consulted in the development and approval of the DRF. The Head of Performance Measurement and the Head of Evaluation were also involved in the development and approval of the DRF. While there were extensive consultations, and some advice was found in the development of PIPs, documentation could not be found to demonstrate that the advice was provided to the Performance Measurement and Evaluation Committee, in the context of the development of the DRF; namely on the availability, quality, utility and use of indicators in the DRF, and their usefulness for supporting evaluations (Appendix A, items 3b and 4b). In addition, while the Chief Information Officer is a member of RIPEC and can verbally share advice, no documentation was found to support advice provided by her to the Performance Measurement and Evaluation Committee on the IT applications and tools associated with the DRF (Appendix A, item 5).

There is an annual process led by the Treasury Board of Canada Secretariat to amend DRFs, and one for preparing Departmental Plans. While there were no amendments to the PCH Departmental Results Indicators themselves for the 2019-20 fiscal year, targets were adjusted during the preparation of the 2018-19 Departmental Plan to reflect new information and analysis. The Strategic Planning, Evaluation and Research Branch indicates that targets will be reviewed again for the next Departmental Plan based on new information and analysis of performance. While there is a formal, departmental process in place for reviewing and updating targets during the preparation of Departmental Plans, attention was not brought to these modifications in management briefings and approvals by senior management.

As PCH is currently in its first year of reporting on results based on the DRF, it will be important to consult all stakeholders on the points outlined above. This will help refine the quality of the DRF and help to streamline the reporting process, by increasing the use of IT tools and applications. It will also be important to keep the Deputy Minister apprised of potential changes as a result of this advice on the Departmental Results Indicators. This will help the Department to anticipate and be able to respond to questions regarding modifications to targets that are included in external reports, for the same reporting period.

Recommendation:

  1. The Assistant Deputy Minister, Strategic Policy, Planning and Corporate Affairs should:
    1. Ensure that the advice of the Head of Performance Measurement, the Head of Evaluation and the Chief Information Officer related to the Departmental Results Framework is documented in line with the requirements of the Directive on Results; and
    2. Keep the Deputy Minister and senior management informed of changes to the Departmental Results Framework, which includes Core Responsibilities, Departmental Results and Departmental Results Indicators.

3.4 Performance measurement capacity

PCH’s capacity for performance measurement is at an early maturity level, as it is at varying levels within the Department. While the Department has made efforts to develop performance measurement capacity, continued attention is required as performance measurement capacity impacts the attainment of the objectives of the Policy on Results.

Human resources capacity has been reinforced. However, it remains a limitation. There is also an opportunity to identify complete financial resources and the IT tools and applications required to collect, use and report performance data.

Performance measurement is the process of collecting, analyzing and reporting information regarding the achievement of results, usually done via a set of indicators established in advance. The PIPs identify the performance information for each of the Department’s Programs outlined in the Program Inventory. This includes the indicators used to support departmental results (deputy head responsibility as per the Policy on Results) and the indicators used internally for the management of programs (a Program Official responsibility as per the Directive on Results). Departments are also expected to ensure that IT tools and applications support the timely collection and use of performance data, and to reporting of information (Directive s. 4.6.2).

In a 2017-2018 self-assessment for the Management Accountability Framework, the cost of the corporate performance measurement function was estimated at $500 000 (including salaries, operating and management). PCH also identified Human Resources capacity for performance measurement as a limitation, indicating that this will be an area for improvement for the future; as management had already identified this issue, it was not examined further in the audit.

Significant efforts have been made by the Strategic Planning Directorate to support Departmental staff in strengthening their performance measurement practices, including establishing a planner’s network and providing performance measurement training and PCH-specific guidance. The effort to collect and manage the data required to report on the indicators included in PIPs within Programs (e.g. data acquisition, systems development, time to capture, analyze and report on results) is not fully known as costs are not captured at that level. This does not allow Program Officials to determine the feasibility of reporting on results when selecting performance indicators for their programs.

The PIPs for PCH’s 13 programs were reviewed (based on the versions approved in 2017):

  • The audit team counted approximately 600 indicators in total. There was no formal inventory of indicators, and due to some variations in how indicators were presented, the Department estimated the count to be upwards of 700.
  • Approximately 7% of indicators included in the PIPs are used for external reporting and results from these indicators have been included in past reports. However, the feasibility of reporting on the results of the remaining 93% of indicators meant to be used for internal reporting is not yet known.

PIPs were updated late in October 2018, resulting in a reduction of approximately 50 indicators, as per the audit team’s count.

The process to develop PIPs in November 2017 and their first update in 2018 included participation from all required parties, except that the Chief Information Officer was involved only as a member of RIPEC. There is also an opportunity going forward to design a consultation strategy which will result in a more in-depth review by key stakeholders. For example, the Head of Performance Measurement and the Head of Evaluation should provide advice to RIPEC and EXCOM on the availability, quality, validity, and reliability of the performance indicators in the PIPs, including their utility for evaluation (Directive s.4.2.4 and 4.4.4.4). The desire to have a more in-depth review is also supported by the governance survey results. The Chief Information Officer’s ability to provide the advice to governance and the Head of performance measurement, as per the Directive (Directive s.4.6.3) on the IT tools and applications required to collect and use performance data, was also limited as a result of gaps in Performance Information Profiles.

Tests were conducted with the PIPs for the Programs with the highest planned spending for each of PCH’s five core responsibilities. This review represented 79% of the 2018-2019 PCH budget, and highlighted the following opportunities for improvement:

  • All PIPs should follow the same template, to allow for comparability and to easily identify performance indicators.
  • All PIPs should be fully populated, including the required information for each indicator: performance targets, thresholds, responsibility for data collection, and data location (Directive s.A.2.2.3.1).
  • As a result of these gaps, taken as a whole PCH’s capacity for performance measurement was assessed at an early maturity level.

A cursory validation of the October 2018 updated PIPs confirmed that progress had been made in identifying targets and thresholds. However, more efforts are required regarding the consistency of use of templates.

In summary, PCH has made significant efforts to further develop the necessary foundations for overall performance measurement. However, performance measurement capacity is unequal across the Department, and the control frameworks and tools (i.e. DRF, PI, and PIPs) that were put in place to measure performance and manage results are not all consistently used for monitoring and reporting on results. Departmental PIPs are a key component of core performance management controls (Directive s.4.3.1) and PCH’s 13 PIPs still require significant improvements as the department recently completed the first round of update for these evergreen documents. This represents a major capacity deficiency in the current control structure and consequently makes it more difficult to link performance information with internal operational planning which is part of the expected results of the Policy on Results (s.3.2.2 and s.3.2.3). Effort should also be placed on identifying IT tools and applications as they will be instrumental in the implementation of PIPs in that they would support cost-effective collection of performance information.

Recommendation:

  1. The Assistant Deputy Minister, Strategic Policy, Planning and Corporate Affairs should design a process for Program Officials to update Performance Information Profiles that will:
    1. Streamline the number of indicators, ensure indicators have all the required components as per the Directive on Results, and review for feasibility of implementation; and
    2. Allow for a more robust, in-depth review and input from the Head of Performance Measurement, the Head of Evaluation and the Chief Information Officer, as per the Directive on Results.

3.5 Evaluation services

The Department generally complies with the Deputy Minister’s responsibilities in the Policy on Results. Evaluation services are situated on the higher end of a maturity scale.

Evaluation is the systematic and neutral collection and analysis of evidence to judge merit, worth or value, generally including questions related to relevance, effectiveness and efficiency. As per the Policy on Results, the Deputy Minister is responsible to establish and maintain a robust, neutral evaluation function (Policy s.4.3.11); ensure access to departmental information (s.4.3.13); complete a departmental evaluation plan (s.4.3.15); approve evaluation reports, including management action plans, and for their publication on the PCH intranet (s.4.3.17 and 4.3.18). Finally, a neutral assessment of the evaluation function should be conducted every five years (s.4.3.19).

PCH has an established evaluation function with staff comprised of approximately 15 full-time equivalents. The function completed and published on average 5 evaluation reports per year (including management responses and action plans), since 2016. The function reports having access to the information required to complete evaluations, insofar as it being available in the DepartmentFootnote 1. The current evaluation plan, covering 2018-2019 to 2022-2023, was approved by the Deputy Minister in July 2018; it includes the required programs, in addition to internal services.

Work is underway to prepare and define the scope for the 2018-2019 neutral assessment of the evaluation function. As a result, the audit did not carry out a detailed review of the evaluation requirements outlined in the Directive on Results, including mandatory procedures and a standard for evaluation (Directive Appendix B and C). It should be noted that the directive contains flexibility with regard to evaluation planning (s.B.2.2.2) and the questions to include in the scope of evaluations (s.B.2.2.5). The Evaluation Service Directorate has begun to take advantage of these new flexibility, and is exploring how to yield efficiency in order to dedicate more of their resources to better addressing departmental risks and priorities.

3.6 Resource allocation

In order to meet the requirements of the Policy and Directive on Results, there is an opportunity for PCH to consider the use of performance information for resource allocation and reallocation, to optimize results.

Linking results and performance indicators to internal operational planning and reporting mechanisms is critical to ensuring that the Department manages for results. It is also consistent with commitments to Parliament. A key outcome of the Policy on Results is for the allocation of resources to be based on performance to optimize results (Policy s.3.2.3). As a result, while it may not be the only consideration, resources should be allocated and aligned with expected results consistent with the DRF.

Current budgets are based on the Annual Reference Level Update, which are allocated to sectors. Fund and cost center managers, with their Resource Management Directorate representative, prepare and allocate budgets based on operational needs. Results information is not systematically used in the budget allocation process and there is no mechanism in place that demonstrates a link between resources, results, and operational planning. In-year reallocation, referred to as a pressures exercise, is reviewed at governance committees; performance information is not a driver in this process. However, interviews noted that the 2018 pressures exercise has begun to generate discussions on performance. Programs also report having started to integrate results information in operational planning and program management.

There is an opportunity for the Department to determine how internal processes could start to integrate performance information with resource allocation and reallocation decisions. Going forward, streamlined and strengthened PIPs will provide an effective platform to integrate results with the budgeting (allocation) and pressures (re-allocation) processes.

Recommendation:

  1. The Assistant Deputy Minister, Strategic Policy, Planning and Corporate Affairs should ensure that discussions take place between the Financial Management Branch and the Strategic Planning, Evaluation and Research Branch, to identify whether there are opportunities to amend internal resource allocation processes based on strengthened performance information, and that proposal for changes be recommended to the Deputy Minister for consideration.

3.7 Monitoring the implementation of policy requirements

While PCH has put a lot of effort to monitor and improve the achievement of results, a formal process to monitor the actual implementation of the Policy on Results was not implemented. A tool was developed to monitor the policy requirements with respect to the evaluation function, but it has not been used.

Monitoring encompasses the processes that management puts in place to assess the adequacy and effectiveness of policy implementation and ensure the achievement of all expected results. Deputy Heads are responsible to ensure that departments’ adherence to the Policy on Results and its associated instruments are monitored (Policy s.4.6.1). The audit team expected to find processes and/or mechanisms within the department in order to track, monitor and report on the status of all requirements set out in the Policy on Results, and to take corrective actions when required.

At PCH, the main requirements of the Policy on Results have been implemented and are currently being maintained. However, through interviews with PCH officials, it was found that the responsibility to monitor the implementation of the requirements of the Policy and Directive on Results had not been assigned; and consequently, a monitoring and reporting mechanism has not been implemented. A tool was developed to monitor evaluation requirements of the Policy and Directive on Results. However, it had not yet been put in place.

PCH has some latitude in its approach to requirements of the policy, and some components are still being finalized by the Treasury Board of Canada Secretariat. As a result, it would be important for PCH senior management to be informed of the fulsome state of the implementation of the requirements of the Policy and Directive on Results, and to have an opportunity to discuss the PCH approach where there is latitude.

Recommendation:

  1. The Assistant Deputy Minister, Strategic Policy, Planning and Corporate Affairs should establish and implement a formal process to ensure all requirements of the Policy on Results and its associated instruments are periodically monitored.

4.0 Conclusion

PCH has put in place a management control framework to implement the requirements of the Policy and Directive on Results. The control framework is at a higher level of maturity for evaluation than for performance measurement. However, efforts have been made to reinforce performance measurement capacity. Opportunities for improvements have been identified related to capacity for performance measurement, governance, resource allocation and monitoring, which will allow PCH to make progress towards achieving the expected results of the Policy on Results. As PCH begins to use the planned and actual results from PIPs in management (which includes results for external and internal reporting), adjustments are to be expected based on lessons-learned, greater input from the required stakeholders and usefulness for decision-making.

Appendix A – Summary of governance requirements

The following table presents a summary of governance requirements only (other requirements were included sections 3.2 to 3.7 of this report). The Policy on Results defines a Performance Measurement and Evaluation Committee (PMEC) as a committee of senior officials designated and chaired by the Deputy Head (for the Department, this is the Deputy Minister (DM) of PCH).

The X indicates by whom the function is performed.

Governance requirements Link to policy/directive Function actually performed by Audit observation
DM EXCOM (with DM) RIPEC (w/out DM)
  1. The DM is responsible to:
    • a. establish, implement and maintain a DRF;
P 4.3.1 x
  • b. approve annually a five-year evaluation plan; and
P 4.3.15 x
  • c. approve evaluation reports and summaries.
P 4.3.17 x
  1. The PMEC is responsible to:
    • a. review and advise the DM on the establishment, implementation and maintenance of the DRF, Program Inventory and PIPs, particularly with respect to the following:
      1. The alignment between the DRF and the Program Inventory;
      2. The availability, quality, utility and use of planned performance information and actual performance information; and
      3. The IT application and tool requirements based on advice from the. CIO.
D 4.1.1

Functions assigned to RIPEC only as per terms of reference.

No specific advice to the DM on points i and iii was found.

Advice on ii was provided on planned performance information; no reporting to the DM observed on advice regarding actual performance information.

  • b. Review and advise the DM on departmental evaluation planning and activities;
D 4.1.2 x x Function assigned to RIPEC only as per terms of reference.
  • c. Review evaluation reports and summaries, including management responses and action plans, and recommend approval to the DM;
D 4.1.3 x x

- Function assigned to RIPEC only as per terms of reference.

- Reports reviewed by RIPEC and approved by EXCOM.

  • d. Monitor follow-up on evaluation recommendations and action plans and advise the DM of any issues;
D 4.1.4 x x Function assigned to RIPEC only as per terms of reference
  • e. Review and advise the DM on the availability, quality, utility, and use of performance information including evaluation; and
D 4.1.5

- Function assigned to RIPEC only as per terms of reference.

- No specific advice to the DM was found.

  • f. Review and advise the DM on the neutral assessment of the evaluation function.
D 4.1.6 x x Function assigned to RIPEC only as per terms of reference
  1. The Head of Performance Measurement is responsible to:
    • a. Report, at least annually, to PMEC, on the availability, quality, utility and use of performance measurement data related to the Program Inventory;
D 4.2.4 Plan to table draft 2017-18 report at EXCOM. x RIPEC only authorized to being consulted on the report, as per terms of reference.
  • b. Advising the PMEC on the availability, quality, utility and use of indicators in the DRF.
D 4.2.5 No specific advice was found.
  1. The Head of Evaluation is to:
    • a. Submit draft evaluation reports directly and simultaneously to the DM and the PMEC;
B 2.2.6 x

x

(2nd)

x

(1st)

Reports are first vetted by RIPEC before being submitted to the DM.
  • b. Advise the PMEC on the validity, reliability of Departmental Results Indicators in the DRF, including their usefulness for supporting evaluations; and
D 4.4.2 Advice limited to programs for which there had been evaluations.
  • c. Report to PMEC at least annually, on a number of requirements.
D 4.4.4 Plan to table draft 2017-18 report at EXCOM. x

- 2 reports completed.

- RIPEC authorized to approve as per terms of reference, should be clear that the report should also be tabled at EXCOM.

  1. The CIO should advise PMEC on the department’s IT application and tool requirements or informing them of updates to requirements associated with the DRF, Program Inventory, and the PIPs.
D 4.6.3 No specific advice was found.

Appendix B – Assessment scale and results summary

The conclusions reached for each of the criteria used in the audit were developed according to the following definitions.

Conclusion Definition
Well controlled Well managed, no material weaknesses noted; and effective.
Controlled Well managed and effective. Minor improvements are needed.
Moderate issues

Requires management focus (at least one of the following criteria are met):

  • Control weaknesses, but exposure is limited because likelihood of risk occurring is not high.
  • Control weaknesses, but exposure is limited because impact of the risk is not high.
Significant improvements required

Requires immediate management focus: At least one of the following three criteria are met:

  • Financial adjustments material to line item or area or to the Department.
  • Control deficiencies represent serious exposure.
  • Major deficiencies in overall control structure.
Audit criteria Results summary
1.1. PCH has established effective oversight structure for the achievement of expected results. Moderate issues
1.2. The roles, responsibilities, and accountabilities for performance measurement, including those of corporate functions and program officials, are well understood and meet the requirements of the Policy on Results. Controlled
2.1. The Departmental Result Framework, Program Inventory and Programs set out the Core Responsibilities of PCH and provide a framework for external reporting and management of results. Controlled
2.2. The Department has the capacity (i.e. financial and human resources, and IT tools and applications) to collect, analyze, utilize and report on quality performance information, consistent with the approved results framework, program information profiles and related requirements. Significant improvements required
2.3. The Department’s evaluation function meets the requirements and takes advantage of the flexibilities of the Policy on Results. Well controlled
2.4. PCH resources are allocated based on performance to optimize results. Moderate issues
2.5. PCH’s adherence to the Policy on Results is monitored. Moderate issues

Appendix C – Management action plan

Recommendations Management assessment and actions Responsibility Target date
  1. The Assistant Deputy Minister, Strategic Policy, Planning and Corporate Affairs, should lead a review of the governance structure for results to ensure it meets all the requirements of a Performance Measurement and Evaluation Committee, either by:
    • a. Updating the terms of reference for the Results, Integrated Planning and Evaluation Committee and for the Executive Committee, for a more complete description of roles; or
    • b. Having a single committee chaired by the Deputy Minister perform all functions outlined in the Policy and Directive on Results.

Management agrees with this recommendation.

Existing two-tier governance for performance measurement and evaluation products will be enhanced by updating the Terms of reference of RIPEC and EXCOM to provide more detailed descriptions of the roles and responsibilities of each committee and to better align with the Policy on Results and Directive on Results requirements.

The ToRs for RIPEC will be presented to EXCOM and recommendations and options will then be presented to the Deputy Minister. The preferred option will be implemented.

Assistant Deputy Minister, Strategic Policy, Planning and Corporate Affairs October, 2019
  1. The Assistant Deputy Minister, Strategic Policy, Planning and Corporate Affairs should reinforce the role of Program Officials by:
    • a. Inviting Program Officials to present the results of PIPs to relevant committees to provide oversight of results; and

Management agrees with this recommendation.

  • a. The Terms of reference of the Grants and Contributions Modernization Initiative Working Group will be updated to include reference to roles and responsibilities of Program Officials.
Assistant Deputy Minister, Strategic Policy, Planning and Corporate Affairs (Director SPD) April 30, 2019
  • b. Ensuring that the terms of reference for the Grants and Contributions Data Management Working Group is in line with the responsibilities for data collection and availability in the Directive on Results.
  • b. An awareness session will be developed and delivered to increase understanding of the role of Program Officials under the Policy and Directives on Results.
Assistant Deputy Minister, Strategic Policy, Planning and Corporate Affairs (Director SPD) December 31, 2019
  • c. Options will be developed and put forward to EXCOM via RIPEC on possible mechanisms for presenting results based on PIPs to relevant committees and for reinforcing the use of performance measurement information for better decision-making.
Assistant Deputy Minister, Strategic Policy, Planning and Corporate Affairs (Director SPD) December 31, 2019
  • d. Program Officials will begin to be invited to present, along with SPD, on results based on PIPs at RIPEC.
Assistant Deputy Minister, Strategic Policy, Planning and Corporate Affairs (Director SPD) March 31, 2020
  1. The Assistant Deputy Minister, Strategic Policy, Planning and Corporate Affairs should:
    • a. Ensure that the advice of the Head of Performance Measurement, the Head of Evaluation and the Chief Information Officer related to the Departmental Results Framework is documented in line with the requirements of the Directive on Results; and

Management agrees with this recommendation.

Strategic Planning Directorate will document the process related to the Departmental Results Framework that outlines when advice is required from the Head of Performance Measurement, the Head of Evaluation and the Chief Information Officer, who has to be consulted and at what stage, and the objective that this advice should inform decisions by the Deputy Minister. This process document will be in line with the requirements of the Directive on Results and with internal governance approval processes.

ADM SPPCA

(in collaboration with SPD, ESD and CIO)

June 30, 2019
  • b. Keep the Deputy Minister and Senior Management informed of changes to the Departmental Results Framework, which includes Core Responsibilities, Departmental Results and Departmental Results Indicators.

The process will include steps to ensure that the Deputy Minister and senior management are kept informed of ongoing changes that will feed into the Departmental Results Framework, which includes Core Responsibilities, Departmental Results and Departmental Results Indicators.

Advice from the Head of Performance Measurement, the Head of Evaluation and the Chief Information Officer will be documented accordingly.

March 31, 2020
  1. The Assistant Deputy Minister, Strategic Policy, Planning and Corporate Affairs should design a process for Program Officials to update Performance Information Profiles that will:
    • a. Streamline the number of indicators, ensure indicators have all the required components as per the Directive on Results, and review for feasibility of implementation; and

Management agrees with this recommendation.

Strategic Planning Directorate will lead an analysis to seek opportunities for optimizing existing indicators in collaboration with Program Officials, will ensure clear messaging through guidance documents, and will make presentations to RIPEC and EXCOM.

Director, Strategic Planning Directorate, in collaboration with Program Officials April 2020
  • b. Allow for a more robust, in-depth review and input from the Head of Performance Measurement, the Head of Evaluation and the Chief Information Officer, as per the Directive on Results.
Strategic Planning Directorate in consultation with Evaluation Service Directorate will formalize a process to ensure Program Officials will engage in timely robust and in-depth review of PIPs, based on risk, where all required stakeholders will provide input in line with the Directive on Results and formal approvals will be sought. Director, Strategic Planning Directorate, in collaboration with Program Officials December 31, 2019
  1. The Assistant Deputy Minister, Strategic Policy, Planning and Corporate Affairs should ensure that discussions take place between the Financial Management Branch and the Strategic Planning, Evaluation and Research Branch, to identify whether there are opportunities to amend internal resource allocation processes based on strengthened performance information, and that proposal for changes be recommended to the Deputy Minister for consideration.

Management agrees with this recommendation.

This is an opportunity for the increased use of performance information to embrace a result-oriented culture.

The Strategic Planning, Evaluation and Research Branch and the Financial Management Branch will undertake discussions and work together to analyze current processes and to identify opportunities to better integrate performance measurement and evaluation with resource allocations. SPERB and FMB will work with sectors and examine departmental reallocation decisions, and will brief PCH’s governance and senior management on findings and options.

Assistant Deputy Minister, Strategic Policy, Planning and Corporate Affairs (Strategic Planning, Evaluation and Research Branch) in collaboration with Chief Financial Officer Briefings to governance – December 2019
Implementation of approved options will occur for fiscal year 2020-2021. - May 2020
  1. The Assistant Deputy Minister, Strategic Policy, Planning and Corporate Affairs should establish and implement a formal process to ensure all requirements of the Policy on Results and its associated instruments are periodically monitored.

Management agrees with this recommendation.

A template will be developed to support annual monitoring of the implementation of the Policy and its related instruments which will be integrated in the annual business process.

SPD in collaboration with ESD, Corporate Affairs in collaboration with the Chief Financial Officer and the Chief Information Officer December 31, 2019

Results stemming from the annual monitoring will be presented to RIPEC and EXCOM.

TBS continues to develop accompanying / support tools. The Department will ensure the template is updated and monitor any new updates.

March 31, 2020

Page details

Date modified: