Language selection

Government of Canada / Gouvernement du Canada

Search

Audit of External Reporting on Performance: chapter 6

Previous page 
Table of contents 

Annex 1 - Audit Methodology and Criteria

Methodology

The audit was carried out using a combination of interviews, testing and extensive document review. Although the AEB included some information obtained from evaluation work in the audit, it did not duplicate areas routinely covered by evaluations.

The audit was conducted in the National Capital Region, with documentation provided electronically from across the country. Interviews in the regions were carried out by teleconference or email.

In order to assess the quality of the data and information reported externally in the Departmental Performance Report (DPR), the audit selected for examination a purposeful sample of 18 expected results reported in the 2012-2013 DPR from a total of 34 programs and sub-programs, comprising 48 expected results and 62 related performance indicators. A minimum of two items from each branch focussed on in the audit scope were selected. Each item was analyzed according to the quality attributes listed in the following table:

Methodology
Information and Data Quality Attributes Description
Robustness The data sources are credible, consistent and verifiable. Performance reports should disclose the basis on which the information was prepared and the limitations that should apply to its use. (For example, a robust data collection system would result in data and information that are consistent and produce repeatable results.)
Reliability The key results are reported against expectations and aligned within the PAA, PMF, DSDS and CESI. Performance reports should disclose the basis on which the information was prepared and the limitations that should apply to its use, including changes from previous reports to the sources and nature of the information. (For example, a reported actual result is reliable if it is consistent with previous results reported and with the expected result.)
Accuracy The reporting is based on quantitative and qualitative information that is fairly presented, based on the best judgement of those reporting. (For example, to be accurate, a result would include all the relevant data in its calculation.)
Timeliness The information and data communicated to the decision makers is timely, by covering an appropriate time frame and being the most recent reasonably available. (For example, data covering the summer period would not be appropriate to an expected result referring to the winter season.)

For one expected result of each program or sub-program sampled, the audit procedures focused on one reported performance indicator and the target and actual results reported against that indicator. The audit did not examine cost information supplied for the related program or sub-program, as the costs were not specifically attached to each performance indicator in the DPR.

The AEB considered two or more incidences of samples not meeting a quality attribute to be material. For example, where two or more sample items were found to be inaccurate, AEB considered this to be material.

Criteria
Audit CriteriaFootnote5 Sources of Criteria Status
a DPR
Met / Not Met		 Status
CESI
Met / Not Met

Environment and Climate Change Canada should have proper governance in place that ensures:

  • Proper coordination and control of the various performance measurement frameworks in the Department;
  • Integration of relevant data and information;
  • A formal system of lessons learned and follow-up.
  1. TB Policy on Management, Resources and Results Structures (MRRS), Section 6.1.1.3
  2. Office of the Comptroller General, 2013: Horizontal Internal Audit of Compliance with the Policy on Management, Resources and Results Structures (MRRS), Appendix B
 Partially Met Footnote6 Met
Performance information is supported by accurate and robustFootnote7 data and explanations, ensuring the credibility and balance of the reported performance information. Performance reports disclose the basis on which the information was prepared and the limitations that should apply to its use. Performance information is supported by timely data.
  1. Office of the Auditor General of Canada, Chapter 6, 2002: A Model for Rating Departmental Performance Reports; Chapter 5, 2005: Rating Selected Departmental Performance Reports; Chapter 1, 2013: Status Report on Evaluating the Effectiveness of Programs.
  2. Canadian Comprehensive Auditing Foundation, 2002 - Reporting Principles: Taking Public Performance Reporting to a New Level.
  3. TB Policy on Management, Resources and Results Structures (MRRS).
  4. Library of Parliament, 2012: A Guide to the Estimates.
  5. Cabinet Directive on Streamlining Regulation, 2007.
  6. National Audit Office, 2007: Performance Frameworks and Board Reporting.
  7. Province of British Columbia, 2003: Performance Reporting Principles For the British Columbia Public Sector - Principles Endorsed by Government, the Select Standing Committee on Public Accounts and the Auditor General of British Columbia.
 Partially MetFootnote8 Met
The information should be reliable in terms of key results reported against expectations, aligned within the PAA, PMF, DSDS and CESI parameters. Performance reports should disclose the basis on which the information was prepared and the limitations that should apply to its use, including changes from previous reports to the sources and nature of the information.
  1. Office of the Auditor General of Canada, Chapter 6, 2002: A Model for Rating Departmental Performance Reports; Chapter 5, 2005: Rating Selected Departmental Performance Reports; Chapter 1, 2013: Status Report on Evaluating the Effectiveness of Programs.
  2. Canadian Comprehensive Auditing Foundation, 2002 - Reporting Principles: Taking Public Performance Reporting to a New Level.
  3. TB Policy on Management, Resources and Results Structures (MRRS).
  4. Library of Parliament, 2012: A Guide to the Estimates.
  5. Cabinet Directive on Streamlining Regulation, 2007.
  6. National Audit Office, 2007: Performance Frameworks and Board Reporting.
  7. Province of British Columbia, 2003: Performance Reporting Principles For the British Columbia Public Sector - Principles Endorsed by Government, the Select Standing Committee on Public Accounts and the Auditor General of British Columbia.
 Partially MetFootnote9 Met

Key Dates

Opening conference (launch memo)
February 2014

Audit plan completed
July 2014

External Audit Advisory Committee tabling of draft report
March 2015

External Audit Advisory Committee tabling of revised report
June 2015

Deputy Minister approval
December 2015

Previous page 
Table of contents 

Page details

Date modified: