Internal Audit - Tax and Benefits Operations results information

Final Report

Audit, Evaluation, and Risk Branch

January 2020

Executive summary

The Canada Revenue Agency (CRA) produces and publishes a series of corporate reports, which are made available to the public on an annual basis. The corporate reports that are released externally provide CRA senior management with the opportunity to share the organization’s objectives, progress, and achievements, and respond to reporting requirements and guidance as set out in Treasury Board Policy and the CRA Act. The readers of these corporate reports are diverse and vary from the everyday taxpayer to parliamentarians. The accuracy and reliability of the information in these reports is vital to the credibility of the CRA as an organization and to those using it for decision making purposes.

The objective of the audit was to provide assurance that tax and benefits operations results information reported to external parties and used within the CRA is accurate, reliable, and consistent.

This audit focused on a sample of results that are annually provided to external stakeholders and included in the Departmental Results Report and the Commissioner’s Annual Report to the Governments of the Provinces and Territories.

The audit team found that the Service, Innovation and Integration Branch provides oversight for the Departmental Results Report and Commissioner’s Annual Report to the Governments of the Provinces and Territories. The program branches maintain the program expertise and are expected to ensure that the data they submit to the Service, Innovation and Integration Branch is relevant and accurate. As a result, no single branch in the CRA has the authority or the mandate to meaningfully challenge how the data reported by the program branches is calculated for the purpose of external reporting.

Overall, the audit team found that the external reporting processing has several strengths. For example, reporting roles and responsibilities are clearly defined for the Service, Innovation and Integration Branch and the program branches. Furthermore, policies, procedures and processes are in place to guide the coordination of external reporting. The audit team also found that established monitoring programs have been implemented for most of the programs reviewed. These monitoring programs have well-documented review procedures, which are consistently applied in order to ensure that the data used for results reporting is complete.

The audit team did however note opportunities to improve the transparency in external results reporting. Due to differences in the various systems, the methodology used to calculate the results from each program differed. Important information concerning the differences were not clearly disclosed in the sample of service standards reviewed. Consequently, stakeholders might not have precise or adequate information required to make informed decisions.

Summary of recommendations

• The CRA, led by the Service, Innovation and Integration Branch, should identify and implement a process to ensure that a robust review function of externally reported results exists, which includes monitoring, oversight, and a dispute resolution mechanism.
• The Assessment, Benefit, and Service Branch should complete the development of their T1 monitoring process, ensure that monitoring processes are consistently applied, and that the results are documented.
• Programs should fully disclose any defining details within the reports concerning their service standard calculation methods to avoid stakeholders or readers from misinterpreting results.
• Service standard evaluation tools should consistently include and define criteria concerning transparency.

Management response

The Assessment, Benefit, and Service Branch and the Service, Innovation and Integration Branch agree with the recommendations in this report and have developed related action plans.

The Assessment, Benefit, and Service Branch has documented and finalized the development of their T1 monitoring process to ensure that monitoring processes are consistently applied.

The Service, Innovation and Integration Branch will complete a full assessment of strategies to improve monitoring and oversight and will present their suggestions to AMC by November 2020. The plan will include a robust review function including attention towards full disclosure of details concerning service standards reported.

The Audit, Evaluation, and Risk Branch has determined that the action plans appear reasonable to address the recommendations.

1. Introduction

The Canada Revenue Agency (CRA) produces and publishes a series of corporate reports. These reports provide CRA senior management with the opportunity to communicate core responsibilities, financial information, and service standard results. The readers of these corporate reports are diverse and include taxpayers, parliamentarians, media, and other interested parties. The accuracy and reliability of the information in these reports is vital for decision making purposes and for maintaining the credibility of the CRA.

The Service, Innovation and Integration Branch oversees, coordinates, and publishes the CRA’s corporate reports, which include departmental performance and service standards results. The branch’s mandate includes ensuring rigour in the CRA’s reporting to Parliament. The Policy, Planning, Partnerships and Reporting Directorate, within the Service, Innovation and Integration Branch is the centre of expertise that oversees the management of the CRA’s governance and accountability activities and ensures that the activities respond to the strategic vision of the Minister, the Commissioner, and the Board of Management.

The main corporate plans and reports that are provided annually to external stakeholders are:

• the Departmental Results Report, which includes the CRA’s audited financial statements
• the Commissioner’s Annual Report to the Governments of the Provinces and Territories
• the Summary of the Corporate Business Plan
• the Departmental Plan

These reports follow the policy requirements set out by the Treasury Board Secretariat and the CRA Act, and respond to the Taxpayer Bill of Rights which states that taxpayers have the right to expect the CRA to publish their service standards and report annually.

These reports contain information related to tax and benefits operational results, including:

• service standards
• other performance results
• financial information

Service standards state the level of performance that citizens and businesses expect from the CRA. These standards are introduced in the Summary of the Corporate Business Plan and the Departmental Plan and include the level of service Canadians can expect under normal circumstances. The Departmental Results Report communicates the actual results achieved against the targets set out in the Departmental Plan and the Summary of the Corporate Business Plan.

Other tax and benefits performance results are communicated in the Departmental Results Report which include key results accomplished during the year in connection with the CRA’s commitment to Canadians for transparency, management accountability, and citizen-focused service.

Financial information is reported in the annual audited financial statements of the CRA. The Finance and Administration Branch coordinates and publishes the financial statements, which are independently audited by the Office of the Auditor General of Canada. Other financial information is included in the Commissioner’s Annual Report to the Governments of the Provinces and Territories.

In addition to the corporate reports that are provided to external stakeholders, the CRA prepares internal reports to track performance and to help management make decisions. These include quarterly performance reports presented by the Service, Innovation and Integration Branch to CRA management and to the Board of Management, and internal reports maintained by individual program areas on tax and benefits operational results.

Complete and accurate information, which can be reconciled and supported for corporate reporting, is vital to the CRA’s ability to make appropriate and informed decisions to manage programs effectively and to inform the public of its level of performance.

This internal audit was performed in order to provide assurance that the CRA is reporting accurate, reliable and consistent information to all internal and external stakeholders.

This audit is important because the CRA has made several commitments regarding service delivery. Many internal and external users rely on the service standard results to react to resource demands and to ensure that all Canadians are receiving the best possible service.

2. Focus of the audit

This internal audit was included in the Board of Management (Board) approved 2016-2019 Risk-Based Audit and Evaluation Plan. The Assignment Planning Memorandum was approved by the Management Audit and Evaluation Committee on June 26, 2018.

2.1. Objective

The objective of the audit was to provide the Commissioner, CRA management, and the Board with assurance that tax and benefits operations results information reported to external parties and used within the CRA is accurate, reliable, and consistent.

2.2. Scope

This audit was conducted in the Headquarters area. The sample of service standards and performance results examined in the Departmental Results Report and Commissioner’s Annual Report to the Governments of the Provinces and Territories were from the 2016 to 2017 reporting period. Given that the audit planning phase occurred at the end of 2017, the 2017-2018 performance results were not available.

The service standards and performance results examined can be found in Appendix B: Service standards and data examined for the audit. The T1, Benefits, and Goods and Services Tax /Harmonized Sales Tax Refund Integrity Program were selected as they represent a broad cross section of activities within CRA and during audit planning were found to have sufficient risks to warrant examination.

2.3. Audit criteria and methodology

The audit criteria and methodology can be found in Appendix A.

The planning phase of the audit took place between December 2017 and May 2018, which included interviews with the Service, Innovation and Integration Branch and other affected program branches. The examination phase followed from June 2018 to February 2019.

During the audit, the opportunity to utilize the skill set of the Audit, Evaluation, and Risk Branch’s data analytics team arose. The data analytics team unsuccessfully attempted to reconstruct the results sampled. Instead, they used the CRA data mart to independently verify the sample’s accuracy.

The audit was conducted in accordance with the International Standards for the Professional Practice of Internal Auditing, as supported by the results of the quality assurance and improvement program.

3. Findings, recommendations, and action plans

The Service, Innovation and Integration Branch and the Assessment, Benefit and Service Branch agree with the recommendations in this report and have developed related action plans. The Audit, Evaluation, and Risk Branch has determined that the action plans appear reasonable to address the recommendations.

3.1. Governance

3.1.1 Policies, procedures and processes are in place to guide and support the coordination and reporting of tax and benefits operations results. However, there are no formal mechanisms in place to resolve potential disputes concerning the integrity of data reported in the Departmental Results Report and the Commissioner’s Annual Report to the Governments of the Provinces and Territories.

Policies, procedures, and processes are in place to guide and support the coordination and reporting of tax and benefits operations results. Guides prepared by the Service, Innovation and Integration Branch clearly state that the program branches are responsible for developing and implementing their service standards or performance measures and for reporting on them. The corporate areas of the program branches administer reporting.

The Service, Innovation and Integration Branch works collaboratively with the branches at multiple stages throughout the reporting cycle. In the event of a disagreement between the Service, Innovation and Integration Branch and the program branch, the branches would work directly to share and discuss inconsistent information, request clarification, and confirm results.

The audit team found that there are no formal dispute resolution mechanisms in place between the Service, Innovation and Integration Branch and the program branches or between the program delivery areas and their corporate areas within the program branches. A formal dispute resolution involves an impartial third party. The impartial third party helps those involved, define the issues and then guides the process to an acceptable solution. Therefore, there is no independent authority in this context to resolve disputes should they arise.

Following the completion of the audit, the audit team learned that the Chief Data Officer, through the Agency Data Clearing house, now formally reviews methodologies and results with reporting branches for the Commissioner’s Annual Report to the Provinces and Territories and the Departmental Results Report.

The recommendation for 3.1.1 is covered in Recommendation 3.

3.1.2 Roles and responsibilities related to the coordination and reporting of tax and benefits operation results are appropriate, clearly communicated, and understood.

The audit team found that roles and responsibilities are clearly defined for the purposes of reporting for both the Service, Innovation and Integration Branch and the program branches. The Service, Innovation and Integration Branch is responsible for preparing the reports on service standards and performance measures and delivering them to internal and external stakeholders. The program branches, on the other hand, are responsible for developing and implementing their service standards and performance measures and reporting them to the Service, Innovation and Integration Branch.

The program branches consult with the Service, Innovation and Integration Branch when establishing and reporting on service standards and their results. The Service, Innovation and Integration Branch also acts as a consultant for the program branches when they report, develop, and implement service standards, and it challenges those branches to ensure that their service standards are relevant, measurable, and meaningful.

The Service, Innovation and Integration Branch clearly communicates its expectations to the branches and defines their roles and responsibilities for reporting.

3.2. Monitoring and Oversight

3.2.1 The Service, Innovation and Integration Branch provides oversight for the Departmental Results Report and the Commissioner’s Annual Report to the Governments of the Provinces and Territories. The Assessment, Benefit, and Service Branch provides additional oversight for the results they provide for the Departmental Results Report.

Oversight is provided by the Service, Innovation and Integration Branch for Departmental Results Report and Commissioner’s Annual Report to the Governments of the Provinces and Territories reporting and also by the Horizontal Integration Directorate in the Assessment, Benefit, and Service Branch for their respective Departmental Results Report reporting.

The oversight primarily consists of checking for variances in amounts reported from year to year. When there is a variance that exceeds 5%, the Service, Innovation and Integration Branch ensures that the program branch provides a rationale. In addition to variance reviews, the Service, Innovation and Integration Branch also questions the branches about the narrative accompanying the quantitative figures when it seems to be inconsistent with the results.

The Service, Innovation and Integration Branch relies on the validity of the information provided by the program branches. Staff in the program branches have the program expertise and are expected to ensure that the data they produce and submit to the Service, Innovation and Integration Branch are relevant and accurate.

Similarly to the Service, Innovation and Integration Branch, Horizontal Integration Directorate of the Assessment, Benefit, and Service Branch monitors results by looking for consistency in the data from prior years. The Horizontal Integration Directorate receives a description of the elements in a formula and the source of the information for a calculated number from the program directorates. In this case, data submitted by the Assessment, Benefit, and Service Branch program directorates are verified for accuracy only by the program directorates themselves, through monitoring activities such as quality assurance.

3.2.2 The review procedures for monitoring in some programs are consistently applied and documented. However, the review procedures for the monitoring of T1 returns processing are under development and as a result, are not consistently applied nor fully documented.

In the Financial Performance Reporting Section of the International, Large Business and Investigations Branch, the monitoring activities are documented and consistently applied and the results are communicated for inclusion in corporate reports and for the use of stakeholders. Throughout the year, the Financial Performance Reporting Section conducts reviews to ensure that the coding for the Goods and Services Tax /Harmonized Sales Tax Refund Integrity Program cases are accurate. In doing so, it is able to verify the coding for approximately 51% of the tax earned by audit with an error rate of half a percent.

The Benefit Programs Directorate in the Assessment, Benefit, and Services Branch has a formal Quality Assurance process to monitor the accuracy of the service standards set for processing benefit applications. The quality assurance process is documented, it is conducted with an identified purpose, and the findings are reported. The directorate conducts this type of monitoring for both the accuracy and timeliness of service standards. Its monitoring methodology is detailed and well-explained in the quality assurance reports.

The Individual Returns Directorate of the Assessment, Benefit, and Service Branch monitors its processing of T1 returns. However, a complete monitoring process is still under development as a result of T1 System Redesign. Review procedures are inconsistently applied as the queries written by the Information Technology Branch for the Individual Returns Directorate varied and produced unreliable results. It has been an iterative process for the Individual Returns Directorate and the Information Technology Branch to produce accurate results during 2017 to 2018. Given that this process is incomplete, monitoring procedures are inconsistently applied and documentation is incomplete.

Recommendation 1

The Assessment, Benefit, and Service Branch should complete the development of its T1 monitoring process and ensure that review procedures are documented and consistently applied.

Action Plan 1

The Assessment, Benefit, and Service Branch has completed the development of its T1 monitoring and review process and the process has been in place since the 2018-2019 reporting period. ABSB intends to ensure that the process continues to be consistently applied.

Target date: Completed

Recommendation 2

The CRA, led by the Service, Innovation and Integration Branch, should ensure that all programs have monitoring processes for externally reported results. The monitoring process should include documented review procedures and be consistently applied. 

Action Plan 2

This recommendation has been addressed in Action Plan 3.

Target date: November 2020

3.2.3 There is no formal independent monitoring of the calculation methodologies and the results reported in the Departmental Results Report or the Commissioner’s Annual Report to the Governments of the Provinces and Territories.

Based on existing roles and responsibilities, program branches are responsible for monitoring the data that they report on their own performance. Each program reviewed during this audit has its own unique monitoring program in place to verify their data results and presents the monitoring results in different formats.

Following the completion of the audit, the audit team learned that the Chief Data Officer, through the Agency Data Clearing House, reviews data relating to taxpayer information prior to publication and release. In addition, the Performance Measurement Centre of Expertise supports the monitoring and oversight of indicator methodologies and results.

Despite this, no single branch in the CRA has the authority to meaningfully challenge how the data reported by the program branches are calculated nor the mandate to conduct formal monitoring. As a result, there may be a lack of knowledge and expertise regarding how to monitor these results outside of the program. This might pose a risk to the completeness, consistency, impartiality and accuracy of externally reported results.

Recommendation 3

The CRA, led by the Service, Innovation and Integration Branch, should identify and implement a process to ensure that a robust review function of externally reported results exists, which includes monitoring, oversight, and a dispute resolution mechanism.

This recommendation also addresses the findings from 3.1.1.

Action Plan 3

Since the launch of this audit in September 2017, the Agency has improved monitoring and oversight activities. The appointment of the CRA’s first Chief Data Officer (CDO) in December 2017 resulted in the subsequent formation of the Agency Data Clearing House. Following this, the Performance Measurement Centre of Expertise leads an annual performance measurement review that involves a targeted examination of indicators and associated methodologies to best demonstrate the performance of the Agency. Moving forward, SIIB is committed to ensuring that results information used for public reporting is addressed in the Agency Data and Information Strategy. The Strategy, scheduled to be submitted to AMC for endorsement in March 2020, will take a client-centric approach to collecting and cleaning information once, to enable more robust and efficient results reporting. Concurrently, SIIB will consult with stakeholders to identify additional opportunities to review and strengthen monitoring and oversight processes and systems for results data. This work will involve consultations with all branches and will also clarify SIIB’s authority in regards to the determination of performance indicators and results. Following the approval of the Agency Data and Information Strategy, SIIB commits to recommending action to AMC that will include monitoring and oversight by November 2020. SIIB will ensure that the AMC-approved actions are implemented within identified timeframes.

Target date: Phase 1 – Present Agency Data and Information Strategy to AMC in March 2020 and recommend action to AMC by November 2020. Phase 2 – date will be determined following AMC-approved action plans

3.3. Reporting

3.3.1 The accuracy and completeness of the results reported could not be fully confirmed however any variances in recreating the data were either explainable or within acceptable tolerances.

The internal audit team attempted to verify the accuracy of specific externally reported results.

This included the timeliness and accuracy service standards for T1 processing and Benefits processing along with the figures from the Goods and Services Tax /Harmonized Sales Tax Refund Integrity Program table within the Commissioner’s Annual Report to the Governments of the Provinces and Territories.

Due to factors such as calculation methodologies that could not be recreated in the present and a lack of detail in data extraction procedures, the audit team was initially unable to replicate the data used to calculate the results reported.

Further attempts were made to independently recreate the results by using the information from the CRA’s data warehouse.

Although the independent results still contained some differences when compared to the amounts provided by the programs, the variances were either explainable or within the range of tolerance as set out by the respective service standards.

3.3.2 The approaches to calculating externally reported service standard measures vary considerably, employing exclusions of some transactions, sampling, and estimating. These inconsistencies were not fully communicated to external stakeholders.

The audit team found that as a result of differences in the various systems, the methodology used to calculate the results from each program differed. The use of averages, estimates, samples, or exclusions of subpopulations were employed in order to overcome these challenges. The audit team also observed that important information concerning certain differences was not disclosed to external stakeholders.

The following are some examples noted during the audit of different approaches that were not fully disclosed.

• The Assessment, Benefit, and Service Branch’s benefit applications processing timeliness service standard measurement was based on 2 one-month samples during the fiscal year. The accuracy service standard was based on a sample taken during a 2-day period during the year. Neither of these sampling methods was disclosed externally.
• In 2016 to 2017, the T1 processing timeliness measure for paper returns was calculated based on estimates and batch averages of all returns filed. As a result of system changes, this service standard is now calculated using all returns filed on time, but excludes late filed returns and returns with filing deadlines after April 30 (for example, self-employed, deceased, and some non-residents). The current exclusions have not been communicated externally.
• When the Assessment, Benefit and Services Branch reports on T1 adjustment service standards, complex adjustments are not included. The service standard clearly indicates that the figures address only straightforward adjustments. However, it does not explain the exact nature of the excluded subpopulation of adjustments.
• The Goods and Services Tax /Harmonized Sales Tax Returns Processing Section reports on only Goods and Services Tax 34 returns for its timeliness service standard. The Goods and Services Tax 34 returns relate to common business activity, and they account for the majority of Goods and Services Tax /Harmonized Sales Tax returns. The other types of Goods Services Tax returns are excluded from the timeliness service standard, and this is not externally disclosed.
• The programs reviewed handle taxpayer-controlled delays differently for the purpose of counting the number of days required for processing. For example, the Benefits Processing Division restarts the timeliness service standard count once the need for applicant contact, such as waiting for additional documents, is resolved. The Goods and Services Tax /Harmonized Sales Tax Returns Processing Section includes the duration of all taxpayer-controlled delays within its count. These variances were undisclosed.

Without full disclosure of these differences, stakeholders do not always have the precise information needed to make informed decisions.

Recommendation 4

Programs should fully disclose any defining details concerning their service standard calculation methods to avoid stakeholders or readers from misinterpreting results.

Action Plan 4

Pursuant to the CRA’s vision of being trusted, fair, and helpful by putting people first, program branches will review the disclosed methodologies for new service standards prior to their publication, to ensure that the externally reported inclusions and exclusions communicate meaningful details of the service standards to taxpayers. SIIB will put in place a process to review details for publication to ensure that they are accurate and transparent prior to the next annual Performance Measurement Review. SIIB’s Agency Data Clearing House and Performance Measurement Centre of Expertise will support the branches as required.

Target date: May 2020

3.3.3 The Assessment, Benefit, and Service Branch’s service standard evaluation tool does not adequately define transparency.

The audit team noted that both the Service, Innovation and Integration Branch and the Assessment, Benefit, and Service Branch used service standard review tools. The Assessment, Benefit and Service Branch used a tool required by Treasury Board Secretariat for the Management Accountability Framework assessment process, while the Service Innovation and Integration Branch developed a Performance Measurement Assessment Tool. The two tools are designed to measure the strength and effectiveness of service standards. Although there are some differences in focus regarding what the review tools are evaluating, both have some commonalities. For example, they both review consistency, measurability, and achievability. The Service, Innovation and Integration Branch’s tool addresses transparency with a clear definition, while the Treasury Board Secretariat tool used by the Assessment, Benefit, and Service Branch did not.

Recommendation 5

The CRA, led by the Service, Innovation and Integration Branch, should ensure that service standard evaluation tools used by program branches include and define criteria concerning transparency.

Action Plan 5

SIIB developed a Performance Measurement Assessment Tool (PMAT) that provides a detailed definition of transparency and other key terms using a hover-over function. The PMAT has been made available and accessible on InfoZone for program branches to use. SIIB will immediately communicate to program Branches the availability of the tool and the importance of its use in assessing new and existing service standards.

Target date: December 2019

4. Conclusion

The objective of this audit was to provide the CRA’s Commissioner, management, and the Board of Management with assurance that tax and benefits operations results information reported in the Departmental Results Report and Commissioner’s Annual Report to the Governments of the Provinces and Territories to external parties and used within the CRA is accurate, reliable, and consistent.

The audit was unable to conclude definitively on the accuracy or reliability of the results reported. However, there was sufficient evidence to determine that any variances in the reporting were within acceptable tolerance levels as defined by the service standards.

The audit team noted that improvements could be made regarding disclosure of the methodologies used for data capture and for reporting of information in the external reports.

The audit further identified the requirement for an independent authority to provide monitoring and oversight of the measuring and reporting of service standards, including a dispute resolution process, in order to reduce the risk of inaccuracies.

The branches agreed with the recommendations and developed action plans to address the issues. The Audit, Evaluation, and Risk Branch has determined that they appear reasonable to address the recommendations.

5. Acknowledgements

In closing, the Audit, Evaluation, and Risk Branch would like to acknowledge, recognize, and thank the Service, Innovation and Integration Branch; the Assessment, Benefit and Service Branch; and the Compliance Programs Branch for the time dedicated and the information provided during the course of this engagement.

6. Appendices

Appendix A: Audit criteria and methodology

Lines of enquiry

Based on the Audit, Evaluation, and Risk Branch’s risk assessment, the following lines of enquiry were identified:

Methodology

The methodology for examination included the following:

• review and analysis of reports, policies, methodologies, and timeline documentation
• review and analysis of files and data supporting accuracy and completeness
• review and analysis of files and data supporting quality assurance, monitoring, and reconciliation activities
• site visits and interviews with management and staff
• testing of selected controls through documentation reviews
• walk-throughs of processes and procedures

Appendix B: Service standards and data examined for the audit