Quality Assessment of Canada Revenue Agency Internal Audit

Final Report

Corporate Audit and Evaluation Branch
June 2011

Table of Contents

• 1.0 Introduction
• 2.0 Objective and scope
• 3.0 Methodology
• 4.0 Findings
  • 4.1 Taking more focused action on deficiencies identified in the Quality Review process
  • 4.2 Timeliness
  • 4.3 Auditee Satisfaction Surveys
  • 4.4 Resource Utilization
• 5.0 Quality Assessment Participants
• Appendix A - Assessment Rating
• Appendix B - Independent Validation
• Appendix C - Independent Validation Statement

1.0 Introduction

In January of 2002, the Institute of Internal Audit (IIA) published new/upgraded standards reflective of the changing role of internal audit. The Office of the Comptroller General of the Government of Canada adopted these new standards.

Standard 1312 requires that an external assessment be performed at least once every five years to obtain an objective appraisal of the activity's compliance with the IIA standards and to improve the organization's operations. The first assessment in relation to this standard was completed in September 2006. To satisfy the requirement in an economical fashion, the Internal Audit Division of the Corporate Audit and Evaluation Branch (CAEB) chose the option of a self-assessment confirmed by independent validation by a qualified independent assessor. (The other option is to have the assessment conducted completely by an independent assessor).

To continue to meet the Standard, CAEB was required to complete its next Quality Assessment by September 2011. CAEB proceeded to conduct a self-assessment with independent validation from January to March 2011.

2.0 Objective and scope

The objective of the self-assessment was to provide assurance to the Canada Revenue Agency's (CRA) Board of Management (Board) and the Management Audit and Evaluation Committee (MAEC) that the internal audit activity of CAEB is conforming to the established professional standards as published in the IIA's Professional Practices Framework.

The self-assessment addressed compliance with the IIA's International Standards for the Professional Practice of Internal Auditing, and the IIA Code of Ethics. Assessment of the internal audit activity was based on CAEB's existing policies and procedures and the results of quality assurance work performed during the 2010-2011 fiscal year. It also included monitoring the progress of actions plans established following the internal risk assessment exercise.

3.0 Methodology

The process was initiated with an internal assessment based on the IIA standards. This resulted in the identification of areas where improvements could be made. CAEB then established action plans to address these issues. Those action plans are included in this report and will be monitored through to completion.

The audit steps and criteria for the self-assessment were based on the IIA's Quality Assessment Manual (6th edition) and CAEB's own quality assurance audit programs. The Professional Practices and Corporate Services team prepared and/or utilized:

• results of quality reviews of completed audit products;
• analyses of results of informal quality reviews on audit deliverables;
• results of client satisfaction surveys;
• reviews of policies and other documentation (internal and external);
• interviews and discussion with CAEB managers and staff; and
• benchmarking where information was readily available.

The independent validators, David Rattray and Robin Sellars of BMCI Consulting Inc., were required to conduct interviews with CRA senior management and members of the Board of Management to determine their opinions regarding the internal audit function. To reduce the burden on interviewees, CAEB did not conduct separate interviews, but relied on the interview results provided by the independent validators.

The independent validation was based on the Statement of Work detailed in the attached Appendix B (Independent Validation).

This Quality Assessment report reflects the findings and recommendations of the self-assessment as well as the opinion of the independent validators (in Appendix C) regarding those findings and recommendations.

4.0 Findings

Overall, CRA's internal audit activity "generally conforms" to the Professional Standards as set by the IIA. In the IIA lexicon, "generally conforms" is the highest rating achievable. However, there are four areas in which CAEB has rated itself as "partially conforms":

4.1 Taking more focused action on deficiencies identified in the Quality Review process

Documented and anecdotal evidence suggests that certain types of deficiencies are repeatedly identified by Professional Practices during the Quality Review (QR) process. These include working papers which do not present an adequate evidence-and-logic-chain and/or do not present evidence of comprehensive supervisory challenge and review. Examples of this include interview notes or analysis documents without summaries or which do not contain conclusions in the summaries.

The QR process ensures that such deficiencies do not impair the results and related products of the internal audits but they undermine confidence in the working papers and can lead to more in-depth reviews of their quality, requiring greater effort by those performing quality reviews.

Documentation and/or tools exist to record most of the deficiencies identified during the QR process and Professional Practices has sporadically taken steps to address recurring deficiencies in the past but only in response to a strong concentration of such deficiencies within a short period of time. A comprehensive analysis of the use of the tools and the root cause of the deficiencies has not yet been performed. In-depth quality assurance reviews have just been started which is the first step in addressing this issue.

Improved quality before the reviews would increase efficiency and timeliness because the projects would not require resources and time to take remedial action after the quality review.

Recommendation

Professional Practices should complete the quality assurance reviews currently in process and perform a detailed analysis of the results, including root cause identification, and determine actions needed to address immediate concerns as well a process going forward to more systematically identify and resolve deficiencies as they arise.

Action plan

Professional Practices will complete the quality assurance reviews already underway, review the use and usefulness of existing processes and tools, and make recommendations to the Chief Audit Executive (CAE) and the Internal Audit Division Management Committee for corrective actions. This will be conducted by June 30, 2011.

4.2 Timeliness

The timeliness of internal audits has improved over the past five years. However, it is recognized that management increasingly requires current information in a rapidly changing environment and the internal audit activity must continually look to improve timeliness. Our initial focus will be the period after completion of the examination phase, before the report is presented.

Significant delays in the internal audit process can produce inefficiencies in terms of staff that are less productive or even idle. These staff members may wait for agreements and approvals at any of the decision points in the internal audit project. Such approvals may be required from a number of authorities in internal audit and among auditees and their superiors.

Delays in presenting the results of an audit, generally in the form of an audit report, can undermine the internal audit function. Senior managers and Board members may see internal audit as reporting on historical rather than current issues, diminishing its value.

The impacts of delays in reporting can be mitigated by informally advising clients of significant risks during the audit. This approach must be used cautiously to ensure that there is adequate evidence to support these preliminary conclusions. This mitigation activity can provide significant benefits in instances where the remedial action is clear and the timeliness of resolution of the issue is important.

In other cases, this mitigation approach may not be appropriate. Some decisions regarding resolution of an issue can benefit from broader senior management input and careful preparation of longer-term implementation plans. Immediate unilateral remedial action may compromise some potential benefits in these instances.

Efforts have been made to address this issue over the last few years by limiting the scope of audits to focus on the areas where the most value can be produced in terms of addressing risks and producing benefits. This narrower scope tends to reduce the number of decision-makers that must be involved in approvals and consequently improve timeliness. We have also developed five-year plans to cover key topics such as IT Security and Procurement which will reduce the work needed to commence audits since much of the planning is done in the development of the five-year plan and can be used by each audit. Other areas will be considered for the same type of five-year plan.

Recommendation

CAEB should continue to seek ways to improve the timeliness of internal audits and internal audit reporting.

Action plan

CAEB will further improve timeliness in some audits by focusing exclusively on the decision-making process and the information provided for those decisions. The CAEB Business Plan for 2011-2012 will include at least one audit that will support a major change initiative in this manner. Further opportunities to apply this approach will be considered for the 2012-2013 fiscal year. In addition, CAEB will continue to explore the use of other audit products, such as business advisory services and health checks that can require less rigor and therefore less time, but still add value.

4.3 Auditee Satisfaction Surveys

For many years, the CAEB audit process has included asking auditee senior managers to provide feedback through a survey questionnaire to be completed at the end of the audit, after the final report has been formally delivered to management. Respondents are asked to rate their experience and the audit team in a number of areas that impact on audit quality. The responses are to be sent directly to the CAE. Respondents are given the option of having their survey feedback and other comments kept confidential.

Since the CAE attends many meetings with senior managers, some feedback is provided verbally in informal discussions, causing some potential respondents to consider the survey redundant. The highlights of feedback from senior managers are shared by the CAE with the Branch management team or the individual managers involved in the specific audits mentioned in the feedback. In general, this feedback is not structured in a manner that is consistent with the survey feedback.

Each year, a summary of the responses is presented to the CAEB management team, and is a part of the evidence used to support the Board of Management Oversight Framework assessment of the internal audit activity. As the summary for this year was completed, we observed that not all surveys had been sent, and some were not returned nor followed-up.

As a result of staff changes, the formal process that was in place to track the distribution and follow-up of these surveys was not maintained. Also, the process to systematically analyze and take corrective actions due to survey results was not performed as often as required.

The lack of broad, structured survey feedback leaves a gap in the performance information available to management of the internal audit function and to those who wish to judge the effectiveness of internal audit. This reduces the effectiveness of efforts to identify areas that generally require improvement and the means of producing that improvement.

Recommendation

CAEB should ensure that the existing process for the distribution, tracking, follow-up and analysis of the auditee satisfaction questionnaire is followed. This must respect the respondent's right to confidentiality when requested.

When the CAE receives feedback informally, efforts should be made to record that feedback within a structure that conforms to the survey. In addition, the auditee senior manager involved should be informed that completion of the survey would be helpful.

Action plan

As the surveys are distributed through the CAE's office, the responsibility for tracking the distribution and return of the surveys will be in CAE's office. Professional Practices will work with the CAE's office to refine the process that will be in effect for all products, beginning with the products presented at the April 2011 MAEC meeting. In addition, Professional Practices will summarize and analyze responses at mid-year and year-end and provide a report to the CAE.

The process to be developed will include a means to collect informal feedback received by the CAE.

4.4 Resource Utilization

Some occasions have arisen where internal auditors experience gaps in audit assignments or other workloads. While this auditor availability is shared within the Branch, which in most cases results in a work assignment of some nature, there have been some occasions where these gaps could have been addressed on a timelier basis. This does not extend to Internal Audit Project Leaders and their superiors. Some of this unassigned time is inevitable as the timing of some internal audit activity is not controllable. Also, there may be delays in having a qualified Internal Audit Project Leader available to provide guidance.

This unassigned time reduces the efficiency of the internal audit activity and may cause projects to be completed later than would otherwise have occurred. The Internal Audit Division management team recently met to determine specific auditor assignments for the coming year, address existing auditor availability and further the process of identifying and addressing the factors which lead to unassigned time.

Recommendation

CAEB should continue to seek opportunities to reduce this amount of unassigned time.

Action plan

The Internal Audit Division will use the detailed resource planning exercise completed in March 2011 to more systematically monitor the availability of audit resources. While discussions can be held at any time between managers, at a minimum, the issue of auditor availability will be a standing item on the second Internal Audit Division Management Committee meeting of each month. Managers will be expected to project availability forward three months. This practice will start in April 2011.

5.0 Quality Assessment Participants

CAEB Self-Assessment:

Patricia MacDonald
Chief Audit Executive

Annie Boudreau
Director – Internal Audit Corporate Functions

Gita Bhatt
Director – Internal Audit Tax Operations

Maura Butko
Director – Professional Practices and Corporate Services

Robin Fullarton
Manager – Professional Practices Unit

John Arnold
Project Leader – Professional Practices Unit

Independent Validation:

David Rattray
Robin Sellar

Appendix A - Assessment Rating

Guidelines for Evaluation of Conformity to the Standards and Code of Ethics:

• When evaluating conformance with the International Professional Practices Framework (IPPF), [Footnote 4] carefully read the Standards and relevant interpretations included and consider only the Standards and corresponding interpretations, not the ideal situations. Please note that only the requirements included in the Standards and interpretations are mandatory.
• Consider each individual standard, including the relevant Implementation Standards (which give additional guidance on assurance and consulting services), and conclude the degree of conformance by the activity to each individual standard using the Key Conformance Criteria and examples of evidence for guidance.
• Any of the Key Conformance Criteria not achieved would strongly suggest a rating of "does not conform" or "partially conforms" for that individual standard.
• Consider each section of the Standards with numbers ending in "00" (for example, Standard 1200: Proficiency and Due Professional Care or Standard 2300: Performing the Engagement), and conclude the degree of conformance by the activity to each section taken as a whole, based on conclusions reached for the related individual Standards in the section and on other relevant observations made during the quality assessment. If all underlying Standards are "does not conform," then the overall rating is "does not conform." Otherwise, the team must make a judgment based on the number of "does not conform" and the specific conditions present as to whether the overall rating is "does not conform" or "partially conforms."
• On the same basis as for sections of the Standards, conclude the degree of conformance by the activity to the major categories of the Standards (Attribute and Performance).
• Consider the four principles and related rules of conduct in the Code of Ethics and conclude whether or not the activity's management and staff uphold each of the principles and apply the related rules of conduct.
• Make an overall evaluation as to the activity's conformance with the Standards as a whole.

Definitions
GC – "Generally Conforms" means the evaluator has concluded that the relevant structures, policies, and procedures of the activity, as well as the processes by which they are applied, comply with the requirements of the individual Standard or element of the Code of Ethics in all material respects. For the sections and major categories, this means that there is general conformance to a majority of the individual Standards or elements of the Code of Ethics, and at least partial conformance to the others, within the section/category. There may be significant opportunities for improvement, but these must not represent situations where the activity has not implemented the Standards or the Code of Ethics, has not applied them effectively, or has not achieved their stated objectives. As indicated above, general conformance does not require complete/perfect conformance or the ideal situation.

PC – "Partially Conforms" means the evaluator has concluded that the activity is making good-faith efforts to comply with the requirements of the individual Standard or element of the Code of Ethics, section, or major category, but falls short of achieving some major objectives. These will usually represent significant opportunities for improvement in effectively applying the Standards or Code of Ethics and/or achieving their objectives. Some deficiencies may be beyond the control of the activity and may result in recommendations to senior management or the Board of the organization.

DNC – "Does Not Conform" means the evaluator has concluded that the activity is not aware of, is not making good-faith efforts to comply with, or is failing to achieve many/all of the objectives of the individual Standard or element of the Code of Ethics, section, or major category. These deficiencies will usually have a significant negative impact on the activity's effectiveness and its potential to add value to the organization. These may also represent significant opportunities for improvement, including actions by senior management or the Board.

Often, the most difficult evaluation is the distinction between "general" and "partial." It is a judgment call keeping in mind the definition of "general conformance" above. Carefully read the Standardto determine if basic conformance exists. The existence of opportunities for improvement or better alternatives do not reduce a "generally conforms" rating.

Appendix B - Independent Validation

The principal elements of the Independent Validation role included:

• review the self-study and other advance materials and note items for follow-up on site visits;
• identify potential best practices and opportunities for improvement;
• on-site, review the documentation of the self assessment;
• perform limited tests, by reference to source documents, the self assessment work and the validity of the information submitted;
• interview Internal Auditors, Project Leaders, Account Managers and Directors to obtain their opinions;
• review the evaluation summary regarding conformity to the Standards and discuss the basis of conclusions with the self-assessment team and the CAE;
• conduct brief interviews with the head of the Agency, the Chair of the Audit Committee, other appropriate senior officials and two or three clients of Corporate Audit services;
• review the CAE's communication with the Audit Committee on the results of the self-assessment;
• prepare a memorandum to serve as the basis of a closing briefing with the self assessment team and the CAE; and
• complete the formal validation report for transmittal to the Audit Committee by September 2011.

Appendix C - Independent Validation Statement

Independent Validation Statement

We, David Rattray, FCGA, CIA, and Robin Sellar, CA, CIA of Ottawa, Ontario were engaged to conduct an independent validation of the Canada Revenue Agency's (CRA) Internal Audit Division's self-assessment. The primary objective of the validation was to verify the assertions made in the attached quality self-assessment report concerning adequate fulfillment of the Agency's basic expectations of the Internal Audit activity and its conformity to The Institute of Internal Auditors' (the IIA's) International Standards for the Professional Practice of Internal Auditing (Standards). Other matters that might have been covered in a full independent assessment, such as an in-depth analysis of successful practices, governance, consulting services, and use of advanced technology, were excluded from the scope of this independent validation by agreement with the Chief Audit Executive (CAE).

In acting as validators, we are fully independent of the organization and have the necessary knowledge and skills to undertake this engagement. The validation, conducted during the period of January to March 2011, consisted primarily of a review and testing of the procedures and results of the self-assessment. In addition, we reviewed CRA supplied documentation, conducted audit working paper file reviews, carried out internal audit staff focus groups and conducted structured interviews with the Commissioner, the Chief Financial Officer, the Chief Informatics Officer, the Assistant Commissioner of the Pacific Region, the Assistant Commissioner of the Compliance Programs Branch, and the Chief Audit Executive as well as Board of Management Audit Committee Members.

We concur fully with the Internal Audit (IA) activity's conclusions in the self-assessment report attached.

Implementation of all the action plans contained in the self-assessment report will improve the effectiveness and enhance the value of the IA activity and ensure its full conformity to the Standards

David Rattray, FCGA, CIA
Independent Validator
BMCI Consulting Inc.

June 3, 2011

Robin Sellar, CA, CIA
Independent Validator
BMCI Consulting Inc.

June 3, 2011

Footnotes