Audit of the Expenditure Management Component System

EMS End-Users

The EMS provides a central focus for the functions that support and strengthen the Secretariat's role as Budget Office, including responsibility for the government's broader expenditure management system.

Within the EMS, the Expenditure Strategies and Estimates Division supports the budget office role in the following areas:

• Provision of a government-wide view of expenditure management data to the annual budget process;
• Provision of advice on reserve management;
• Preparation of the government's Main Estimates and Supplementary Estimates for tabling in the House of Commons; and
• Management of the process of obtaining Governor General Special Warrants.

The Policy and Systems group of the Expenditure Strategies and Estimates Division was the key representative of the EMS end-users for the EMC project. The group served as the mechanism to review the project work, provided qualitative information about issues, and participated in the discussions and approval of any changes to the business functionality requirements. The Policy and Systems group was responsible for the development of training material and provided training to the EMS end-users of the EMC system.

System Developers

The Application Development and Data and Database Administration Division of IT-CSS was the group responsible for working with the EMS on the design and implementation of the EMC system. Members of this division were experienced in supporting and maintaining the previous legacy systems and therefore had relevant expertise regarding the business requirements of the EMS. Their knowledge and skills facilitated the design of the application functionality and reports.

Continued support of the EMC system remains the responsibility of IT-CSS.

Scope Exclusions

The audit excluded the following:

3.1.1 Use of standardized methodologies

The audit team expected that the project would use standard methodologies for the IT industry to guide the EMC system implementation to ensure that:

• The project met the requirements of the EMS end-user;
• Approvals were obtained for key decisions and deliverables; and
• The system was configured according to best practices and in conformance with internal architecture standards.

The Accelerated SAP (ASAP) methodology, which is the standard methodology, developed for implementing the SAP system, was used. It assisted the project team in implementing SAP effectively because it ensured that key development steps were followed and that the EMS end-users were involved. This methodology included the use of templates that helped maintain an end-user focus and promoted frequent communication between the EMS end-user group and IT-CSS. The use of standard templates also enabled consistency, increased overall document quality, and reduced the complexity, time and cost of document creation.

Using the ASAP methodology, business requirements were validated at the beginning of the project. Although a process for change management had been defined, there were no required changes to the business requirements during the implementation.

The Secretariat's A Guide to Project Gating for IT-Enabled Projects, a gating model which maintains alignments with internal architecture standards, was used to monitor the progress of the project.

IT project management methodology was supported through the use of the IT-CSS JIRA Issue and Project Tracking Tool. This tool allowed the team to track system bugs, supported issues tracking, and helped facilitate the system change management process. The workflow functionality of the tool tracked the completion and approval of the steps for key project activities prior to "go-live."^{Footnote 4} This functionality mitigated the risks that key project tasks were not completed prior to the go-live date.

3.1.2 Iterative user involvement

The audit team expected that input from the EMS end-users would inform the development of the EMC application, which would contribute to a better alignment of functionality to business requirements and to overall user satisfaction with the end product.

Agile Development, an iterative and incremental application development technique, involving continuous feedback from users, was used during the project. This user-centric approach promoted communications with, and the involvement of, the EMS end-users, ensuring that the application met the defined business requirements. User sessions were organized to validate application functionality at the conceptual level and to obtain feedback on demonstrations of the application's detailed functionality. This approach contributed to active project support from EMS senior management and EMS end-users.

A pre-production environment of the application, known as the "sandbox," allowed EMS end-users to become familiar with the application during the development period. This environment provided opportunities for valuable feedback on the alignment of system functionality to the detailed business needs and promoted user buy-in.

This user-centric approach focused on involving the EMS end-users and ensuring effective communications with them to maximize the value of the project deliverables. This approach also reduced the risk of unexpected delays or user dissatisfaction with an application that did not meet their business requirements.

3.1.3 Identification and implementation of business requirements

It was expected that the business requirements would be defined, articulated and translated to high-level design documents.

Business requirements for the EMC system were listed in a formal business requirements document, which provided an overview of the business processes and details on the requirements of each business process area. A "fit assessment," conducted by the project team, confirmed the extent to which the SAP solution accommodated the business requirements for the functionality required to support the Secretariat's budget office responsibilities.

The audit team found that the design documents demonstrated how the system was configured to support the business requirements. Specifically,

• Business activities were mapped to SAP functionality;
• System usage requirements also identified technical requirements;
• Documents for the decision to deploy the enhancement packages clearly outlined the functional and technical considerations for introducing greater functionality; and
• Documentation of the high-level design of relevant business objects were mapped to SAP objects.

Having clear and structured approaches for identifying and assessing IT solutions is critical to the identification and definition of business requirements, and to the translation of those requirements into an effective and efficient design of automated solutions.

The audit team found that key documents developed to support the project demonstrated how the business requirements were defined and implemented in the design of the EMC system.

3.1.4 User training

It was expected that training documentation would be developed to support EMS end-users in the correct usage of the system. It was also expected that these end-users would receive the appropriate training to allow them to achieve the business objectives.

EMS end-users, who were part of the project team, developed the training material and trained the other EMS end-users in the use of the new EMC system. During the project, approximately 48 documents were developed with detailed steps on how to enter master data, enter transactions, run reports and run queries in the system. These documents were shared with the broader EMS end-user group during training sessions.

Because of the small size of the group of users to be trained, a formal training plan was not developed and user participation in training sessions was not tracked. EMS end-users participated in specific training sessions that were designed to support them in their business function.

The development and maintenance of end-user documentation to ensure that it reflects current processes is critical to supporting the end-user in the correct usage of the system. It also reduces the risk of making errors in data input or setting incorrect parameters on reports used for analysis and to support decisions. Since system implementation, the EMS end-users have indicated that the system procedures are being maintained and enhanced as the system and processes are enhanced. EMS end-users indicated that additional training was provided following the initial training sessions. They also reported that a variety of training methods were implemented, including ad hoc or one-on-one training, weekly Question and Answer sessions, and a refresher session at the beginning of each Estimates exercise.

The audit team found that the training documents were well developed. They were presented in a standard layout and included tables describing data elements to be entered in the system and screen shots with comments inserted to assist the EMS end-users in the use of the EMC system.

3.2.1 Project information management

The audit team expected that the documentation to support the definition of requirements, decisions and activities for the development and implementation of the EMC system would be kept in a central location. Once the project was complete, the key documentation considered to be project information resources of business value would be identified and retained in a central location accessible to those IT individuals required to provide ongoing support of the EMC system.

The audit team observed that project files were kept in various server locations, including the shared directories within the IT-CSS and EMS organizations, individuals' email files, and the Secretariat's official document repository GCDocs.

The project files held a large number of documents, including project management documentation, formal business requirements, report mock-ups and report technical specifications, which were provided to the audit team for review. The following documentation weaknesses were observed by the audit team:

• Evidence of EMS end-users' feedback for the various sessions organized to validate the concepts and the functionality of the system were not available;
• Results and approvals of EMS end-user acceptance testing were not available;
• Functional specifications representing the detailed EMS end-user requirements for each EMC report and query were not documented and retained during the project;
• Data validation activities demonstrating that data integrity was maintained through the conversion process were not available; and
• The final approved document indicating user acceptance of data conversion and the system configuration was not available.

Retention of the key project documentation in a central location mitigates the risk of losing documents owing to employee departures, and supports the application of new change requests and system upgrades without disturbing the intended system design. Changes to the system or reports, without the knowledge about the intent of the designed functionality, may lead to data integrity issues, which can impair the ability of the business to make decisions with the information stored in the system.

3.2.2 Data conversion planning

The audit team expected that a data conversion plan would be in place, which would ensure that all risks and critical activities would be identified and addressed. This plan, in turn, would ensure that the conversion of data from legacy systems to the new EMC system would meet data integrity standards and could be relied upon for reporting and decision making.

The data conversion plan documentation was missing key elements that would have been included in accordance with a quality plan or organizational standard. The existing documentation identified critical data items, and details were provided about the data elements that needed to be migrated. Functional and technical testing requirements were also specified. However, the documentation did not adequately describe the planned conversion process.

Specifically,

• An assessment of the project's risks related to data conversion testing was not developed;
• Criteria defining the success of each data conversion test phase were not clearly articulated;
• Remediation procedures for instances where success criteria were not met were not defined; and
• There was no evidence of fallback and recovery plans.

Interviews indicated that the EMS end-users were frequently consulted during the conversion process. IT-CSS and EMS end-users on the project team stated that there were no issues with the data conversion.

The audit team was not able to validate the data integrity during the conversion process owing to the limitation of detailed documentation regarding planning and testing. However, the following factors mitigated the risks of data integrity issues:

• EMS end-users were involved during the planning phase and were frequently consulted during the conversion process.
• A collaborative approach was used to engage IT-CSS and the EMS end-users. These individuals, including those on the conversion team, had a high level of knowledge and skills about the business and the business processes.
• Explanations provided by IT-CSS and the EMS of the processes for handling the data conversion activities were at a sufficient level of detail to provide the audit team with a level of confidence that there was a very low risk of error in the data. These explanations included a description of error handling and controls in place to prevent the transfer of erroneous data in the EMC system.
• The first production cycle of Main Estimates and Supplementary Estimates A were subject to a detailed reconciliation process by the EMS end-users on the opening balance information to ensure that it was consistent with previously published information. This process remained consistent with the reconciliation processes that were regularly done for the publication of all Estimates documents from the previous legacy systems.
• Subsequent to the production of the 2013–14 Main Estimates, there have been five Supplementary Estimates published as well as the 2014–15 Main Estimates. Each of these publications has been subject to scrutiny by federal departments and agencies to ensure their information is accurate. The EMS indicated that it has not received any information regarding errors in the publications.

The EMS end-users indicated that they are satisfied with the completeness and accuracy of the data that was transferred from the EMIS legacy system.

The audit team determined that a recommendation is not required because of the aforementioned mitigating factors and the confidence of EMS end-users in the reconciliation process to mitigate the risk of data conversion errors.

Management Response

We agree with the recommendation. Currently, both sectors are using the JIRA application with links to functional specifications, mock-ups and technical specifications to perform accurate and consistent configuration management of the system.

Management Response

We agree with the recommendation.