Review of the Departmental Financial Management System Renewal Project: chapter 4

3. Findings and Recommendations

Overall, the review team found that the project management approach used by the Departmental Financial Management System (DFMS) Renewal Project team was appropriate and that project management controls and practices were applied effectively by Environment and Climate Change Canada (ECCC). Strong project management practices related to regular and timely reporting of project status, issues and risks; ECCC’s focus on the need to ensure that business process changes were well aligned with the implementation of new SAP technology; and the embedding of ECCC functional experts with the technical SAP team at Agriculture and Agri-Food Canada were noted. The review, however, did find several exceptions that posed risks to the overall success of the implementation project.
 
In all, seven findings on project risks related to ECCC’s readiness to begin using Systems, Applications and Products (SAP) on April 1, 2015, and detailed recommendations for addressing these risks, were communicated by the review team in its two interim reports to the ECCC project team during the course of the review. These seven findings were related to:

• composition and role of project steering committees in project governance for the initial implementation of DFMS;
• preparation of agreements to govern the ongoing host-client relationship with AAFC;
• documentation and testing of internal controls over financial reporting;
• testing of new SAP asset management functions by knowledgeable business users at ECCC;
• preparation of the ongoing support organization for SAP users at ECCC;
• leveraging the experience gained by the DFMS Renewal project team prior to April 2015 in upcoming Enterprise Asset Management module (EAM Phase 2) of the project; and
• governance of the upcoming EAM Phase 2 of the project.

Some findings and recommendations were focused on point-in-time risks that were specific to the implementation phase of the project, while others dealt with broader risks to the overall success of the implementation, the ongoing support of the system and its use by ECCC. In March 2015, a briefing was provided to the External Audit Advisory Committee (EAAC) on the seven findings, as well as comments provided by ECCC management on their response to recommendations made by the review team.

Of the seven findings reported during the review, four continue to be relevant to ECCC following the SAP implementation go-live date of April 1, 2015, as reflected through the remaining recommendations.

3.1 Project Management and Governance

Project management activities related to the development and maintenance of project plans were effective, as were the reporting of project risks, project issues, and the status of actions taken to address these risks and issues.

The review found that although there appeared to be effective oversight of the DFMS Renewal project by ECCC executive management and governance structures, an opportunity for improvement was noted with respect to better representation of all project stakeholders in project governance.

The review also found that the management of project resources was generally effective; however, a recommendation was made concerning the continuity of project resources for the upcoming EAM Phase 2 of the project.

EAM Phase 2 Governance Structure and Direction

Continuation of the DFMS Renewal project to implement Enterprise Asset Management (EAM) across all business units at ECCC is planned for FY2015-2016 and will be sponsored by the The Assistant Deputy Minister (ADM), Corporate Services Branch (CSB). Although sponsorship of the project is changing, the review team expects EAM Phase 2 to be governed effectively and that governance will ensure continuity with project work completed to date.

The review team found that asset management business practices, processes and supporting systems vary widely among business units at ECCC, challenging project milestones related to business process definition, data migration, and testing activities for asset management functions.

Strong governance over this work will be required in EAM Phase 2 to ensure that deadlines are met and that tasks are completed. Initiating and sustaining a wholesale change in the culture of the business organizations--i.e. from one where developing and maintaining unique systems and processes is acceptable to one using a common system (SAP) and approach--will require top-down leadership over change management activities in the project.

Resources for Asset Management Implementation

The scope of work to implement SAP’s EAM functions for ECCC represented a higher potential risk to the overall success of the DFMS Renewal project, due to factors that include:

• complexity of business processes and variability of business processes among various user communities within ECCC;
• differing technical system requirements among user communities, often dependent on the type of assets managed;
• a large number of existing legacy asset management software tools, ranging from MERLIN’s Asset Lifecycle Management (ALM) module to simple spreadsheets or other ”black book” systems;
• geographical dispersal of business units and prospective users (i.e. small numbers of users in many locations, including remote locations), complicating participation of some business representatives regarding requirements definition, testing, and training; and
• AAFC did not include EAM in its configured SAP solution prior to ECCC’s partnership on DFMS Renewal, so EAM requirements were ”net new” to the technical team and thus required more design, configuration and testing time than other SAP modules.

The DFMS Project Team recommended to EMC in December 2014 that an additional phase of the DFMS Renewal project be defined to focus on completion of the EAM implementation in SAP. Preliminary, high-level estimates of additional funding and human resource requirements were provided to EMC in support of the recommendation.

The review team expected to find that planning for this additional work would be structured so that the upcoming EAM Phase 2 of the project would be continuous with the current DFMS Renewal Project and that key project resources would be retained for EAM Phase 2 to ensure that knowledge of the work conducted to date would not be lost.

The review team found that additional project phases were not anticipated in approved project documentation, including the Project Charter and Business Case.  The review team also noted that the decision to limit the scope of EAM work in the initial phase of the project reduced the risk that delays on asset management project activities would threaten the planned implementation of SAP at ECCC on April 1, 2015. However, significant risks to ECCC remain regarding the management of the continuation of the work on EAM functions.

In particular, there is a risk that the departure of key DFMS project personnel who are scheduled to leave the project after the April 2015 implementation could result in a loss of continuity in the management of the project and a loss of expertise and technical knowledge gained on the project to date.

There may be a loss of expertise in terms of asset business processes, their configuration in SAP relevant project management processes and lessons learned if there is inadequate knowledge transfer from the current project team. Reduced continuity and a loss of technical knowledge among project personnel could also affect the ability of the EAM Phase 2 team to accurately estimate the time and cost required to complete the remaining work.

3.2 Functional Readiness

Overall, the review team found that project tasks related to the functional readiness of the new SAP-based DFMS were well managed. ECCC business stakeholders were involved in the specification of business requirements and the documentation of business processes that were in scope for the DFMS Renewal Project. In particular, the review team noted that communities of practice (CoPs) were established to focus on business change and technical requirements associated with a specific set of business processes, such as procurement-to-payment. The CoPs were a communication link between the DFMS Renewal Project team and business units within ECCC; each CoP was led by a Business Transformation team lead from the DFMS Renewal project team.

The review noted that project activities were well coordinated with the AAFC SAP project team. Embedding of ECCC functional experts with the team at AAFC provided strong communication and knowledge exchange, particularly with respect to ECCC specific requirements such as those related to asset management.

The review team also noted that a stream of project activities was dedicated to ensuring that the SAP financial system, hosted by AAFC, was assessed by ECCC’s IT security specialists to ensure that appropriate certification and accreditation of the application was conducted prior to its implementation. Activities associated with the preparation and migration of legacy data from ECCC’s MERLIN system were adequately planned and monitored.

One finding, previously communicated to management, was related to the adequacy of user acceptance testing of SAP by ECCC prior to implementation. The review found that test plans included an abbreviated testing approach for SAP functions already running in production at AAFC, but did not include more detailed user acceptance testing (UAT) cycles for enterprise asset management (EAM) functions, which were new to the AAFC SAP environment. The review recommended that sufficient user acceptance testing for SAP functions that were specific to ECCC and new to the AAFC SAP environment be conducted prior to implementation. The Finance Branch indicated that, as the project uses an adopt/adapt approach and the core SAP project is already configured, there was no need for user acceptance testing. However, a proof of concept was demonstrated to users for EAM, and the revised proof of concept was approved by key users (e.g. Meteorological Service of Canada and CSB) and is planned to be used as part of training sessions.

3.3 Deployment Readiness

Overall, the review team found that project tasks related to the readiness of ECCC to migrate to the new SAP-based DFMS were well managed. The DFMS Renewal Project prepared detailed cutover plans and lists of critical go/no-go criteria as controls over decisions and activities related to the migration from its legacy MERLIN system to SAP on April 1, 2015.

Communications plans and a broad business transformation strategy were also included in project plans, and activities described in these plans were in progress at the time of the review. The review team also noted that organizational change management plans will be critical to the success of ongoing work related to asset management business functions in EAM Phase 2.

One finding was communicated to management regarding this line of enquiry; it was related to the adequacy of preparations completed to establish an effective user support organization for users of SAP at ECCC, as summarized in the following paragraphs.

Readiness and Coordination of ECCC Support Organizations

The review team expected to find project tasks intended to set up an effective user support organization for the SAP-based DFMS to be included in the project plan. It also expected the support organization to include processes and tools to ensure a seamless coordination with SAP support at AAFC. Finally, the review team expected key ECCC managers and staff who would operate the support organization to participate in its setup.

Environment and Climate Change Canada reorganized support for its enterprise financial system, migrating from a systems-oriented support model to a client-oriented one. The support organization includes a functional help desk to assist users of the system navigate through menus, answer questions about specific transactions or other system features, diagnose error messages encountered during processing, and route requests that cannot be solved to other support organizations (e.g. ECCC’s national IT help desk, or AAFC’s SAP help desk). The support organization also provides ongoing SAP training and is responsible for maintaining common master data stored in SAP and for managing controls over the operation of the system, including those related to the management of system access, assignment of security profiles to users and segregation of duties.

ECCC’s support organization must also be managed to coordinate efficiently and seamlessly with AAFC’s SAP support organization, so as to provide effective service to ECCC’s SAP user community, ensure compliance with support principles and standards that are documented in service level agreements (SLAs) between ECCC and AAFC, and maintain controls related to the operation of the system.

Operational demands are placed on the support organization immediately upon opening the new DFMS for production use. Gaps in understanding of the support organization’s function, in business procedures to be followed, in staffing of support positions and in the readiness of tools, such as the help-desk service ticket software system, increase the risk that errors or problems experienced by early users of the system will not be addressed efficiently; there is an added risk that the system implementation will be viewed as unsuccessful by the user community.

The review team found that work to develop and implement a support organization for DFMS appeared in the project plan, that planned steps were being carried out and that the project team had begun to communicate how the support organization will work. In particular, attendees of SAP training in early February 2015 received instructions on how to contact support resources.

The review team also found that work associated with the completion of the installation and configuration of the ticket system software to be used by the new support organization had been delayed.

Finally, the review team found that ECCC managers identified as responsible for the support organization after implementation had limited involvement in project work related to setting up the support organization.

3.4 Business Process Readiness

Overall, the review team found that project tasks related to the readiness of ECCC to begin using the new SAP-based DFMS were well managed. However, exceptions were noted.

The review team found that the DFMS Renewal Project had appropriately planned and prepared for training courses to be delivered to business users at ECCC and that Wave 1 training courses (SAP users were divided into training cohorts based on their assessed priority for access to the system) were delivered to user communities according to the plan. Evaluations of training courses indicated that the course material was relevant to the training objectives. The review team noted that the training material provided to the team by the AAFC SAP team had been substantially modified to include ECCC-relevant references and exercises, and effective job aids (e.g. an automated MERLIN-to-SAP account code translation tool) were developed and ready for use by the training teams when the classroom sessions began.

The review team also noted that ECCC seized the opportunity presented by the implementation of a new financial system to standardize and document business processes to be supported by the new system. Business process changes, a key component of the DFMS Renewal project at ECCC, included the identification of business process control points in process documentation, but did not include detailed design effectiveness testing of control activities.

The following finding was communicated to management related to controls, specifically regarding the readiness of ECCC’s internal controls over financial reporting.

Update and Testing of Internal Controls

Since ECCC is changing its financial system of record and is modifying the definitions of its major financial business processes, the review team expected to find a stream of activity in the DFMS Renewal project plan dedicated to the review and assessment of the impact that changes to ECCC’s systems and processes would have on its internal controls over financial reporting (ICFR). Specifically, the review team expected that the scope considered by this stream of work would include the development of control frameworks to support the newly designed ECCC-business processes and specific design effectiveness testing relating to those controls and covering business process controls, application security controls, IT general controls specific to ECCC, controls over interfaces between SAP and other systems, and controls over data conversion from existing systems to SAP.

The review team found no specific stream of activity in the project plan to review and assess internal controls. The review team obtained completed business process documentation and noted that control points are captured in revised business process documentation, but did not find that control activities associated with these control points had been formally documented in a control framework to assess the completeness of the controls, nor had they been assessed for design effectiveness.