Review of the Departmental Financial Management System Renewal Project: chapter 6

Annex 1 - Methodology

Review Criteria

Review criteria were developed to address risks to the DFMS Renewal Project from two sources:

inherent risks that, based on the experience of the review team, are frequently associated with projects of this type within the federal government; and,
risks specific to ECCC that were identified during the review of DFMS Renewal project documentation and preliminary interviews with project stakeholders.

Criteria to address these risks were organized into four broad lines of enquiry, as shown in the table below. Lines of enquiry, project risks and corresponding review criteria were developed and validated against the TBS project risk framework (which serves as the basis for the TBS’s Project Complexity and Risk Assessment workbook and methodology) and IT audit frameworks found in CobiT 5^Footnote4 and the Global Technology Audit Guide^Footnote5 (GTAG):

1. Project Management and Governance
Review Criteria	Sources of Criteria	Met/Not Met
1.1 Project management and governance structure has been established to provide oversight and ensure deadlines are met, scope is adhered to and objectives are met.	CobiT 5: BAI01.07, BAI01,11, BAI02.04. GTAG 12: 1.6, 3.3, 3.4	Partially Met ^Footnote6
1.2 A steering committee has been established that has a clear mandate and representation from key stakeholder groups.	CobiT 5: BAI01,01, BAI01.07. GTAG 12: 3.4, 3.5, 3.7	Partially Met^Footnote7
1.3 A project plan has been established and accepted by all parties and stakeholders to assist in meeting project deadlines and objectives.	CobiT 5: BAI01,08 GTAG 12: 2.1, 2.3, 2.4, 2.5, 2.6	Met
1.4 Adequate and appropriate resources have been assigned to execute the project plan.	CobiT 5: BAI01,08. GTAG 12: 2.3, 13.3	Not Met^Footnote8
1.5 Issues and risks are formally identified, tracked, managed and resolved in a timely and appropriate manner.	CobiT 5: BAI01,01, BAI01.10, BAI01.06, BAI02.03. GTAG 12: 3.7	Met

2. Functional Readiness
Review Criteria	Sources of Criteria	Met/Not Met
2.1 Changes to financial business processes have been analyzed and reviewed with business users.	CobiT 5: BAI02,01, BAI03.03, BAI03.09. GTAG 12: 4.1, 4.2	Met
2.2 Documentation on project business requirements, including change requests, has been developed and approved by project stakeholders.	CobiT 5: BAI03.09. GTAG 12: 2.1	Partially Met^Footnote9
2.3 Application security requirements have been analyzed and a plan has been established to ensure adherence to departmental and GC security requirements.	CobiT 5: BAI03.02, BAI03.05. GTAG 12: 8.12, 11.1, 11.3	Met
2.4 A comprehensive testing strategy and plan has been developed for all business process changes and data migrations. Testing includes business user involvement.	CobiT 5: BAI01.08, BAI03.07, BAI07.03. GTAG 12: 6.1, 8.8, 9.1	Partially Met^Footnote10
2.5 Adequate testing has been planned for data migration.	CobiT 5: BAI03.07, BAI07.03, GTAG 12: 7.1, 7.3	Partially Met^Footnote11

3. Deployment Readiness
Review Criteria	Sources of Criteria	Met/Not Met
3.1 A system cutover plan is part of the overall project plan and includes documented conversion specifications.	CobiT 5: BAI01,08, BAI05.05, BAI07.02. GTAG 12: 14.1, 14.2	Met
3.2 For major milestones, management documents go/no-go criteria and subsequent decisions.	CobiT 5: BAI01,07, BAI01.11.	Met
3.3 A communication plan informs stakeholders and management of the progress of the roll-out.	CobiT 5: APO08.04, BAI05.05. GTAG 12: 3.1	Met
3.4 An organization-wide change management strategy to identify and document technical and operational aspects of the cutover to SAP at ECCC has been prepared and implemented.	CobiT 5: BAI05.05, BAI07.02. GTAG 12: 3.1, 12.1, 14.1, 14.2	Partially Met^Footnote12
3.5 ECCC has established an appropriate operating framework with AAFC related to its post-project role as a service provider to ECCC for SAP. The operating framework includes a process for adding changes to SAP that impact only ECCC users and for supporting EC-specific SAP functions.	CobiT 5: APO09.04 APO09.04) APO10.03. GTAG 12: 16.1, 16.3	Partially Met^Footnote13

4. Business Process Readiness
Review Criteria	Sources of Criteria	Met/Not Met
4.1 A training program has been developed and executed prior to deployment of the new system; affected business functions are trained prior to implementation.	CobiT 5: BAI05.05, BAI05.07. GTAG 12: 12.2	Met
4.2 Business process controls are reviewed for potential changes based on changes to business processes. If there are changes, documentation is updated accordingly.	CobiT 5: EDM01.03, MEA02.01, DSS07.04. GTAG 12: 11.2, 16.4	Not Met^Footnote14

Key Dates

Opening conference (launch memo)
August 2014

Review plan completed
October 2014

Review plan to EAAC for information
November 2014

Interim reports
December 2014, February 2015

Interim report presented to EAAC
March 2015

External Audit Advisory Committee tabling
June 2015

Deputy Minister approval
December 2015

Top of page

Footnotes

Footnote 4

COBIT 5: An ISACA Framework; ISACA; 2012.

Return to footnote4 referrer

Footnote 5

Global Technology Audit Guide 12, Auditing IT Projects; The Institute of Internal Auditors; 2009.

Return to footnote5 referrer

Footnote 6

Governance over the DFMS Renewal project was adequate; consistent governance structures and strong leadership over project activities will be required to ensure that implementation of EAM is completed on time and within the budget established for the next phase of the project.

Return to footnote6 referrer

Footnote 7

Steering committees were established at ECCC; a joint governance committee, with participation of executives from ECCC and AAFC, was not part of the governance structure for the project.

Return to footnote7 referrer

Footnote 8

Adequate and appropriate resources were assigned to execute the project to date; the preparation of accurate plans and estimates and the continuity of work on the upcoming Phase 2 of EAM is threatened by the departure of key project team members.

Return to footnote8 referrer

Footnote 9

Business requirements were developed and approved for the April 1 implementation of SAP; divergence in processes among asset management organizations at ECCC and the departure of key project personnel are risks to the success of Phase 2 of EAM.

Return to footnote9 referrer

Footnote 10

A testing strategy and plan were developed; testing did not include formal user acceptance testing by ECCC business users of new asset management functions in SAP.

Return to footnote10 referrer

Footnote 11

Adequate testing has been planned for data migration; continuity of work on asset management data migration in Phase 2 of EAM is at risk due to the departure of key project personnel.

Return to footnote11 referrer

Footnote 12

A change management strategy was prepared and implemented; implementation of changes to asset management business processes will be completed in Phase 2 of EAM.

Return to footnote12 referrer

Footnote 13

An appropriate operating framework with AAFC was established; ECCC will operate separate support organizations for Finance/Procurement and Asset Management, requiring coordinated management of the host-client relationship with AAFC.

Return to footnote13 referrer

Footnote 14

Documentation prepared by the business transformation teams identifies control points in all processes; analysis of controls did not include update of existing internal control matrices, nor creation of new controls documentation.

Return to footnote14 referrer

Table of contents

Page details

Date modified:: 2016-04-05

Language selection

Search

Review of the Departmental Financial Management System Renewal Project: chapter 6

Annex 1 - Methodology

Review Criteria

Page details