Follow-up on Audit of Non-Public Property Information Management / Information Technology Rationalization: Phase 1 – Governance

July 2016

1259-5-002 (ADM(RS))

Reviewed by ADM(RS) in accordance with the Access to Information Act. Information UNCLASSIFIED.

Caveat

Non-Public Property (NPP) is created under the National Defence Act (NDA). The purpose of NPP is to provide benefit to serving and former members of the Canadian Armed Forces, and their dependants or for any other purpose approved by the Chief of the Defence Staff (CDS). Each unit’s NPP is vested in the commanding officer of that unit.

NPP is a unique type of crown property, the governance of which is assigned to the CDS. Pursuant to subsection 41(1) of the NDA, the CDS shall exercise his authority subject to any directions that may be given to him by the Minister.

Article 41(3) of the NDA provides that the Financial Administration Act does not apply to NPP. Revenues from NPP operations constitute non-public property pursuant to section 2 of the NDA.The result of this work does not constitute an audit of the NPP Information Management / Information Technology (IM/IT) Rationalization: Phase 1 – Governance project. Rather, this report was prepared to provide reasonable assurance that management actions that resulted from the 2011 Audit of NPP IM/IT Rationalization: Phase 1 – Governance effectively addressed the recommendations.

Acronyms and Abbreviations

ADM(RS)

Assistant Deputy Minister (Review Services)

Assoc DG

Associate Director General

BNA

Business Needs Assessment

CDS

Chief of the Defence Staff

CFMWS

Canadian Forces Morale and Welfare Services

CFO

Chief Financial Officer

CIO

Chief Information Officer

DGMWS

Director General Morale and Welfare Services

DRP

Disaster Recovery Plan

FY

Fiscal Year

IM/IT

Information Management / Information Technology

IM/IT CC

Information Management / Information Technology Coordination Committee

IS

Information Services

MAP

Management Action Plan

NDA

National Defence Act

NPP

Non-Public Property

OPI

Office of Primary Interest

PM

Project Manager

PMF

Project Management Framework

QA

Quality Assurance

RMC

Resources Management Committee

SRB

Senior Review Board

ToR

Terms of Reference

WG

Working Group

1.0 Introduction

The Internal Auditing Standards for the Government of Canada require Assistant Deputy Minister (Review Services) (ADM(RS)) to conduct a follow-up to monitor and ensure that management actions have been effectively implemented in response to previous audit recommendations. The purpose of this follow-up is to determine the progress made on the implementation of the management actions in response to the 2011 Audit of NPP IM/IT Rationalization: Phase 1 – Governance. This follow-up was included in the ADM(RS) Risk-based Audit Plan (Non-Public Property and Military and Family Services) for fiscal years (FY) 2014/15 to 2016/17.

The objective of the 2011 audit was to provide assurance to the CDS on the effectiveness and adequacy of the risk management, governance processes, and management controls that are in place subsequent to organizational changes in NPP IM/IT governance. The results of the 2011 audit are outlined as follows:

Governance. The transparency and accountability for strategic-level decision making in relation to IM/IT matters needs to be strengthened. The Information Services (IS) Division’s organizational structure can be more efficiently aligned to support the delivery of the Canadian Forces Morale and Welfare Services (CFMWS) IM/IT goals and objectives.

Risk Management. A comprehensive risk management plan for the IS Division, including the identification of risk impact thresholds and risk response strategies, is needed. The absence of an up-to-date disaster recovery plan (DRP) that has been tested, represents a substantial risk to CFMWS critical business data and information.

Management Controls, Accountability, and Stewardship. A forecasting tool (based on growth in demand) needs to be constructed to facilitate the estimation of NPP IM/IT resource requirements and priority setting. CFMWS’s delivery model for IM/IT customer support services beyond Monday to Friday (7 a.m. to 5 p.m.) needs to be re-assessed. Compliance with the NPP IM/IT Project Management Framework (PMF), especially in the area of project cost estimation, requires strengthening. An independent quality assurance (QA) function needs to be established to monitor the IS Division’s software testing process and procedures. A comprehensive performance management regime needs to be implemented to effectively monitor the IS Division’s success in achieving CFMWS IM/IT goals and objectives.

In order to address these issues, 18 recommendations were put forward (See Annex C). To address these recommendations, the CDS and the Managing Director, NPP developed Management Action Plans (MAP) to address the audit findings specific to their area of responsibility.

Back to Table of Contents

2.0 Methodology

This audit follow-up is the outcome of a review of documentation and evidence to assess the progress made in implementing the MAPs, based on the assessment criteria in Annex A. The following methods were used:

  • Interviews with the Chief Information Officer (CIO) and other key stakeholders within CFMWS;
  • Research, assessment and review of relevant information and documentation pertaining to the MAPs; and
  • Follow-up questions based on the results of the analyses.

Statement of Conformance

The audit follow-up conclusions contained in this report are based on sufficient and appropriate audit evidence gathered in accordance with procedures that meet the Institute of Internal Auditors’ International Standards for the Professional Practice of Internal Auditing. The audit follow-up thus conforms to the Internal Auditing Standards for the Government of Canada, as supported by the results of the quality assurance and improvement program. The opinions expressed in this report are based on conditions as they existed at the time of the audit follow-up, and apply only to the entity examined.

Back to Table of Contents

3.0 Overall Assessment

The original audit contained 18 recommendations. The follow-up found that seven recommendations had been fully implemented. Improvements have been made in the following areas:

  • All recommendations related to governance have been fully implemented. NPP Resources Management Committee (RMC) Terms of References (ToR) have been updated and implemented and the IS Division’s organizational structure has been revised to respond effectively to CFMWS’ IM/IT challenges and meet goals and objectives.
  • In the area of internal controls, several recommendations have been either fully or partially implemented. An activity tracking tool to facilitate the estimation of NPP IM/IT resource requirements has been developed and implemented, and measures have been undertaken to extend IM/IT customer support service hours.

Progress has been made with respect to all other recommendations with two exceptions: the recommendations to develop a DRP and an evaluation of the funding structure. In addition, six recommendations were found to be obsolete.

MAPs in the area of management controls, accountability and stewardship have not been fully implemented. The NPP IM/IT PMF needs to be updated to reflect procedures currently in place; compliance with the Framework and the delegated financial authorities needs to be strengthened and monitored. In addition, a comprehensive CFMWS technology roadmap to guide business system decision needs to be completed.

The Office of Primary Interest (OPI) concurs that not all MAPs are fully completed and indicated that staff will continue to work on outstanding action items.

A scorecard of the MAPs can be found in Annex B, and a more detailed assessment of progress with updated target dates of completion can be found at Annex C.

Back to Table of Contents

Annex A—Assessment Criteria

Line of Enquiry: Progress made on the June 2011 Audit Recommendations

Progress has been made on the implementation of the MAP identified in the June 2011 audit.

The following criteria were used to assess the level of completion for each recommendation:

1. Obsolete or Superseded.

Audit recommendations that are deemed to be obsolete or have been superseded by another recommendation.

2. No Progress or Insignificant Progress (0-24 percent complete)

No action taken by management or insignificant progress. Actions such as striking a new committee, having meetings and generating informal plans are insignificant progress.

3. Partial Implementation (25-74 percent complete)

The entity has begun necessary preparation for implementation, such as hiring or training staff, or developing or acquiring the necessary resources to implement the recommendation and/or actions taken have not fully addressed the associated risks/gaps.

4. Substantial Implementation (75-99 percent complete)

Structures and processes are in place and integrated in some parts of the organization, and some achieved results have been identified. The entity has a short-term plan and timetable for full implementation.

5. Full Implementation (100 percent complete)

Structures and processes are operating as intended and are implemented fully in all intended areas of the organization.

Back to Table of Contents

Annex B—Management Action Plan Scorecard

Recommendation OPI ADM(RS) Assessment of Progress on MAPs
1. RMC ToR Director General Morale and Welfare Services (DGMWS) Full Implementation
2. IS Division organizational chart CIO Full Implementation
3. Divisional risk management plan Associate Director General (Assoc DG) and CIO Obsolete or Superseded
4. DRP Assoc DG and CIO No progress
5. Contributions funding methodology Chief Financial Officer (CFO) No progress
6. IM/IT services demands trends CIO Full Implementation
7. Analysis of the demand for help desk customer support services CIO Full Implementation
8. Category II IT Support Specialist help desk duties and responsibilities CIO Full Implementation
9. CFMWS technology roadmap CIO Partial Implementation
10. IM/IT Project Priority Scoring Sheet CIO Obsolete or Superseded
11. Project Business Needs Assessments (BNA) CIO Substantial Implementation
12. Time tracking system CIO and CFO Full Implementation
13. Project risk assessments CIO Obsolete or Superseded
14. QA function CIO Obsolete or Superseded
15. Project client signatures CIO Obsolete or Superseded
16. Compliance with NPP Financial Authorities CIO and CFO Partial Implementation
17. Performance measurement regime Assoc DG Obsolete or Superseded
18. Project progress reports CIO Full Implementation

Table B-1. MAP Scorecard. This table shows the ADM(RS) assessment of progress on the MAP.

Back to Table of Contents

Annex C—Summary of Management Action Plan Status

Recommendations OPI Target Date Progress to Date Status of Action Item
Governance
Recommendation 1. To ensure adequate transparency and accountability, strategic-level IM/IT decisions should be formally recorded in the minutes and records of decisions of a strategic-level committee. ToR for the RMC should include the production of meeting agendas, minutes and records of decisions. These documents should be maintained in an appropriate database. DGMWS Completed

CFMWS has developed ToR for the RMC, which were approved by the DGMWS on July 8, 2013 and amended on January 11, 2016. The ToR require production of agendas and minutes for every meeting.

CFMWS has also created a specific file directory to store RMC agendas, minutes and supporting documents.

Full Implementation
Recommendation 2. The CIO should conduct a reorganization of the IS Division to better support CFMWS objectives and, in accordance with best practices, should continuously re-evaluate the IS Division structure for efficiencies and effectiveness. The reduction of the number of national managers, the establishment of a Strategic Analyst/IM/IT Coordination Committee (IM/IT CC) Coordinator position, the establishment of an additional project manager (PM) position, and a reassessment of the number of Application Developers should be considered as high priorities. CIO

1. Reduction of National Manager positions (July 2011);

2. Director IM/IT Operations hired (July 2011);

3. Request for RMC approval of additional PM (September 2011);

4. Strategic Analyst IM/IT CC Coordinator position considered by RMC (September 2011); and

5. Initial re-assessment of the number of Application Developers (December 2011).

Since 2011, several modifications have been made to the IS Division organizational chart and the CIO has confirmed his commitment to re-evaluate the organizational chart regularly.

In order to support CFMWS activities more effectively, the operations of the IS division have been centralized under four management groups (Operations, Business Services, Administration Services and IM/IT Strategic Planning) and the percentage of managerial positions has been reduced from 43 percent to 22 percent.

In addition, to reduce delays in the application development field, the number of PMs has increased from four to seven and external resources are being used more frequently to address temporary requirements.

Full Implementation
Risk Management
Recommendation 3. The CIO should develop a comprehensive risk management plan that includes the identification of risk impact thresholds and risk response strategies. Assoc DG and CIO September 2011

The development of a comprehensive divisional risk management plan has not yet been undertaken.

This recommendation has been rendered obsolete. Risk management is an issue that affects multiple divisions within CFMWS. ADM(RS) will undertake an engagement to assess the maturity of Risk Management within CFMWS in FY 2016-2017.

Obsolete or Superseded
Recommendation 4. A DRP should be developed as soon as possible. The establishment of an alternative recovery site should be considered as part of the overall strategy. This could be one of the top priorities for the proposed Strategic Analyst/IM/IT CC Coordinator position. Assoc DG and CIO

1. “Data only” recovery site implemented in Valcartier (June 2011);

2. Complete disaster recovery site implementation (May 2012, assuming corporate Business Continuity Plan is completed by February 2012)

The development of a DRP has not yet been undertaken.

A “data only” recovery site has not yet been created.

The development of a DRP cannot be undertaken before the requirements are set-up in a corporate Business Continuity Plan. The CFMWS Business Continuity Plan is presently under development with Military Personnel Command and is expected to be developed by Spring 2017.

This MAP item will remain open until a DRP is developed. Management expects to have completed the development of a CFMWS DRP (including high level costing estimates) by March 31, 2017.

No progress
Management Controls, Accountability and Stewardship
Recommendation 5. The methodology used to determine the contributions from Canadian Forces Exchange System, North Atlantic Treaty Organization Exchange System, and Service Income Security Insurance Plan Financial Services should be re-evaluated. CFO September 2011

A re-evaluation of the methodology used to determine the contributions from Canadian Forces Exchange System, North Atlantic Treaty Organization Exchange System, and Service Income Security Insurance Plan Financial Services has not yet been undertaken.

Since 2011, the IS Division management activity has been streamlined and resources are no longer associated with specific divisions. Recommendations are supported and will be addressed through:

  1. The ongoing development of a service-level agreement between CFMWS and Military Personnel Command;
  2. The development of the Total Cost of Ownership exercise that will be led by the IS Division with support by the Finance Division; and
  3. A zero-based baseline review of the IM/IT allocation agreed upon at reference.

This MAP item will remain open until the actions above have been undertaken. Management expect to have completed all the actions by March 31, 2017.

No progress
Recommendation 6. For purposes of strategic planning and resource allocation, the CIO should ensure that the demand trends for IM/IT services are tracked and that this information is used to estimate CFMWS NPP IM/IT resource requirements and as input into the establishment of the IM/IT funding baseline. CIO March 2012

During FY 2013/14, a help desk activity tracking system was implemented. The system tracks information and generates various reports on Help Desk demands, productivity and trends.

Due to a high degree of variability in customer service demands on the Help Desk, several years of data will be required to identify trends that help reliably establish an IM/IT funding baseline. Nonetheless, management has used trend information generated from the data collected to date in order to support a request for increase in baseline funding for temporary Help Desk urgency at the RMC.

Full Implementation
Recommendation 7. The CIO should undertake an analysis of the demand for help desk customer support services. Based on the result, a cost-benefit/options analysis should be conducted with regard to the provision of these services. CIO March 2012 As indicated above, a Help Desk activity tracking system was implemented in 2013-2014. With the introduction of the Help Desk activity tracking system, management has been able to analyse their Help Desk customer support services and improve their handling of after-hours Help Desk demands by covering only emergencies. Full Implementation
Recommendation 8. The CIO should ensure that Category II IT Support Specialist help desk duties and responsibilities are clearly defined and communicated to all stakeholders. CIO September 2011

An “off-hours requirements of the Network and Systems section of the IS Division” agreement has been developed.

The agreement provides clarification on the duties, responsibilities and compensation of “off hours” work for Category II IT Support Specialist.

The agreement has to be signed by every Category II IT Support Specialist on a yearly basis.

Full Implementation
Recommendation 9. The CIO should ensure that a CFMWS technology roadmap is developed and updated periodically. CIO December 2011

The development of a CFMWS technology roadmap has not yet been undertaken.

An Enterprise Architect position, specializing in IM/IT strategic planning has been established to obtain the sets of skill and knowledge required to develop a technology roadmap. The position is presently in the process to be staffed.

This MAP item will remain open until a CFMWS technology roadmap is developed and updated periodically. Management expects to have completed the development of a CFMWS technology roadmap by December 31, 2016.

Partial Implementation
Recommendation 10. The CIO should amend the IM/IT Projects Priorities Scoring Sheet to include “impact/risk of doing nothing” as a vital part of the “value to the business line” criterion. Two additional criteria should be considered: (1) project urgency; and (2) project risk. CIO

1. Inclusion of “impact/risk of doing nothing” (completed);

2. Not applicable

This recommendation is considered obsolete since the methodology used to prioritize projects has changed from a risk-based process to a requirement/classification based process.

The prioritization criteria have been simplified and reduced to two criteria in order to facilitate more objective and consistent project assessments.

Obsolete or Superseded

Recommendation 11. The CIO should revise their PMF to incorporate guidance on the BNA, risk assessments and appropriate sign-offs for all associated tools. (This recommendation supersedes recommendations 11,1 13 and 15.)

CIO Completed

The BNA form has been modified. PMs are no longer required to develop a project schedule before the project is officially launched. The PMF is currently under review to include further guidance on the BNA requirements.

A compliance review has been conducted on 21 projects managed by the IS Division. The results noted that a risk statement had been developed for 84 percent of the projects and a rough order of magnitude estimate for project cost had been created 87 percent of the time.

This MAP item will remain open until the PMF is reviewed. Further guidance should be included on the BNA requirements, risk assessment obligations and client signatures; in addition, actions should be taken to ensure compliance. Management expects to have completed the actions above by October 1, 2016.

Substantial Implementation
Recommendation 12. The CIO and CFO should ensure that a formal time tracking system is developed and implemented and that proper cost estimates are prepared for every IM/IT project. These cost estimates should provide information about the forecasted total cost of owning the asset over its useful life. CIO and CFO September 2011 As indicated above, CFMWS has implemented a time tracking system for the IS Division. The system automatically calculates the cost of the time allocated to specific projects, or activities. Full Implementation
Recommendation 13. The CIO should ensure that proper risk assessments are developed and included in project documentation and progress reports to Senior Review Boards (SRB) and working groups (WG). CIO Completed

Current guidance does not require IS Division PMs to develop a formal project risk assessment. The PMF is under review to include the requirement for a project risk assessment.

This recommendation has been rendered obsolete. This issue will be resolved through the implementation of recommendation 11.

Obsolete or Superseded
Recommendation 14. The CIO should establish a QA function that is independent of the quality control function. In addition, the CIO should require that two copies of all software testing documentation be maintained. One copy should be maintained by the PMs and one copy should be kept in a separate QA directory to which only the QA staff has access. CIO

1. Establish independent QA function (May 2015); and

2. Create separate directory (June 2011)

Management has accepted the risk of not implementing this recommendation on 1 June 2016. Obsolete or Superseded
Recommendation 15. The CIO should ensure that the PMs obtain client signatures on the BNAs, Statements of Requirements, User Acceptance Testing, and Release documentation for NPP IM/IT projects. Copies of these sign-offs should be maintained as part of the project documentation. CIO Completed

Since the initial audit, two new project sign-offs templates have been created and implemented – the Solution Development Approval and Solution Implementation Approval. The PMF is under review to formalize the requirements for the use of these templates.

A compliance review has been conducted on 21 projects managed by the IS Division. The results were as follows:

  • BNA forms are signed off 89 percent of the time;
  • Statements of requirements are signed off 45 percent of the time;
  • Project Charter / Definition documents are signed off 45 percent of the time;
  • Solution Development Approval documents are signed off 40 percent of the time;
  • Solution Implementation Approval documents are signed off 33 percent of the time;
  • User acceptance testing is signed off 33 percent of the time; and
  • Project Completion Sign-Off documents are signed off 75 percent of the time.

This recommendation has been rendered obsolete. This issue will be resolved through the implementation of Recommendation 11.

Obsolete or Superseded
Recommendation 16. The CIO should ensure that the PMs obtain the proper authorizations for payment of NPP IM/IT-related invoices. In addition, the CFO should ensure that the Non Public Funds accounting staff are aware of the IM/IT authorities and confirm that these invoices are properly approved prior to processing the payment. CIO and CFO September 2011

A compliance review has been conducted on 87 financial transactions made by the IS Division as part of their projects.

It was noted through the review that the commitment of funds was exercised appropriately in each case; however, some payments were approved by individuals without the appropriate IM/IT signing authority, and the requisition process did not identify these anomalies.

Since the time of this review, the delegation of authority has been updated of include provisions that address the process-related shortcomings associated with payment approval and requisitioning. These changes will be communicated in writing to all requisitioning authorities by September 30, 2016. In addition, approval of a new delegation of authority document will be sought by March 31, 2017 to better address operational realities of payment approvals.

Partial Implementation
Recommendation 17. The performance measurement regime that is being developed as part of the CFMWS New Deal initiative should incorporate key IM/IT metrics. Assoc DG April 2012 This recommendation has been rendered obsolete since the performance management regime that was going to be developed under the New Deal initiative has been cancelled. However, the requirement for an appropriate performance measurement framework remains. This issue will be monitored as part of the follow-up of the Audit of Non-Public Property Governance, Strategic Management and Business Planning. Obsolete or Superseded
Recommendation 18. The CIO should ensure that the PMs are provided with specific guidance on the format and content of project progress reports and that these reports are provided to SRB and WG members at least annually. CIO December 2011

The IS Division PMs provide multiple project status reports throughout the year to members of the SRB and WG.

The reporting process has been formalized through a template and the PMF is under review to include further guidance on the development of project status reports.

Full Implementation

Table C-1. Status of the Implementation of MAP Items. This table gives details on the progress of implementation of MAP items.

Back to Table of Contents

___________________________________________________________________________________________________________________________

Footnote 1 Original recommendation 11: The CIO should ensure that, at a minimum, rough order of magnitude estimates for project cost, risk and schedule are completed in the BNA for each project.

Page details

Date modified: