Follow-up on Internal Audit: Chemical, Biological, Radiological and Nuclear (CBRN) Defence Omnibus Project

December 2013

7050-11-48 (CRS)

Reviewed by CRS in accordance with the Access to Information Act (AIA). Information UNCLASSIFIED.

Caveat

The result of this work does not constitute an audit of the CBRN Defence Omnibus project. Rather, this report was prepared to provide reasonable assurance that the Management Action Plans (MAP) that resulted from the May 2008 CBRN Defence Omnibus project audit were implemented as stated and as such have addressed the associated recommendations.

Acronyms and Abbreviations

Introduction

In keeping with the Treasury Board Policy on Internal Audit,¹ Chief Review Services (CRS) is required to undertake audit follow-ups to assess the implementation status of MAPs developed in response to previous CRS audit recommendations. In accordance with the CRS Risk based Audit Plan for fiscal year 2013/14 to 2015/16, this audit follow-up was selected to ensure that some of the systemic capital acquisition process issues, as well concerns specific to the CBRN Omnibus project have been addressed.

In May 2008, CRS completed an audit of the $217-million CBRN Omnibus project. The objective of the audit was to provide assurance that the CBRN Defence Omnibus project governance, risk and management control frameworks were in place to ensure a cost-effective acquisition.

While the May 2008 audit concluded moderate improvement was required in governance and risk management, it also identified opportunities to streamline the capital equipment approval process. Nevertheless, there were some innovative management practices observed. For example, the centralization of the functions in the CBRN Defence Omnibus project office provided efficient services to the sub-projects. Also, the project had developed its own project director handbook to document the project office procedures. The findings of the 2008 audit are summarized as follows:

Project Governance. Project leader positions of some capital projects were not staffed in line with the Project Approval Guide,² and 62 percent of the Omnibus project umbrella positions were vacant. In addition, only eight projects of the 16 CBRN inter-related defence projects were governed by the Omnibus project.

Operational Requirements. The CBRN Defence Omnibus sub-projects’ Statements of Requirements were not consistent for all of the Department of National Defence (DND)’s operational tasks.

Schedule. CBRN Defence Omnibus sub-project delays resulting from expenditure approval and contract award increased project management costs and delayed availability of operational capability.

Contract Management. A lack of DND participation in contract negotiations led to the acquisition of consumables in excess of actual requirements.

Options Analysis. The project documents were missing the fully costed options analysis that was a mandatory requirement of the Project Approval Guide.

Information for Decision Making. Some of the project documentation and vendor reports lacked complete information to support sound decision making.

Risk Management. CBRN Defence Omnibus sub-projects did not quantify the cost of schedule risks, align risk severity with contingency funding, or report risks to senior management.

In order to address the above issues, seven recommendations were made (see Annex A ). To address these recommendations, the Vice Chief of the Defence Staff (VCDS), and Assistant Deputy Minister (Materiel) (ADM(Mat)) developed a set of MAPs to address the audit findings specific to their area of responsibility.

Methodology

This audit follow-up is based on a review of documentation and evidence to assess the progress made in implementing the MAP. The following methods were used:

• Analysis of data from the Audit and Evaluation Recommendation Management System (AERMS)³ and the Capability Investment Database (CID);
• Interviews with personnel from the CBRN Defence Omnibus project office and the VCDS and ADM(Mat) staff; and
• Review of key project documentation and policies pertaining to the MAPs.

Statement of Conformance

The audit follow-up conclusions contained in this report are based on sufficient and appropriate audit evidence gathered in accordance with procedures that meet the Institute of Internal Auditors’ International Standards for the Professional Practice of Internal Auditing. The audit follow-up thus conforms with the Internal Auditing Standards for the Government of Canada, as supported by the results of the quality assurance and improvement program. The opinions expressed in this report are based on conditions as they existed at the time of the audit follow-up, and apply only to the entity examined.

Overall Assessment

The follow-up on the May 2008 audit identified that progress has been made in some areas. For example, there have been improvements in controls that relate to requirements definition, and information for decision making.

At the time of follow-up, there were 20⁴ out of 22 MAP items designated as fully implemented in AERMS. Upon further examination, seven out of 22 MAP items require additional measures to fully address the issues. As for those MAPs that have yet to be fully addressed at the time of the follow-up, the majority will be implemented by 2016 due to the systemic nature of the issues. An implementation status of the MAP items can be found in Annex A.

Until all MAPs are fully implemented, there remain risks in the following areas:

• The lengthy project approval process that prolongs the delivery of operational capability and increases cost will not be resolved until the validation of the Project Approval Process Redesign is complete in 2016.
• The issue of Service-level Agreements with Public Works and Government Services Canada (PWGSC) that define the time required to award contracts has not been resolved as performance standards based on procurement complexity will be examined over the next year. Project offices are unable to accurately forecast contract award milestones without these performance standards.
• There is limited participation of DND in contract negotiations led by PWGSC unless the contract is a sole-source procurement. Technical expertise offered by DND is necessary for successful negotiations of contract amendments and rate negotiations for competitively tendered contracts.
• Appropriate ranks for project leaders based on the project value and risks are not in place for all projects. Under-ranked project leaders may not have the experience level to manage high-risk projects.

For the purpose of monitoring the implementation of the seven action plans that have not yet been fully implemented, the AERMS status of these action plans will be revised by CRS to reflect the results of this audit follow-up.

Annex A—Summary of Management Action Plan Status

Project Governance

Management Action Plan and AERMS Status

1.a)1) Project leaders in the CBRN Defence Omnibus Project are now appointed in accordance with the instructions in the Project Approval Guide (4.3.7).

OPI: VCDS/Chief of Force Development (CFD)

Target Date: April 2008

AERMS Status: Full implementation

1.a)2) The project leader’s delegation guidelines in the Project Approval Guide are considered correct and need to be reinforced at the time of departmental project approval when the resources are assigned to the project at Program Management Board. We will look into how this might be done in the Project Approval Guide revision. Those 76 capital projects where the project leader is either over- or under-ranked will be rectified.

OPI: VCDS/Chief of Programme (C Prog)

Target Date: December 2009

AERMS Status: Full implementation

1.a)3) DAOD 1000-1 will be revised in consultation with ADM(Fin CS) and ADM(Mat) to assign an OPI for the project management function.

OPI: VCDS/C Prog

Target Date: December 2009

AERMS Status: Full implementation

1.b) The Decontamination, Collective Protection, and New Gas Mask projects have been moved under the Omnibus project. Two mature legacy projects remain outside the Omnibus. Other CBRN defence projects belong to external organizations and are not appropriate for transfer. The CBRN Defence Program will assign all new projects to the Omnibus project.

OPI: VCDS/CFD

Target Date: Ongoing

AERMS Status: Full implementation

Assessment of Progress

1.a)1) Improvements were evident in the rank of the project leader of the CBRN Omnibus sub-projects.

MAP Status: Full implementation

1.a)2) From a Department-wide perspective, there remain many projects where the project leader is either over- or under-ranked according to the new criteria in the Project Approval Directive. It was found in a recent sample of 80 active projects based on the project risk that 30 projects had project leaders who were over-ranked and 12 projects had under-ranked project leaders. Under-ranked project leaders may not have the experience to mitigate higher-risk projects, whereas over-ranked project leaders could face an excessive workload if they are leading too many projects; however, it was indicated that such deliberate appointments were made to mitigate the risk associated with particular projects. For future submissions, VCDS project analysts will continue to challenge (through the chain of command) project leader rank appointments that are not aligned with the Project Approval Directive.

OPI: VCDS/C Prog

Target Date: Ongoing

MAP Status: Insignificant progress

1.a)3) Regarding the appointment of ADM(Mat) as the functional lead in project management, DAOD 1000-1 is under review by the Corporate Secretary as part of the Defence Renewal Team initiative to update all DAODs and will be complete in 2014. This DAOD provides a management framework for 11 Level One managers who are accountable for certain functional areas across the Department. To appoint a lead to address a project management training shortfall observed in the CBRN Defence Omnibus audit, the intent is to formally appoint ADM(Mat) as the OPI for project management. This role has already been assumed by ADM(Mat) through the recent development of the Project Management Competency Development initiative for all types of projects.

OPI: Corporate Secretary

Target Date: December 2014

MAP Status: Preparation for implementation

1.b) The CBRN Omnibus project now includes five active projects; also, two new projects are pending approval. Not all of the CBRN Defence projects are governed under the Omnibus project. CFD directed the project office to include only CFD-sponsored projects as part of the Omnibus project. Therefore, the CBRN projects sponsored by the Chief of Military Personnel such as the Casualty Protection System and the Biological Diagnostics System are not within the scope of CBRN Defence Omnibus Project.

MAP Status: Full implementation

Management Action Plan Status

The scope of the CBRN Defence Omnibus Project is now appropriate as well as the ranks of the sub-projects’ project leader. However, the VCDS/C Prog action plans (1.a)2) and 1.a)3) regarding project leader ranks and the functional lead for project management have not progressed.

Operational Requirements

Management Action Plan and AERMS Status

Until recently, projects followed the 2002 Capability-based Planning instructions (Canadian Joint Task List and Force Planning Scenarios) and were adjusted by the requirements of DND operational tasks. Over the life of the project, Vital Point Bio Sentry saw three different Defence policies and a substantial downsizing of the Canadian Armed Forces (CAF). Therefore, the baseline has been a moving target. In the future, Sections 1 and 2 of the Statement of Requirements will reflect the changes in Defence Policy as well as the structure and concept of employment of the CAF as they come into force. The Vital Point Bio Sentry Project purchased the full operational requirement for the CAF as understood at that time but will consider additional procurements in response to evolving operational requirements. The CBRN Omnibus Project Director Handbook will expand on this issue and standardize the approach as required.

OPI: VCDS/CFD

Target Date: September 2008

AERMS Status: No information

Assessment of Progress by CRS

The CBRN Project Director Handbook that was revised in June 2013 includes guidelines to define the baseline requirements of projects. An amendment to the Vital Point Bio Sentry project was considered but full requirements were being met with existing detectors.

MAP Status: Full implementation

Management Action Plan Summary Status

Full implementation.

Schedule

Management Action Plan and AERMS Status

3.a) This is an ongoing process issue that VCDS, with ADM(Fin CS), are dealing with. A new Program Guidance Memorandum will be released soon and the new process instituted as quickly as possible.

OPI: VCDS/C Prog

Target Date: December 2009

AERMS Status: Full implementation

3.b)1) Director Material Group Strategic Plan (DMGSP) will examine the feasibility of negotiating specific service-level standards with PWGSC, in consultation with key stakeholders from Director Material Policy and Procedures (DMPP) and PWGSC. Discussions will take place to examine the requirement to include a reference for negotiations within the Procurement Administration Manual and the Project Approval Guide.

OPI: ADM(Mat)/DMGSP

Target Date: December 2008 (revised October 2012)

AERMS Status: Full implementation

3.b)2) The Omnibus Project Manager agrees with this option and plans to co-locate PWGSC contracting officers.

OPI: ADM(Mat)/ Director General Land Equipment Program Management (DGLEPM)

Target Date: December 2008

AERMS Status: Full implementation

Assessment of Progress by CRS

3.a) A Project Approval Process Redesign Working Group was initiated at the direction of the VCDS in April 2012, with the objective of doubling the throughput of the process while reducing the cycle time by half. The working group mapped the current process, and developed alternatives to meet the objective. The results were briefed to the Project Approval Process Redesign Council in May 2013. The new approval process for a sample of projects has begun, and will be completed by April 2016.

OPI: VCDS/C Prog

Target Date: April 2016

AERMS Status: Preparation for implementation

3.b)1) A Department-level DND/PWGSC Memorandum of Understanding has been established since 2011 to facilitate the central oversight of the cost of PWGSC revenue dependency fees. The CBRN Defence Omnibus Project has been included in the annual revenue dependency estimates. A chapter on managing revenue dependency fees has been included in the DND Procurement Administration Manual.

Although the Memorandum of Understanding of 2011 between PWGSC and DND has established revenue dependency fees, it does not provide a performance management framework, a requirement in the Treasury Board policy on Common Service Providers (last updated 2006-10-26). PWGSC has developed contract award time target for contracts less than $1 million in value. However, these targets are now under revision in order to link performance levels to contract complexity by 2014. DND will follow PWGSC’s lead with respect to their performance metrics development. Negotiations with PWGSC have been suspended until the completion of the government-wide military procurement review.

OPI: ADM(Mat)/ Director General Material Systems and Supply Chain (DGMSSC)/DMPP

Target Date: December 2015

MAP Status: Planning stage

3.b)2) The PWGSC contracting authority is now co-located with the project office.

MAP Status: Full implementation

Management Action Plan Summary Status

While progress has been made in the co-location of the contracting authority with the CBRN Defence Omnibus project, work remains to be done to fully address the recommendation. Two of the MAPs (3.a) and 3.b)1)) regarding the project approval process and PWGSC performance standards are only partially complete.

Contract Management

Management Action Plan and AERMS Status

4.a)1) This is a lesson learned that will be passed to other CBRN projects by Director Combat Support Equipment Management.

OPI: ADM(Mat)/DGLEPM

Target Date: September 2008

AERMS Status: Full implementation

4.a)2) Contract clarifications/negotiations will be identified as a Work Breakdown Structure activity prior to contract signature and it will be accounted for in-project timelines and funding. The requirement for DND to participate in contract clarification/negotiation will be addressed in the Service-level Agreement between ADM(Mat) and PWGSC. This lesson learned will be recorded in the CID.

OPI: ADM(Mat)/DGLEPM

Target Date: September 2008

AERMS Status: Full implementation

4.b) Project management course material is being reassessed as part of the Material Acquisition and Support Learning Strategy and the Project Management Competency Development initiative. Negotiation skills have been identified as a required competency for project managers.

OPI: ADM(Mat)/DGMSSC

Target Date: March 2009

AERMS Status: Full implementation

Assessment of Progress by CRS

4.a)1) Lessons learned from the Vital Point Bio Sentry project have been documented in the CID and indicate that the lack of DND participation in contract negotiations resulted in a misunderstanding of the requirements by the vendor.

MAP Status: Full implementation

4.a)2) Service levels were to be established in a Memorandum of Understanding between DND and PWGSC.⁵ This Memorandum of Understanding refers to the responsibility matrix in the PWGSC Supply Manual, Annex 1.1.1 which allows DND participation in negotiations pertaining only to sole-sourced contracts. There remains a need for DND to participate in competitive contract amendments and annual rate negotiations in order to provide the necessary technical expertise. Negotiations with PWGSC have been suspended until the completion of government-wide military procurement review.

OPI: ADM(Mat)/DGMSSC/DMPP

Target Date: December 2014

MAP Status: Insignificant progress

4.b) The pilot courses of the Project Management Competency Development have been completed and this certification program includes negotiation skills.

MAP Status: Full implementation

Management Action Plan Summary Status

Contract negotiation skills are now part of a DND certification program and the CBRN Defence Omnibus project lessons learned on negotiations have been recorded by the project office. One management action plan (4.a)) regarding DND participation in contract negotiations has had insignificant progress.

Options Analysis

Management Action Plan and AERMS Status

5.a) An instruction for the conduct of options analysis by sub-projects has been developed and published in the CBRN Omnibus Project Director Handbook. All new projects will require a Synopsis Sheet (Identification) complete with costing data.

OPI: VCDS/CFD

Target Date: March 2008

AERMS Status: Full implementation

5.b) The significance of the options analysis and a business case approach to reporting the results of the options analysis needs to be reinforced. We will take the necessary steps in the Project Approval Guide revision to ensure "Omnibus" projects have only one Synopsis Sheet (Identification) that is revised for new sub-projects.

OPI: VCDS/C Prog

Target Date: December 2009

AERMS Status: Full implementation

Assessment of Progress

5.a) New CBRN Defence Omnibus sub-project submitted a Synopsis Sheet (Identification), now known as Project Brief (Identification) over the audit follow-up time frame.

Options analysis instructions for the CBRN Omnibus Project staff were developed, as indicated in the action plan. The CBRN Defence Omnibus project plans to perform a detailed business case analysis for the upcoming two new projects.

OPI: VCDS/CFD

MAP Status: Substantial implementation

5.b) The Project Approval Directive, effective October 2011, does not indicate if a revised Omnibus Project Brief (Identification) is required for every sub-project. The Project Approval Directive will clarify which approval document should contain the options to be analyzed for Omnibus sub-projects. However, the Project Approval Directive includes clear instructions for a complete options analysis, including cost-benefit analysis of viable options. Furthermore, the DND Costing Handbook of April 2006 also provides a step-by-step approach to complete a business case analysis.⁶

OPI: VCDS/C Prog

Target Date: April 2014

AERMS Status: Preparation for implementation

Management Action Plan Summary Status

The CBRN Defence Omnibus project office has addressed the need for a separate Project Brief (Identification) and cost options analysis for each sub-project. However, the revision of the Project Approval Directive to ensure a Project Brief (Identification) for each Omnibus sub-project has not been completed.

Information for Decision Making

6.b) Ensure that the Project Office link key vendor reports to payments and holdbacks in future contracts.

Management Action Plan and AERMS Status

6.a)1) Documents are being uploaded once they are approved. The CBRN Omnibus Project Director Handbook will expand its instruction on the use of the CID.

OPI: VCDS/C Prog

Target Date: September 2008

AERMS Status: Full implementation

6.a)2) C Prog does indeed require complete approval documentation for all activities seeking a new expenditure authority. The CBRN project will need to comply.

OPI: VCDS/C Prog

Target Date: Ongoing

AERMS Status: Full implementation

6.a)3) The sustaining support to the CID has suffered from funding constraints, and its future will be examined as part of the Investment Plan’s lessons learned.

OPI: VCDS/C Prog

Target Date: December 2010

AERMS Status: Preparation for implementation

6.b)1) Procurement and finance awareness sessions are mandatory for incoming ADM(Mat) personnel. These sessions promote payment based on deliverables or, when not feasible, payment based on progress reports showing evidence of value for money.

OPI: ADM(Mat)/DGMSSC/DMPP

Target Date: As required

AERMS Status: Full implementation

6.b)2) The Director Combat Support Equipment Management agrees with the observation. The problem can best be addressed during contract clarifications/negotiations.

OPI: ADM(Mat)/DGLEPM

Target Date: As required

AERMS Status: Full implementation

Assessment of Progress by CRS

6.a)1) The CBRN Project Director Handbook, updated in June 2013, specifies the need to keep the CID updated periodically. Improvements were evident in project documents being uploaded to the CID soon after being signed off.

MAP Status: Full implementation

6.a)2) According to VCDS Directives, the project office and the VCDS project analysts are to ensure that key documents are up to date prior to any project expenditure approval.

MAP Status: Full implementation

6.a)3) The CID data replication reliability issues have not been resolved due to plans for incorporation of CID functionality in Defence Resource Management Information System. This recommendation should be closed.

MAP Status: Obsolete

6.b)1) “Basic Procurement and Contracting,” “Complex Procurement,” and “Finance Management for Procurement” were courses offered by the Material Management Training Centre.⁷ Although the course description does not identify that it is mandatory for all ADM(Mat) staff, the project management staff has Personal Learning Plans that includes this training on a need-to-know basis.

MAP Status: Full implementation

6.b)2) For CBRN Defence sub-projects, improvement is required in linking payments to vendor reports. Of the seven contracts that were initiated and/or amended after the initial May 2008 audit, only one contract had payments tied to its quarterly third-line maintenance service activity report. For current contracts, the project office is linking payments to vendor reports.

MAP Status: Full implementation

Management Action Plan Summary Status

The management action plans have been completed.

Risk Management

Management Action Plan and AERMS Status

7.a)1) The project team will consult more closely with both ADM(Fin CS) and Director Force Planning and Program Coordination (now known as Director Defence Program Coordination) to make sure there is a better correlation between a project risk level and the amount set aside for contingency.

OPI: VCDS/CFD

Target Date: April 2008

AERMS Status: Full implementation

7.a)2) The Project Management Working Group is developing standards for risk management, including the quantification and reporting of risks. Furthermore, the Project Management Working Group was tasked by Program Management Board to develop risk management process and reporting requirements with a goal to having an integrated approach to risk management in the Department. The CBRN Omnibus project will conform to the revised corporate process.

OPI: ADM(Mat)/DGMSSC

Target Date: March 2009

AERMS Status: Full implementation

7.b)1) The CBRN Omnibus project has replaced its own risk management direction with the ADM(Mat) Risk Instruction.

OPI: VCDS/CFD

Target Date: January 2008

AERMS Status: Full implementation

7.b)2) We will seek to ensure improved alignment with the VCDS directives on Risk and Performance Management/Measurement in the Project Approval Guide revision.

OPI: VCDS/C Prog

Target Date: December 2009

AERMS Status: Full implementation

Assessment of Progress by CRS

7.a)1) It has not been the practice with CBRN Defence Omnibus sub-projects to quantify their risk impact and mitigation costs to link with contingency funds prior to cost validation by the Director Costing Services (D Cost S). The project office staff does, however, recommend an overall risk level and a contingency to D Cost S based on their best judgement considering the open risks at the time. D Cost S is the final arbitrator in setting the amount of the contingency. The project office works very closely with the D Cost S staff during the cost validation process, and intends to use the Project Approval Directive risk register to correlate project risks to contingency for two upcoming new projects. The October 2011 Project Approval Directive risk register is much more comprehensive than the standard risk registers that have been used for some time, and will greatly assist in closing this gap.

OPI: VCDS/CFD

Target Date: December 2013

MAP Status: Preparation for implementation

7.a)2) The Project Management Working Group has developed a risk management tool within the Defence Resource Management Information System. This tool requires some improvements but would help in standardizing templates for effective risk management within the Department. Moreover, the use of the Defence Resource Management Information System tool will aid in linking risk analysis to contingency funds.

MAP Status: Full implementation

7.b)1) The June 2013 CBRN Defence Omnibus Project Director Handbook reflects the VCDS risk management guidance.

MAP Status: Full implementation

7.b)2) The VCDS issued guidance on corporate risk management and it was incorporated into the Project Approval Directive in October 2011.

MAP Status: Full implementation

Management Action Plan Summary Status

ADM(Mat) and the project office have aligned their risk management process with the DND guidelines. As well, a new ADM(Mat) risk management tool in the Defence Resource Management Information System should assist in the management of risk. The MAP (7.a)1)) regarding the linkage of risk to contingency fund has not been fully implemented.

___________________________________________________________________________________________________________________________

Footnote 1 Policy on Internal Audit http://www.tbs-sct.gc.ca/pubs_pol/dcgpubs/ia-vi/ia-vi_e.asp.

Footnote 2 The Project Approval Guide was superseded by the Project Approval Directive in October 2011.

Footnote 3 AERMS is a system that tracks the progress of the MAP items.

Footnote 4 Two MAPs did not have information regarding status in AERMS; upon further review, one was found to be complete while the other could not progress due to funding constraints.

Footnote 5 Memorandum of Understanding, 11 May 2011.

Footnote 6 Program Guidance Memorandum 01/13 is replacing Project Brief (Identification) with a Strategic Context Document.

Footnote 7 “MA & S Training,”.