Follow-up on Internal Audit of the Joint Support Ship Project

Caveat

The result of this work does not constitute an audit of the Joint Support Ship (JSS) project. Rather, this report was prepared to provide reasonable assurance that the Management Action Plans (MAPs) that resulted from the 2011 audit were implemented as stated and as such have addressed the associated recommendations.

Acronyms and Abbreviations
1.0 Introduction
2.0 Objective
3.0 Scope and Methodology
- 3.1 Approach
- 3.2 Statement of Conformance
4.0 Overall Assessment
Annex A—Assessment Criteria
Annex B—Management Action Plan Scorecard
Annex C—Detailed Assessment of Management Action Plan Progress

Alternate Formats

PDF Version
(188 KB, 18 pages)

Assistant Deputy Minister (Review Services)

1.0 Introduction

In keeping with policy and standards for Internal Audit,^{Footnote 1} Assistant Deputy Minister (Review Services) (ADM(RS)) monitors that management action plans (MAPs) have been effectively implemented in response to previous ADM(RS) audit recommendations. In addition to reporting twice per year to the Departmental Audit Committee on the status of MAP completion, ADM(RS) conducts detailed follow-ups on selected audits based on risk. In accordance with the ADM(RS) Risk-Based Audit Plan for Fiscal Year 2017/18 to 2019/20, this audit follow-up was conducted to assess the progress made towards the implementation of the MAP from the 2011 Internal Audit of the Joint Support Ship (JSS) Project.

The Royal Canadian Navy Auxiliary Oiler Replenishment vessels, operated by the Royal Canadian Navy for over 40 years, supported several ships or submarines at sea for up to 30 days of operations. The Auxiliary Oiler Replenishment ships resupplied other ships with fuel, water, rations, and spares, and also carried three maritime helicopters. The last ship in the Auxiliary Oiler Replenishment fleet was retired from service in 2016.

To replace the Auxiliary Oiler Replenishment fleet, the definition phase of the JSS project was originally approved in November 2004 with a total project cost of $1.99 billion for the purchase of three JSS. A revised definition phase was approved in June 2010 with a project cost estimated at $2.33 billion for two JSS with an option for a third.

Subsequent to the original audit, the introduction of the new Defence Policy: Strong, Secure, Engaged, has established the need for two JSS to allow for “a fleet built around an ability to deploy and sustain two naval task groups, each composed of up to four combatants and a joint support ship.”^{Footnote 2} While the project schedule and costs are currently being reassessed, the two ships are estimated to be in service the first half of the next decade.^{Footnote 3}

The scope of the 2011 Internal Audit of the JSS Project included the JSS project from inception, with a focus on current and future planned activities from 2010 onward. The scope did not include the performance of the design contractors, as this is the audit responsibility of Public Works and Government Services Canada (now Public Services and Procurement Canada). The audit conduct phase was performed between October 2010 and April 2011.

The 2011 internal audit found opportunities for improvement, of which the most significant are shown below:

Capability Cost Trade-Off. The original audit found that the JSS statement of operational requirements did not provide sufficient information for decision makers to show the operational impact if a fleet of two JSS were acquired.

It was recommended that in consultation with Assistant Deputy Minister (Materiel) (ADM(Mat)), the Royal Canadian Navy should amend the statement of operational requirements to specify the operational risks associated with the JSS fleet size to ensure sufficient information would be available should it become possible to exercise the contract option.

Project Schedule. The original audit found that the Project Management Office (PMO) did not maintain and update a comprehensive project schedule that allocated and tracked personnel by task to reduce the risk of delay to task completion; the overall delivery schedule of the project was dependent upon the sequencing of the JSS project with other ship projects within the Canadian Coast Guard.

It was recommended that the JSS PMO allocation of human resources be completed with appropriate productivity settings and that ADM(Mat) take the necessary steps to ensure the optimum scheduling of JSS within the National Shipbuilding Procurement Strategy (NSPS).

Contract Management. The original audit found that the Request For Proposals for the design and future construction of the JSS would benefit from increased subcontract visibility, linking terms of payment to milestones for key deliverables and requiring more information in vendor-provided reports, including better cost estimate and schedule information.

It was recommended that the NSPS contract negotiations for the JSS build include contract terms and conditions that address the observations in the audit report.

It was also recommended that the JSS PMO revise the Request For Proposals for the design contracts to improve terms of payment, vendor report content, the statement of work, and consider similar improvements to other contracts in the project.

Back to Table of Contents

2.0 Objective

The objective of this audit follow-up is to assess the level of progress made by the Department of National Defence (DND)/Canadian Armed Forces (CAF) in implementing the MAP from the 2011 Internal Audit of the JSS Project.

3.0 Scope and Methodology

3.1 Approach

This audit follow-up examines whether the issues identified in the 2011 internal audit have been addressed by assessing the progress made on implementing the MAP based on the assessment criteria in Annex A. It was not a re-performance of the original audit. The following methods were used:

Review of supporting documentation; and
Interviews with key project personnel.

Prior to this audit follow-up, three of the 10 MAP items from the original audit were validated by ADM(RS) as being fully implemented. These were, therefore, not re-assessed as part of this follow-up and are shown in Annex B.

The audit follow-up considered information received up to October 2017.

3.2 Statement of Conformance

The audit findings and conclusions contained in this report are based on sufficient and appropriate audit evidence gathered in accordance with procedures that meet the Institute of Internal Auditors’ International Standards for the Professional Practice of Internal Auditing. The audit thus conforms to the Institute of Internal Auditors’ International Standards for the Professional Practice of Internal Auditing, as supported by the results of the quality assurance and improvement program. The opinions expressed in this report are based on conditions as they existed at the time of the audit, and apply only to the entity examined.

Back to Table of Contents

4.0 Overall Assessment

The follow-up found that significant progress was made by the DND/CAF on implementing the MAPs developed in response to the 2011 Internal Audit of the JSS Project. Most MAP items have been fully implemented.

Of the seven MAP items assessed during this follow-up, four were deemed to be fully implemented. Two MAP items were assessed to be obsolete, and one MAP item was assessed to be in the preparation for implementation stage.

The two MAP items assessed as obsolete relate to the project schedule and contract management. The circumstances relating to the project schedule have changed since the recommendations were issued. Specifically, the project schedule, which previously consisted of PMO-led activities in the early stages of the project, is now focused on contractor-led activities. For this reason, the attributes of that recommendation are no longer relevant. The recommendations involving contract management were for specific definition contracts which are no longer in place since the project has advanced to a stage where those specific services are no longer required.

Lastly, the implementation of one MAP was still in progress and could not be assessed as fully complete as it relates to the inclusion of specific terms and conditions in the build contract, which has not yet been finalized. A review of other current definition phase contracts was performed, and the audit follow-up noted significant progress made with the inclusion of the necessary clauses in the definition phase contracts currently used by the PMO. However, until the future implementation build contract is in place with clauses to address risks raised in the original audit, the MAP cannot be considered complete.

The PMO has committed to a revised due date of November 2019 for the finalization of the MAP as it coincides with their current project approval implementation timeline.

Should ADM(Mat) continue to use clauses as seen in current definition contracts, the related risks in the original audit will be minimized.

A scorecard of the MAP items can be found in Annex B, and a more detailed assessment of progress can be found in Annex C.

Back to Table of Contents

Annex A—Assessment Criteria

The following criteria were used to assess the level of completion for each MAP item:

1. Obsolete or Superseded
Audit recommendations that are deemed to be obsolete or have been superseded by another recommendation.

2. No Progress or Insignificant Progress (0-24% complete)
No action taken by management or insignificant progress. Actions such as striking a new committee, having meetings and generating informal plans are insignificant progress.

3. Planning Stage (25-49% complete)
Formal plans for organizational changes have been created and approved by the appropriate level of management (at a sufficiently senior level, usually at the Executive Committee level or equivalent) with appropriate resources and a reasonable timetable.

4. Preparation for Implementation (50-74% complete)
The entity has begun necessary preparation for implementation, such as hiring or training staff, or developing or acquiring the necessary resources to implement the recommendation.

5. Substantial Implementation (75-99% complete)
Structures and processes are in place and integrated in some parts of the organization, and some achieved results have been identified. The entity has a short-term plan and timetable for full implementation.

6. Full Implementation (100% complete)
Structures and processes are operating as intended and are implemented fully in all intended areas of the organization.

Back to Table of Contents

Annex B—Management Action Plan Scorecard

Table B-1. Management Action Plan Scorecard. This table shows the ADM(RS) assessment of progress on the MAP.
Recommendation #	MAP		OPI	ADM(RS) Assessment of Progress
1. Capability Cost Trade-Off	1.1 The operational impact and availability of JSS in relation to the number of ships will be articulated in the next revision of the statement of operational requirements to be presented and endorsed at the annual Senior Review Board meeting in 2012.		Commander Royal Canadian Navy	Full Implementation
2. Project Schedule	2.1 ADM(RS) project scheduling recommendations will be adopted. The JSS PMO will update its Project Master Schedule to ensure that all identified Project Definition activities have appropriate resources assigned with realistic productivity considerations and levels.		ADM(Mat)	Obsolete or Superseded
2. Project Schedule	2.2 The JSS PMO will ensure the NSPS Secretariat maintains the issue of non-combat work package sequencing as an open action item within its DND/Canadian Coast Guard Project Action Log. This will promote a regular dialogue on this schedule risk between NSPS, DND, Department of Fisheries, and the Canadian Coast Guard on the optimal scheduling of the affected projects. Finally, given the interdepartmental significance, visibility into this risk has been elevated to the Major Crown Project-Interdepartmental Oversight Committee level.		ADM(Mat)	Full Implementation^{Footnote 4}
3. Contract Management	3.1 The recommendations proposed by ADM(RS) in support of contract management will be taken into consideration when developing future contracts in support of JSS, specifically the build contract with the designated NSPS shipyard.		ADM(Mat)	Preparation for Implementation
4. Contract Management	4.1 ADM(RS) recommendations concerning vendor reporting and statement of work clarity have been implemented in both the Military Off The Shelf and New Design procurement documents. Terms of payment are presently being negotiated to ensure payments will be in line with actual work (milestones) completed.		ADM(Mat)	Obsolete or Superseded
5. Human Resources Management	5.1 Project Management Support Office, in consultation with Director Materiel Group Operational Research Acquisition Support Team, will undertake to develop a PMO staffing model to supplement the existing departmental methodology and guidance.		ADM(Mat)	Full Implementation^{Footnote 5}
5. Human Resources Management	5.2 The JSS PMO has drafted a Human Resource Management Plan that reflects ADM(RS) recommendations to better address succession planning and surge requirements within the project. The Human Resource plan is currently under review and will be promulgated when ready.		ADM(Mat)	Full Implementation
6. In-Service Support Strategy	6.1 Linking vendor performance to exercising an In-Service Support contract option will be considered in the next In-Service Support Statement of Work and In-Service Support Pro-Forma contract revisions. The criteria, metrics and associated penalties tied to maintenance deferral by the vendor will also be considered and will be articulated in detail in the above-mentioned revised documents.		ADM(Mat)	Full Implementation^{Footnote 6}
7. Risk Management	7.1 ADM(RS) risk management recommendations have been incorporated into the project’s draft Risk Management Plan. This revision of the Risk Management Plan is presently under review and will be promulgated in the fall of 2011. Updates include improved definitions for risk impact levels and the adoption of the five levels of risk severity, in accordance with the DND Integrated Risk Management policy.		ADM(Mat)	Full Implementation
7. Risk Management	7.2 The JSS PMO will rescore all project risks in light of the new impact definitions and complete a risk quantification exercise using Project Management Body Of Knowledge’s Expected Monetary Value practices to assess the adequacy of the project’s existing risk mitigation and contingency levels.		ADM(Mat)	Full Implementation
Legend
Obsolete or Superseded	No Progress or Insignificant Progress	Planning Stage	Preparation for Implementation	Substantial Implementation	Full Implementation

Table B-1 Details – MAP Scorecard

Recommendation 1

Table C-1. Progress in Implementing the MAP for Recommendation 1. This table shows the ADM(RS) assessment of recommendations concerning amendments to the Statement of Operational Requirements.
MAP	OPI	Target Date	Progress to Date		Assessment
1.1 The operational impact and availability of JSS in relation to the number of ships will be articulated in the next revision of the statement of operational requirements to be presented and endorsed at the annual Senior Review Board meeting in 2012.	Commander Royal Canadian Navy	November 2012	In 2012 the JSS Statement of Operational Requirements was updated to address the following three issues: how operations would be impacted under scenarios of various JSS availability; further information to clarify the operational risk to the fleet under these scenarios; and information acknowledging that challenges exist to meet full capability requirements within the assigned budget. Subsequent to the original audit of JSS, the new Defence Policy: Strong, Secure, Engaged (SSE) specified two JSS. The PMO has updated its communication products accordingly.		Full Implementation
ADM(RS) Assessment of Progress on MAP: Full Implementation
Obsolete or Superseded	No Progress or Insignificant Progress	Planning Stage	Preparation for Implementation	Substantial Implementation	Full Implementation

Table C-1 Details – Status of the Implementation of the MAP Item for Recommendation 1

Recommendation 2

Table C-2. Progress in implementing the MAP for Recommendation 2. This table shows the ADM(RS) assessment of recommendations concerning project scheduling.
MAP	OPI	Target Date	Progress to Date		Assessment
2.1 ADM(RS) project scheduling recommendations will be adopted. The JSS PMO will update its Project Master Schedule to ensure that all identified Project Definition activities have appropriate resources assigned with realistic productivity considerations and levels.	ADM(Mat)	November 2011	Since the original audit, roles regarding the project schedule have evolved. The Shipyard Contractor now manages the majority of the tasks that make up the greater project schedule, as opposed to the PMO. The PMO now monitors and provides feedback on the Shipyard Contractor’s Integrated Master Schedule. In particular, the PMO is monitoring and completing several tasks related to the Integrated Master Schedule with the Shipyard Contractor. These actions include: reviewing/contributing to the schedule created and managed by the Shipyard Contractor, including visibility into the Shipyard Contractor allocation of human resources; and managing the review and approval of JSS deliverables from the Shipyard Contractor through a detailed tracking sheet that ensures PMO staff know when to expect a deliverable and expected timeframes for PMO review and feedback; working collaboratively with the contractor to determine acceptable durations for the completion of milestones and events; and lastly, by conducting regular communication with the contractor through various methods such as monthly progress reports, teleconferences and meetings. Due to these changes, this MAP is obsolete and the original risk of project definition activities being delayed due to insufficient schedule controls within the PMO is no longer a significant concern. The risk has largely been transferred to the monitoring of the Shipyard Contractor’s schedule.		Obsolete or Superseded
ADM(RS) Assessment of Progress on MAP: Obsolete or Superseded
Obsolete or Superseded	No Progress or Insignificant Progress	Planning Stage	Preparation for Implementation	Substantial Implementation	Full Implementation

Table C-2 Details – Status of the Implementation of the MAP Item for Recommendation 2

Recommendation 3

Table C-3. Progress in implementing the MAP for Recommendation 3. This table shows the ADM(RS) assessment of recommendations concerning contract terms and conditions.
MAP	OPI	Target Date	Progress to Date		Assessment
3.1 The recommendations proposed by ADM(RS) in support of contract management will be taken into consideration when developing future contracts in support of JSS, specifically the build contract with the designated NSPS shipyard.	ADM(Mat)	Effective Project Approval	The original audit included a review of the draft NSPS Request For Proposal and identified areas for improvement with respect to contract clauses. It was recommended that ADM(Mat) include these clauses in the JSS implementation build contract. Although the implementation build contract has not been finalized, considerable progress has been made in addressing original concerns in two of the current definition phase contracts – the Design and Production Engineering and Long Lead Items contracts. The following progress was observed: Better visibility of principal subcontractors was noted for these contracts and one contract included the requirement for subcontract competition. In addition, the discretionary audit clause was required for all subcontracts where it added value. Both contracts required a ‘most favoured customer’ certification by stipulating that subcontracted rates cannot be higher than the best rates provided to others for like quantity and quality. The concern over the timing of payments to the Shipyard Contractor for Canada’s share of shipyard improvement costs has been resolved. The two contracts stated that either party could seek resolution from the courts at any time during the dispute resolution process as recommended in the original audit. A clause regarding liquidated damages was included in both contracts for Industrial and Regional Benefits policy commitments, but not to protect against delay. Much progress has been made, however until the future implementation build contract is in place with clauses to address risks raised in the original audit, this MAP cannot be considered complete. The PMO has agreed to a revised due date of November 2019 for the finalization of the MAP – current expected Project Approval – Implementation date.		Preparation for Implementation
Revised target date: November 2019
ADM(RS) Assessment of Progress on MAP: Preparation for Implementation
Obsolete or Superseded	No Progress or Insignificant Progress	Planning Stage	Preparation for Implementation	Substantial Implementation	Full Implementation

Table C-3 Details – Status of the Implementation of the MAP Item for Recommendation 3

Recommendation 4

Table C-4. Progress in implementing the MAP for Recommendation 4. This table shows the ADM(RS) assessment of recommendations concerning vendor reporting and Statement of Work clarity.
MAP	OPI	Target Date	Progress to Date		Assessment
4.1 ADM(RS) recommendations concerning vendor reporting and Statement of Work clarity have been implemented in both the Military Off the Shelf and New Design procurement documents. Terms of payment are presently being negotiated to ensure payments will be in line with actual work (milestones) completed.	ADM(Mat)	March 2011	The recommendation and associated MAP 4.1 had many elements that were specific to two 2011 definition phase contracts. These specific definition phase contracts are no longer in place as the scope of work has been completed. This MAP is therefore obsolete. Nonetheless, the follow-up examined the progress made before the obsolescence and observed that some progress had been made: The terms of payment were updated to link milestones to key deliverables and a time verification clause was added to one of the two contracts. Additional schedule information was requested from the contractor as was recommended in the original audit. Secondly, a cost confidence level to support the strength of the contractor’s cost estimate was included in one of the two contracts as recommended. Finally, the PMO requested the contractor(s) provide a cost estimate for a third ship, which was included in one of the two contracts. Recommended clarity in language in the contract’s statement of work was included as recommended.		Obsolete or Superseded
ADM(RS) Assessment of Progress on MAP: Obsolete or Superseded
Obsolete or Superseded	No Progress or Insignificant Progress	Planning Stage	Preparation for Implementation	Substantial Implementation	Full Implementation

Table C-4 Details – Status of the Implementation of the MAP Item for Recommendation 4

Recommendation 5

Table C-5. Progress in implementing the MAP for Recommendation 5. This table shows the ADM(RS) assessment of recommendations concerning HR planning.
MAP	OPI	Target Date	Progress to Date		Assessment
5.2 The JSS PMO has drafted a Human Resources Management Plan that reflects ADM(RS) recommendations to better address succession planning and surge requirements within the project. The Human Resources plan is currently under review and will be promulgated when ready.	ADM(Mat)	February 2012	The Human Resources Plans from fiscal year 2011/2012 and 2017 (current version) include a section on succession management. It describes how the JSS PMO will identify positions that require added attention, and the activities that can be completed to mitigate possible future turnover of key personnel. The fiscal year 2011/2012 Human Resources Plan includes sharing of resources from Director General Maritime Equipment Program Management and the Engineering Logistic Management Support contractor to handle surges during the Project Definition phase. Further, the 2017 Memoranda of Understanding with the Canadian Coast Guard establishes how both organizations are expected to manage their various responsibilities required for the review work for project inspections.		Full Implementation
ADM(RS) Assessment of Progress on MAP: Full Implementation
Obsolete or Superseded	No Progress or Insignificant Progress	Planning Stage	Preparation for Implementation	Substantial Implementation	Full Implementation

Table C-5 Details – Status of the Implementation of the MAP Item for Recommendation 5

Recommendation 7

Table C-6. Progress in implementing the MAP for Recommendation 7. This table shows the ADM(RS) assessment of recommendations concerning the Risk Management Plan Revisions.
MAP	OPI	Target Date	Progress to Date		Assessment
7.1 ADM(RS) risk management recommendations have been incorporated into the project’s draft Risk Management Plan. This revision of the Risk Management Plan is presently under review and will be promulgated in the fall of 2011. Updates include improved definitions for risk impact levels and the adoption of the five levels of risk severity, in accordance with the DND Integrated Risk Management Policy.	ADM(Mat)	December 2011	The updated Risk Management Plan addressed the four risk management observations from the audit report. The Risk Management Plan and relevant documents included: the development of the criteria for five threshold levels of risk (in accordance with the DND Integrated Risk Management Policy and Guidelines); the severity of the risk scale was modified to be more aligned with the impact/probability of the risks; improved reporting of each project’s respective top risks; and the adoption of the Project Management Body Of Knowledge Expected Monetary Value practice when quantifying risks.		Full Implementation
ADM(RS) Assessment of Progress on MAP: Full Implementation
Obsolete or Superseded	No Progress or Insignificant Progress	Planning Stage	Preparation for Implementation	Substantial Implementation	Full Implementation

Table C-6 Details – Status of the Implementation of the first MAP Item for Recommendation 7

Recommendation 7

Table C-7. Progress in implementing the MAP for Recommendation 7. This table shows the ADM(RS) assessment of recommendations concerning risk rescoring.
MAP	OPI	Target Date	Progress to Date		Assessment
7.2 JSS PMO will rescore all project risks in light of the new impact definitions and complete a risk quantification exercise using Project Management Body of Knowledge’s Expected Monetary Value practices to assess the adequacy of the project’s existing risk mitigation and contingency levels.	ADM(Mat)	March 2012	All project risks in the updated risk register were assessed using the Expected Monetary Value method in accordance with the Project Management Body of Knowledge and had a quantitative measure associated with it. While there was no specific evidence of the PMO assessing the totals against their contingency amounts, they did calculate the total Expected Monetary value. The contingency available to the project, when compared to the total Expected Monetary value, should be sufficient to mitigate the risks.		Full Implementation
ADM(RS) Assessment of Progress on MAP: Full Implementation
Obsolete or Superseded	No Progress or Insignificant Progress	Planning Stage	Preparation for Implementation	Substantial Implementation	Full Implementation