Internal Audit of Medium Support Vehicle System (MSVS) Project

March 2014

7050-11-47 (CRS)

Reviewed by CRS in accordance with the Access to Information Act (AIA). Information UNCLASSIFIED.

Acronyms and Abbreviations

Results in Brief

In 2012, the Department of National Defence (DND) was directed to undertake an internal audit of the MSVS project prior to any further approval requests for future phases of the project’s equipment. Thus, the audit was added to the Chief Review Services (CRS) fiscal year (FY) 2013/14 to FY 2015/16 Risk Based Audit Plan. The objective of the audit was to assess the adequacy of the management control framework, governance processes and risk management strategy in place to ensure a cost-effective acquisition.

The project was announced by the government in June 2006 to replace the Medium Logistics Vehicle Wheeled (MLVW) fleet of ▓▓▓▓▓▓▓▓▓▓▓▓▓ vehicles. Contracts have been awarded for three of five phases – a ▓▓▓▓▓▓▓▓▓▓▓Military Commercial Off-the Shelf (MilCOTS) vehicle fleet, ▓▓▓▓▓▓▓ vehicle mounted shelters, and ▓▓▓▓▓▓▓ vehicle kits for the shelters. Two phases remain; Phase IV, consisting of a planned 1,500 Standard Military Pattern (SMP) fleet of logistics vehicles, and Phase V, for infrastructure in support of the project. The focus of the audit was on Phases IV and V.

Findings and Recommendations

Phase IV Standard Military Pattern Vehicle Scope. The full Canadian Armed Forces (CAF) requirement for ▓▓▓▓▓▓▓▓▓▓▓ SMP vehicles will not be met unless vehicle contract options are exercised. To control project costs, the SMP phase acquisition contract has been capped at $725 million and 1,500 vehicles. However, should there be sufficient funding sources, there will also be contract options for up to 650 additional SMP vehicles to address the expansion of the CAF that has taken place since 2006. The main reasons for the increase in SMP vehicle requirements are the growth of the Canadian Army and other new operational units.

It is recommended that Assistant Deputy Minister (Materiel) (ADM(Mat)) ensure that the MSVS project approval documentation for the SMP vehicle phase reflects the full CAF requirement, and to exercise the appropriate contract options should funding permit.

Governance – Information for Decision Making. As a result of a 2009 request for price and availability with the defence industry, the MSVS project office required additional funding for the Phase IV SMP vehicle purchase. The Department was advised to proceed with the project in October 2010, as there was sufficient policy coverage for the indicative cost growth. Despite receiving departmental approval and a funds transfer from the Logistics Vehicle Modernization project in 2009, the Major Crown Project Interdepartmental Oversight Committee (MCP-IOC) monthly reports did not reflect the change in project value until 2012, as the format of the report focused on externally approved indicative project costs instead of on the departmentally approved allocations. Although the combined cost of both vehicle projects was unchanged, the Department was directed in July 2012 to cancel the SMP fleet Request for Proposal (RFP) to industry due to cost growth resulting in an 18-month project delay. The MCP-IOC report format has now changed to show both departmental and external expenditure approvals. Assistant Deputy Minister (Finance and Corporate Services (ADM(Fin CS)) is working on a summary report of approved Investment Plan Change Proposals for the Treasury Board Secretariat to improve visibility over such matters. Policy changes are needed to provide guidance to multi-phase projects and other projects in definition phase regarding when they are required to seek revised indicative estimate approvals for future project phases. The effective management of the capital program will also require greater accuracy and more complete project information in the Capability Investment Database (CID), in-service support (ISS) estimates, and Senior Review Board (SRB) performance reports.

It is recommended that ADM(Mat) ensure that the MSVS project’s CID Monthly Progress Report, SRB Performance Report and the MCP-IOC documentation are kept up-to-date and complete. The Vice Chief of the Defence Staff (VCDS) should revise the Project Approval Directive (PAD) to clarify the timing of advising the approval authority of changes to the total indicative price for all projects, in particular multiple phase projects; and VCDS should take measures to ensure that project management offices update key information in the CID on a recurring basis.

Contracted In-Service Support. The limited funds available for contracted ISS costs for all wheeled logistic vehicle fleets ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ The MLVW fleet, which is being replaced by the MSVS SMP fleet, accounts for ▓▓▓▓▓ percent of the annual allocation for all wheeled logistic vehicle maintenance. The SMP fleet annual ISS indicative estimate of ▓▓▓▓▓ million, ▓▓▓▓▓ percent of the total allocation, is unaffordable. However, the planned usage rate of ▓▓▓▓▓▓▓▓▓▓▓ kilometres per year exceeds the historical usage rate, and has resulted in overstated contracted ISS cost estimates for the SMP fleet.

It is recommended that ADM(Mat) revise the contracted ISS cost estimates for the SMP fleet, based on the historical annual usage rate of other vehicle fleets, and take other measures to manage the constraints of limited sustainment funds.

Infrastructure – Phase V. The MSVS project delivered ▓▓▓▓▓▓▓▓▓▓▓ MilCOTS vehicles to the reserves by March 2011. However, due to delays in requirements definition and Phase V approval, the required infrastructure modifications were not put in place to coincide with vehicle delivery. Postponing this work to a single and final infrastructure phase, (Phase V), that has yet to receive approval, will further the delay of infrastructure modifications. Early engagement, initially with Director Land Infrastructure and, subsequently, with Assistant Deputy Minister (Infrastructure and Environment) (ADM(IE)), would have minimized the impact on the units and equipment. At the time of the audit, a consolidated Statement of Requirement (Infrastructure) (SOR(I)) for over ▓▓▓▓▓ armouries was not complete or coordinated by the project office. Despite the uncertainty of the anticipated modifications, the estimate for infrastructure was significantly reduced in October 2013, from ▓▓▓▓▓▓▓million to ▓▓▓▓▓ million.

In order to avoid similar delays in supporting infrastructure modifications in the future, for off-the-shelf procurement such as vehicles, it is recommended that ADM(IE) provide an SOR(I) for the indicative cost for infrastructure prior to equipment project approval. For these types of projects, it is recommended that ADM(Fin CS) consider accepting indicative infrastructure estimates in order to obtain equipment project approval based on the risk and magnitude of the construction in support of the equipment. The DND Costing Handbook should also be amended to more clearly reflect this process.

Note: For a more detailed list of CRS recommendations and management response, please refer to Annex A—Management Action Plan.

Introduction

Rationale for Audit

Due to cost growth concerns in 2012, DND was directed to undertake an internal audit of the MSVS project prior to expenditure approvals for the last two phases of the five-phase project. The audit was included in the CRS FY 2013/14 to FY 2015/16 Risk Based Audit Plan, and the audit was conducted from January 2013 to December 2013.

Background

The objective of the MSVS project is to replace the medium lift capability currently provided by the 2.5 ton MLVW fleet.² This fleet of ▓▓▓▓▓▓▓▓▓▓▓▓▓ ³ vehicles was fielded in 1982, with an anticipated life span of 15 years. The life span of the MLVW has, however, been extended as over half of the fleet remains in service today.

The MSVS project is one of the first procurement initiatives of the Canada First Defence Strategy announced in June 2006 to strengthen the CAF. The procurement strategy,⁴ approved by the interdepartmental Senior Procurement Advisory Committee, included contract options to allow for additional MSVS vehicles as the number of CAF personnel has grown by 4,000 regular force members and the reserves by 4,000 members.⁵ The trucks will be used to support and sustain both domestic and deployed operations, as well as training missions.

There are five phases in the MSVS project. Phase I delivered ▓▓▓▓▓▓▓▓▓▓▓8-ton MilCOTS vehicles to support the reserves in domestic operations. Phase II was the purchase of ▓▓▓▓▓▓▓ baseline shelters for special equipment vehicles. Phase III was the acquisition of ▓▓▓▓▓▓▓ special equipment vehicle kits. Phase IV will involve the purchase of up to 1,500 8-ton⁶ SMP vehicles, while Phase V will be for the infrastructure component of the project. The indicative estimate for the whole project is ▓▓▓▓▓▓▓▓▓▓▓▓▓ billion.

The Logistics Vehicle Modernization project will replace the 1.5-ton Light Support Vehicle Wheeled fleet and the 10-ton Heavy Logistics Vehicle Wheeled fleet. This project is closely linked to the MSVS project, as both projects will combine to provide the CAF’s logistics vehicle lift requirements. The total budget for replacement of the logistics vehicles was set at ▓▓▓▓▓▓▓▓▓ billion in the 2009 Investment Plan. In October 2010, the Department was advised to proceed with the MSVS project as there was sufficient policy coverage for indicative cost growth of the SMP phase. Funding increases for the MSVS project required reductions of the Logistics Vehicle Modernization project budget. As such, these projects have interdependent requirements and budgets. Although the combined total of these two logistics vehicle projects has remained the same, concerns over the cost increase of the MSVS project led to cancellation of the SMP fleet RFP in July 2012. This has resulted in an 18-month delay in the project.

Project SMP RFP Review. As part of the audit, CRS reviewed the revised RFP for the Phase IV SMP vehicle fleet prior to its release in July 2013. A number of improvements were recommended in a management letter, which are summarized as follows:

• The content of the vendor’s risk register needed to provide more information, such as the quantified impact of risk, the five levels of risk severity, and the difference between inherent and residual risk.
• The RFP needed a clause that would allow for the exchange of “nil usage” or slow moving stock, which was recommended by the contractor. A Maximum Repair Cost of ▓▓▓▓▓ percent for repair and overhaul components was also needed to control costs.
• The ▓▓▓▓▓▓▓ percent performance incentive on the relatively immaterial management fee was too low to influence contractor performance.
• Greater clarity in the bid evaluation plan was also required in the life cycle cost escalation formulae, to ensure the rates were compounded over 20 years for fuel, spares and labour.

Given the affordability concerns that led to the need for a revised RFP to be issued to industry, the audit also included a review of the RFP cost controls. A review of the 2009 price and availability information from industry, which was used as the basis for the SMP vehicle contract cap, determined that:

• The weighted average of the price and availability quotes was well under the $725 million ceiling.
• Since 2009, the actual compounded inflation rate in the DND Historical Economic Model for military vehicles was found to be ▓▓▓▓▓▓▓ percent, reducing the likelihood of cost increases over this time period.
• A comparison of the estimated project cost, at the ▓▓▓▓▓▓▓▓▓▓▓▓United States Dollar exchange rate in FY 2009/10, against the actual average rate in FY 2011/12, resulted in a favourable compounded difference of ▓▓▓▓▓▓▓ percent. This increased the project’s flexibility to withstand future changes to the exchange rate.
• Given the current vehicle cost and foreign exchange rate trends, ▓▓▓▓▓▓▓ of the bidders that responded to Public Works and Government Services Canada⁷ in January 2013 stated they could provide a bid under the $725 million cap for 1,500 SMP vehicles.

Since the most recent SMP RFP, released in July 2013, the audit team considered a worst case scenario due to the recent foreign exchange loss on the Canadian Dollar. Should the winning contender bid the maximum $725 million cap for the SMP equipment, and the foreign exchange adjustment set-aside be insufficient for fluctuations, the project contingency may have to be supplemented or adjustments be made to other deliverables.⁸

The MSVS project office included several good practices in the RFP, outlined as follows:

• The $725 million bid evaluation cost cap included three incremental quantities of vehicles to provide assurance that the MSVS project will deliver a certain capability within the project cost constraints.
• To ensure best value, the bid evaluation weighting between price and technical criteria was a reasonable ratio, with less emphasis on technical merit than other fleets. As well, the second highest technical score may be considered if the bid price is ▓▓▓▓▓ percent lower than the bidder with the highest total score.
• The RFP for the contracted ISS includes incremental option periods up to 5 years in duration that are awarded for performance. This will encourage the ISS contractor to provide good value during the contract performance period so that DND will be inclined to exercise option years.

Objective

The audit objective was to assess the adequacy of the management control framework, governance processes and risk management strategy in place to ensure a cost-effective acquisition.

Scope

The audit scope of the MSVS project included only Phases IV and V, which still required expenditure approval. The Logistics Vehicle Modernization project was also included in the scope due to the requirements and funding source interdependency with the MSVS project.

Auditing potential vendors was not within the mandate of the audit and therefore out of scope. The audit included all MSVS project activities from 2006 to 2014 that related to Phases IV and V.

Methodology

The audit results are based on evidence from the following sources:

• interviews with DND, Defence Construction Canada and Treasury Board Secretariat staff;
• an examination of capital acquisition policies and project documentation;
• analysis of data from information systems such as the Fleet Management System, CID, Defence Resource Management Information System and Military Command Software Establishment; and
• a site visit to 2 Service Battalion at Canadian Forces Base Petawawa.

Criteria

Audit criteria used to assess the objectives are outlined in Annex B.

Statement of Conformance

The audit findings and conclusions contained in this report are based on sufficient and appropriate audit evidence gathered in accordance with procedures that meet the Institute of Internal Auditors’ International Standards for the Professional Practice of Internal Auditing. The audit thus conforms to the Internal Auditing Standards for the Government of Canada, as supported by the results of the quality assurance and improvement program. The opinions expressed in this report are based on conditions as they existed at the time of the audit, and apply only to the entity examined.

Findings and Recommendations

Phase IV Standard Military Pattern Vehicle Scope

Unless the SMP vehicle contract options are exercised, the full CAF requirement for ▓▓▓▓▓▓▓▓▓ SMP vehicles will not be met.

RFP Cap. Based on price and availability quotes received from industry in 2009, a bid evaluation cost cap of $725 million was put in place for the SMP fleet RFP in December 2011 to control costs and remain within the MSVS departmental project budget. As well, a cap was set at 1,500 SMP vehicles, with lower increments of 1,400 and 1,300 vehicles, given the potential variance in unit prices. However, in accordance with the procurement strategy approved by the interdepartmental Senior Procurement Advisory Committee, the RFP and project approval documentation also included options to procure an additional 650 SMP vehicles to accommodate the additional lift requirements of the expansion of the CAF in the 2008 Canada First Defence Strategy. The intent of this procurement strategy was to ensure best value within the project cost cap and mitigate the risk of an unsuccessful bidding process.

Full Requirement. Since 2010 it has been known that the full CAF requirement for the SMP phase is a fleet of ▓▓▓▓▓▓▓▓▓ vehicles.⁹ The increase in requirements from ▓▓▓▓▓▓▓▓▓ vehicles is due in part to the growth of the Canadian Army since 2008 in personnel, number of armoured vehicles, and firepower.¹⁰ The total Canadian Army requirement alone now stands at ▓▓▓▓▓▓▓ SMP vehicles, while additional vehicles are needed for non-Army requirements. The increase in the number of vehicles can also be attributed to new CAF units in the Canadian Operational Support Command, the Canadian Special Operations Forces Command, and the new Medium to Heavy Lift Helicopter squadron. As such, to fill the full requirement of ▓▓▓▓▓▓▓▓▓ SMP vehicles, the exercising of ▓▓▓▓▓▓▓ contract options will be required.

Other Project Requirements. For some new units, projects have been created that include emerging requirements for SMP vehicles with the additional funding to exercise some SMP vehicle contract options. At the time of the audit, the MSVS project had not been made fully aware of the other project requirements. A project such as the ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ project will determine its requirements prior to the MSVS Phase IV project scheduled approval date in ▓▓▓▓▓▓▓▓▓▓▓

Bid Price. In the most recent industry consultation in 2013, ▓▓▓▓▓▓▓ of the SMP vehicle contenders agreed that ▓▓▓▓▓▓▓▓▓ vehicles could be provided within the cost cap. Should the winning SMP vehicle bid price come in lower than the cost cap, the available MSVS project funds may be used to offset the current capability gap to meet the full requirement for ▓▓▓▓▓▓▓▓▓ SMP vehicles. If other projects have insufficient funds to meet their SMP vehicle requirements, the remaining MSVS project budget should be considered to meet the full requirement.

Summary. In order to control costs, the project RFP for the SMP fleet included a cost cap for 1,500 vehicles. However, some of the 650 vehicle options will need to be exercised to satisfy the lift requirements of the 2008 Canada First Defence Strategy expansion of the CAF. Not meeting the full requirement of ▓▓▓▓▓▓▓▓▓▓vehicles will negatively affect CAF operations, particularly the second line replenishment support for the Canadian Army offered by three service battalions.

Governance – Information for Decision Making

Not all MSVS project information available for senior management and third party review was complete, accurate and documented.

Project Value Increase Transparency

Based on the price and availability information received from industry in 2009, the MSVS project office concluded that there ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ At that time, the Departmental Finance Committee approved an Investment Plan Change Proposal to amend the 2009 Investment Plan due to the Logistics Vehicle Modernization project funds transfer. In spite of the 27 percent increase in the indicative project cost estimate, the Department was advised in October 2010 to proceed with the project as there was sufficient policy coverage. With the addition of the 2012 Investment Plan Change Proposal funds transfer for the shelters/kitting, the project value was raised to ▓▓▓▓▓▓▓▓▓▓▓▓▓ billion.¹¹

Despite the departmental approval of a new MSVS project budget in 2009, the reported value on the MCP-IOC monthly report remained at ▓▓▓▓▓▓▓▓▓▓▓▓▓ billion until mid-2012, which led to other departments being unaware of the updated estimated cost. The report format focussed on the approved project indicative cost, instead of the most recent Departmental funding allocation. However, the report should have indicated that the project was compromised and unable to deliver the requirement within the external budget limitations. Instead, the project reported the cost as low risk, and made reference to the departmental approval, but it did not include the project value increase. The MCP-IOC report format has recently changed, and projects are required to show both internal and external approved project values, which should mitigate the risk of this happening with future projects.

This increase in funds for the MSVS project was offset by a transfer from the ▓▓▓▓▓▓▓ billion Logistics Vehicle Modernization project that is still in options analysis phase. Both of these vehicle projects were included in the DND 2009 Investment Plan, and the combined value of the two projects has remained unchanged. In retrospect, the audit concludes that the ▓▓▓▓▓▓▓ percent increase in the MSVS fleet lift capacity over the current MLVW fleet will change the requirements of the two fleets of vehicles that will be delivered by the Logistics Vehicle Modernization Project.¹²

Policy

From 2009 to 2012, no formal reporting mechanism existed for the Department to report to Treasury Board Secretariat all approved Investment Plan Change Proposals. ADM(Fin CS) staff intend to develop a formal summary report to provide the Treasury Board Secretariat with changes to the Investment Plan. The specific format and frequency has yet to be determined.

While there is a need to return to Treasury Board to receive revised project approval for total indicative project values, there are no policies that direct when this should occur in a multi-phase project. Prior to the implementation of each project phase, the MSVS project office requested project budget increases only for that specific phase, and the total indicative estimate was adjusted accordingly. This was to be the case for the Phase IV SMP fleet cost increase as well, once the bid evaluation had determined a substantive¹³ cost estimate. Additional guidance for multi-phase project estimates, such as the MSVS project, would be of benefit in the future.

Corporate Investment Database Information Reliability

The CID is the principal tool used by VCDS project analysts to manage the capital acquisition program, and thus requires accurate and up-to-date project information. Some MSVS project information in the CID monthly progress report was incorrect, outdated or incomplete. This issue is not a rare occurrence, and has been reported in several CRS capital acquisition audits. At the time of this audit, 30 percent of all projects in the options analysis, definition or implementation phases had not been updated in the CID in the past three months.

Senior Review Board Project Performance Reporting

The September 2011 PAD required the SRB to receive performance information on cost, schedule and technical variances. This information was not reported to the three MSVS project SRBs that have taken place since September 2011.

Summary. Capital acquisition policies do not provide direction on approval of changes to the total indicative estimates on multi-phase projects. Nor is there policy direction for single-phase projects that become aware of significant cost growth early in the definition phase. As other departments had not been informed formally of the MSVS project increase until mid-2012, this led to the RFP’s cancellation, resulting in an additional 18-month project delay. Had the project office reported full indicative costs on the MCP-IOC report in 2009, or if it had another formal reporting mechanism, this issue could have been resolved earlier and the schedule delay avoided. Failure of projects to update project information in the CID does not provide sufficient information for the management of the capital program.

Contracted In-Service Support

Contracted in-service support estimates for the MSVS SMP fleet exceed the allocation of sustainment funds.

Affordability of In-Service Support

There will be insufficient funds for the indicative estimate of ▓▓▓▓▓▓▓ million per year for contracted ISS of the SMP fleet. The contracted ISS funds available for the maintenance of all ▓▓▓▓▓▓▓▓▓ wheeled logistic vehicles maintenance have been limited to ▓▓▓▓▓▓▓ million on average for FY 2014/15 to FY 2016/17. The MLVW fleet, to be replaced by the MSVS SMP fleet, only accounts for ▓▓▓▓▓ percent ( ▓▓▓▓▓▓▓ million) of this ▓▓▓▓▓▓▓▓▓ year allocation. In contrast, the SMP fleet annual estimate of ▓▓▓▓▓▓▓ million would account for 63 percent of the allocation. This will have a significant impact on the sustainment funds available for the other 10 wheeled fleets competing for the same maintenance resources.

Usage Rate

The principal cost driver for the high contracted ISS SMP fleet estimate is the spare parts needed to sustain the annual forecasted vehicle usage rates. Other projects have reduced the fleet usage in order to remain within vehicle sustainment budgets.¹⁴ Although the lift capacity of the SMP fleet ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ the MLVW fleet, the planned annual usage rate is ▓▓▓▓▓▓▓ kilometres per vehicle. However, this usage is not based on historic rates. Historical annual usage of the MLVW ranged from ▓▓▓▓▓▓▓▓▓ to ▓▓▓▓▓▓▓▓▓ kilometres. At these usage rates, a more reasonable estimate for a more capable vehicle would be ▓▓▓▓▓▓▓▓▓ million per year for the SMP fleet. This estimate is still considerably higher than the allocation for the MLVW fleet, and may place further limitations on utilization of the SMP fleet.

Summary. The limited sustainment funds available for the fleets of wheeled logistic vehicles are insufficient to support the estimated SMP contracted ISS cost. Although contracted ISS costs for a more capable vehicle are likely to be higher, these estimates can be significantly reduced if they are based on historical usage.

Infrastructure – Phase V

Infrastructure impacts of the MSVS project were not well planned or accounted for in a timely manner.

Although the MSVS project delivered ▓▓▓▓▓▓▓▓▓ MilCOTS vehicles by March 2011 to replace the Canadian Army reserves’ MLVW fleet of ▓▓▓▓▓▓▓ vehicles, very few of the 34 armouries have been modified to accommodate the new vehicles.¹⁵ ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ lift capacity, the new fleet requires larger armoury doors and additional parking space. The shortfalls in infrastructure are due to delays in defining requirements and obtaining the infrastructure phase approval. Early engagement, initially with Director Land Infrastructure, and subsequently with ADM(IE), would have minimized the impact on the units and equipment.

Infrastructure Approval

The MilCOTS fleet contract award was announced in December 2008, but the MSVS Phase V infrastructure modifications are not scheduled until spring 2015. Although infrastructure design funds are normally part of a project definition phase, the armoury modifications cannot be implemented until approval is received for the actual construction. Given the delay in the infrastructure phase, at least two armouries completed the modifications with funds from their own operating budgets.

In November 2007, a two-phased project was the original plan; Phase I delivery of the MilCOTS fleet and shelters, and Phase II delivery of the SMP fleet and kitting. Infrastructure funds were to be incorporated into each phase. The change to a single infrastructure phase, Phase V, did not enable infrastructure modifications prior to the vehicle delivery. The reason for a Phase V for infrastructure was that the necessary substantive cost estimate could not be obtained for approval prior to the selection of vehicles for the MilCOTS and SMP fleets.

Statement of Requirement (Infrastructure)

A consolidated SOR(I) was not prepared or coordinated by the project office to support the infrastructure activities of the MSVS Project for all locations.¹⁶ By 2011, only ¹⁷ armouries that had received MilCOTS vehicles had completed an SOR(I). Seventeen other armouries had studies under way, and no requirements had been received from the reserves in the Maritimes or Eastern Ontario. Five of these armouries are planning to implement their modifications with their own operating budgets, rather than the MSVS project funds. This raises the concern that infrastructure costs will not be funded by the MSVS project if they are funded by other budgets. The SMP fleet will be similar to heavier vehicles held by regular force units, so there should be less significant infrastructure work.17

Infrastructure Modification Estimates. Due to the incomplete requirements definition, the most recent infrastructure estimates may be understated. Although the requirements definition for at least 17 armouries was unknown, the Phase V infrastructure estimate of ▓▓▓▓▓▓▓▓▓ million was reduced to ▓▓▓▓▓▓▓ million.¹⁸ The original estimate of ▓▓▓▓▓▓▓ million may be overstated, but the most recent infrastructure estimate could be understated as there are many armouries that do not have modification estimates.

Summary. Infrastructure modifications for the MSVS fleet have been delayed until the final project phase. The MilCOTS fleet delivered in 2011 may incur additional maintenance costs due to the shortfalls in infrastructure. With the current expenditure approval process, substantive costs for an equipment estimate typically include indicative infrastructure estimates, as design has generally lagged the selection of a weapon system by one year. In some cases, the ADM(Fin CS) project cost validation reports recommend expenditure authority approval with both substantive and some insignificant indicative estimates, but this standard has not been included in the DND Costing Handbook. Unless the contender with the largest vehicle was used as the basis of infrastructure requirements, the infrastructure design impacts could not be fully known until the MilCOTS vehicle contract was awarded.

Risk Management

Not all DND risk management practices were followed by the MSVS project.

Risk Management Plan. The MSVS risk management plan did not incorporate some of the DND practices outlined in the Integrated Risk Management policy and guidelines, and the PAD. The project risk management plan only included an overview of the risk management process, detailed impact and likelihood tables for qualitative assessments, a 5 by 5 heat map, a risk response flowchart, and a list of risk response strategies. Normally, risk management plans would also include risk quantification plans and risk categories. Incorporating this information would strengthen risk assessment and mitigation for the MSVS project.

Risk Assessments. The PAD requires projects to perform both a qualitative and quantitative risk assessment to assess risk severity. The MSVS project performed a qualitative risk assessment and assigned scores for impact and probability. However, a quantitative risk assessment was not done to calculate and allocate the contingency funds necessary to mitigate each risk.¹⁹

Risk Mitigation. The risk information presented to the MSVS project SRB could be improved. Mitigation strategies to demonstrate how the project intends to manage risk have not been presented, nor has there been a clear distinction between inherent and residual risk.

Inconsistent Risk Severity Reporting. The risk severity assessments in various project documents were found to be inconsistent. For example, many of the risks in the project risk register were assessed as high or very high, but in the Project Profile and Risk Assessment most of these risks were assessed as medium or low. In the Project Complexity and Risk Assessment, the project was assessed as a level three evolutionary project – a relatively complex project. However, the low to medium risk severity assessments in the CID would be more representative of a less complex project. Risk severity that is consistent across all project documents provide senior management more confidence that risks will be effectively managed.

Summary. The MSVS risk management plan does not include some of the DND standard practices. As a result, full mitigation costs of risk have not been determined and the project contingency has not been allocated accordingly.

General Conclusion

The MSVS project office identified the major project risks, simplified the design variants, and exhibited proactive succession planning. Improvements are needed in governance policy for the communication of any project’s indicative cost increases, the MSVS project office’s risk management and controls that affect the MSVS fleet size, infrastructure modifications, and contracted ISS costs. Since this project was announced in June 2006, the expansion of the CAF has increased the MSVS SMP vehicle requirement. To ensure that the CAF receives its full requirement of MSVS SMP vehicles, this need should be reflected in the upcoming approval documentation for expenditure approval. As well, the requirements definition for infrastructure modifications for the MSVS fleets need better controls and coordination.

To accommodate cost increases of the MSVS project that would ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ lift capacity, funds were transferred in 2009 from another interdependent logistic vehicle project that was also in the approved DND investment plan. Although the Department was advised to proceed with the project in October 2010, as there was sufficient policy coverage for the indicative cost growth, the project experienced considerable delay over the uncertainty as to when to report this increase of the indicative cost of a later phase of the MSVS project. Initiatives are under development to relay the latest cost information that results from Investment Plan changes. With respect to life cycle support, a more realistic forecast of the SMP fleet usage will likely reduce the contracted in-service support cost estimate. Risk management practices need to be updated to reflect DND risk management practices in order to enable the project to succeed in meeting its goal of a ▓▓▓▓▓▓▓▓▓ project approval for the SMP and infrastructure implementation phases.

Annex A—Management Action Plan

CRS uses recommendation significance criteria as follows:

High—Controls are not in place or are inadequate. Important issues are identified that could negatively impact the achievement of program/operational objectives.

Moderate—Controls are in place but are not being sufficiently complied with. Issues are identified that could negatively impact the efficiency and effectiveness of operations.

Low—Controls are in place but the level of compliance varies.

Phase IV Standard Military Pattern Vehicle Scope

Management Action

ADM(Mat) will ensure that the MSVS project approval (implementation) documents reflect government policy coverage.

OPI: ADM(Mat)/DGMPD L&S

Target Date: December 2014

Auditor’s Note. Government policy does allow for vehicle options to be exercised as was done in the MSVS project Phase I with the procurement of ▓▓▓▓▓▓▓ additional MilCOTS vehicles.

Governance – Information for Decision Making

Management Action

The MSVS project's CID Monthly Progress Report, the SRB Performance Report and the MCP-IOC documentation will be kept up-to-date. The MSVS project has designated a project OPI to update these documents routinely, as required.

OPI: ADM(Mat)/DGMPD L&S

Target Date: September 2014

Management Action

VCDS will ensure direction is incorporated into the current update of the PAD.

OPI: VCDS

Target Date: July 2014

Management Action

The VCDS’s bi-annual call letter to update the submissions forecast will contain specific direction to ensure Level Ones (L1) verify currency of CID entries for their projects.

Preparation of “Yellows” by VCDS analysts prior to a project appearing before Defence Capability Board or Program Management Board will continue to specifically report on currency of SRB and CID data updates.

VCDS analysts attending SRBs will ensure they take the opportunity to query the project staff on CID status. The responsible L1 will be encouraged to record this in the SRB Record of Decisions.

The VCDS Direction letter on the CID will be refreshed and re-issued to include specific guidance on verification of CID data. The PAD amendment process currently in progress will refer to this direction.

During the annual Program Management Board presentation by Chief of Programme will include a CID Health report to assess the Department’s performance across all L1s.

OPI: VCDS

Target Date: March 2015

Contracted In-Service Support

Management Action

MSVS ISS cost estimates are being reviewed and updated in consultation with in-house stakeholders and service providers such as Director General Land Equipment Program Management, Director General Materiel Systems Supply Chain, Director Land Requirements, and Director Materiel Group Operational Research. The ISS project cost estimates will be refined for the SMP implementation phase once the winning bidder and costs are known. ADM(Fin CS) is being engaged in the cost validation process.

OPI: ADM(Mat)/DGMPD L&S

Target Date: September 2014

Infrastructure – Phase V

Management Action

The following procedures have been implemented. Initially, the equipment project can allocate 10 percent of the project’s budget for infrastructure purposes as a preliminary indicative estimate to ensure sufficient funding is reserved for that purpose. SOR(I)s and indicative estimates will be produced following initial development activities such as having determined the number of vehicles per location and assessed the existing space to determine infrastructure options. The SOR(I) can be used as a basis for determining the indicative cost for project approval. The design stage will confirm or adjust the construction cost estimates.

ADM(IE) is currently working closely with ADM(Mat) to jointly review new equipment projects at the identification and options analysis phases to identify the potential requirements for Construction in Support of Equipment as early as possible. A draft Terms of Reference for Infrastructure Project Managers for Construction in Support of Equipment projects will be posted on the ADM(Mat) Knowledge Network, and is currently under review by ADM(IE)/DCPD. The intent is to help equipment project managers gain greater understanding of the infrastructure requirements.

OPI: ADM(IE)/DCPD

Target Date: May 2014

Management Action

After considering recommendation seven, CRS comments are acknowledged; however, it should be noted that the proposed action is already ADM(Fin CS)’s current practice, in that ADM(Fin CS) staff do look at the quality of the individual components of a project cost estimate and form our recommendations on the quality of the overall estimate. Further, the Costing Handbook already differentiates between the data integrity and risk of individual elements and for the overall project. That said, in a future update of the Costing Handbook, this section will be amended to more clearly reflect this standard.

OPI: ADM(Fin CS)

Target Date: February 2015

Risk Management

Management Action

The MSVS Risk Management Plan is being revised and will be in accordance with departmental direction on risk management.

OPI: ADM(Mat)/DGMPD L&S

Target Date: July 2014

Annex B—Audit Criteria

Criteria Assessment

The audit criteria were assessed using the following levels:

Assessment Level and Description

Level 1: Satisfactory

Level 2: Needs Minor Improvement

Level 3: Needs Moderate Improvement

Level 4: Needs Significant Improvement

Level 5: Unsatisfactory

Governance

1. Criterion. An adequate monitoring process is in place that uses high-quality, up-to-date and accurate information as the basis for decision making; roles and responsibilities are defined, and necessary skills, staff and resources are available to govern the project. (Governance and Strategic Directions 6, Risk Management 8, Results and Performance 2, People 2, 4, Public Service Values 5, Accountability 1, Stewardship 10, 20)

Assessment Level 4 – The project continued to report the outdated project estimate after departmental approval to the MCP-IOC, with no other formal mechanism in place to inform other departments of a DND estimate increase. It was observed that information in the CID Monthly Progress Report necessary to manage the capital program is often incorrect, outdated or missing. The SRB was not being briefed on some key performance information.

Internal Controls

2. Criterion. Project schedule is achievable and is managed to avoid impact on operational requirements. (Risk Management 7, Stewardship 1, 15, 16, 22)

Assessment Level 3 – SMP phase first delivery has been delayed due to a combination of external and internal causes. Internal schedule management requires improvement to meet future milestones and to avoid further project delays, as project resources were not being assigned to tasks in its project Master Schedule. (Management letter)

3. Criterion. Operational requirements are in accordance with defence policy. They are clearly defined, complete, prioritized, consistent and traceable throughout the project activities, from Statement of Requirements development to performance specifications test, evaluation and training plans. (Governance and Strategic Direction 4, Risk Management 7, Stewardship 22)

Assessment Level 4 – The full requirement for ▓▓▓▓▓▓▓▓▓ SMP vehicles will not be met unless contract options are exercised.

4. Criterion. Financial management and materiel asset accountability are in accordance with the Financial Administration Act, DND and Treasury Board regulations, while ensuring the best value total cost of ownership and facilitated with reliable and relevant cost estimates. (Stewardship 1, 3)

Assessment Level 3 – There are insufficient funds for the MSVS SMP fleet estimated ISS estimate, due to a planned annual usage rate that is overstated and not reflective of historic rates. Infrastructure modifications remain uncertain; however, the estimate for infrastructure Phase V was significantly reduced and could be understated.

5. Criterion. Contract terms and conditions optimize value for money. (Stewardship 14)

Assessment Level 3 – The RFP for the ISS contract includes incremental option periods that are awarded for performance; this will encourage the contractor to provide good value in order to obtain option years. Increased visibility of the vendor’s risk information should be sought by DND. The RFP did not have clauses to avoid the procurement of excess stock and control costs of repairable components. The performance incentive that was proposed in the ISS contract was too low to influence any prospective contractor. The cost cap employed was considered a good practice to control costs.

Risk Management

6. Criterion. Risks are identified, assessed, ranked, mitigated, quantified with cost impact, and reported in accordance with relevant policy and best practices. (Risk Management 1, 2, 4, 5, 7)

Assessment Level 2 – Not all good risk management practices for identifying, assessing, ranking, responding to and monitoring risk were in place.

Sources of Criteria

1. Audit criteria related to the Management Accountability Framework: A Tool for Auditors, March 2011 (see reference after each criterion above).
2. DND Project Approval Directive 2011-2012, VCDS.
3. Project Management Body of Knowledge, Edition 4, 2008.

___________________________________________________________________________________________________________________________

Footnote 1 Indicative cost estimates are defined in the DND Costing Handbook as + 25 percent.

Footnote 2 These SMP vehicles can lift a ▓▓▓▓▓ ton load on roads but are limited to ▓▓▓▓▓ tons on rough terrain.

Footnote 3 Fleet Management System includes the number of MLVW still in service and the number of MLVW replaced by the MSVS MilCOTS fleet.

Footnote 4 Approval of the procurement strategy was 13 June 2008. Membership of the Senior Procurement Advisory Committee includes Public Works and Government Services Canada, Industry Canada, Western Economic Development Diversification, Atlantic Canada Opportunities Agency, and Canada Development for Quebec Regions Agency, and is chaired by ADM(Mat).

Footnote 5 Budget 2006 provided sufficient funds for the regular force to grow from 64,000 to the current strength of 68,000 personnel, and for the reserves to grow from 26,000 to 30,000 – Canada First Defence Strategy June 2008, Section V Rebuilding the Canadian Forces – Personnel.

Footnote 6 The SMP RFP requires a minimum eight-ton vehicle with rated criteria up to 10 tons. The RFP also includes 650 SMP vehicle options.

Footnote 7 January 2013 Industry Consultation meetings. ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ contenders did not respond.

Footnote 8 Other contract deliverables include spares, interim support and engineer change proposals.

Footnote 9 The full requirement of ▓▓▓▓▓▓▓▓▓ SMP vehicles was also stated at the Program Management Board by the Canadian Army in August of 2012.

Footnote 10 During the SMP phase implementation, the Canadian Army will be equipped with ▓▓▓▓▓▓▓ additional Tactical Armoured Patrol vehicles that will require additional lift for fuel, ammunition and spare parts. The upgrade of the engines and armour of the ▓▓▓▓▓▓▓ Light Armoured Vehicles III will also require an increase in fuel replenishment.

Footnote 11 At this time in the MSVS project, these estimates were indicative.

Footnote 12 In January 2010, it was concluded that the SMP fleet would have a minimum capacity of 8 tons and a palletized load system for sea containers. Compared to the MLVW fleet, the MSVS volume capacity will increase by ▓▓▓▓▓▓▓ percent and the weight capacity by ▓▓▓▓▓▓▓ percent.

Footnote 13 Substantive costs are defined in the DND Costing Handbook as estimates with a variance of + 15 percent.

Footnote 14 The Close Combat Vehicle project had to reduce its annual usage per vehicle from ▓▓▓▓▓▓▓▓▓ to ▓▓▓▓▓▓▓▓▓ kilometres to remain affordable.

Footnote 15 For those 17 armouries with SOR(I)s, only 2 of a sample of 13 armouries had undergone modifications and used their own operating budgets as a funding source.

Footnote 16 The Director Construction Project Delivery Quality Manual, published byADM(IE), states that the SOR(I) is a mandatory project document as it communicates the characteristics of the operational requirement.

Footnote 17 The MSVS project infrastructure study estimated only ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ for SMP fleet infrastructure modifications.

Footnote 18 The Phase V ▓▓▓▓▓▓▓ million estimate tabled at the 2012 MSVS SRB was reduced to ▓▓▓▓▓ million at the 2013 SRB.

Footnote 19 For example, the Project Management Body of Knowledge, Chapter 10, recommends an Expected Monetary Value calculation—a product of financial impact and probability.