Language selection

Government of Canada / Gouvernement du Canada

Search

Follow-up on Internal Audit: Land Command Support System (LCSS) Contract

PDF Version

(155KB, 21pages)

March 2014

7050-28-4 (CRS)

Reviewed by CRS in accordance with the Access to Information Act (AIA). Information UNCLASSIFIED.

Caveat

The result of this work does not constitute an audit of the LCSS Contract project. Rather, this report was prepared to provide reasonable assurance that the Management Action Plans (MAP) that resulted from the 2008 Internal Audit of the LCSS Contract were implemented as stated and as such have addressed the associated recommendations.

Table of Contents

Acronyms and Abbreviations

ADM(Mat)

Assistant Deputy Minister (Materiel)

ASC

Audit Services Canada

CRS

Chief Review Services

DLP

Director Land Procurement

DND

Department of National Defence

LCMM

Life Cycle Materiel Managers

LCSS

Life Cycle Materiel Managers

LCSS

Land Command Support System

AERMS

Audit and Evaluation Recommendation Management System

DGLEPM

Director General Land Equipment Program Management

MAP

Management Action Plan

OPI

Office of Primary Interest

PMIS

Program Management Information System

PPAS

Program Performance Assurance Sub-section

PWGSC

Public Works and Government Services Canada

R&O

Repair and Overhaul

RPC

Repair Priority Code

SACC

Standard Acquisition Contract Clause

DLCSPM

Director Land Command Systems Program Management

E&I LTSC

Engineering and Integration Long-Term Support Contract

1.0 Introduction

In keeping with the Treasury Board Policy on Internal Audit,1 Chief Review Services (CRS) is required to undertake audit follow-ups to assess the implementation status of MAPs developed in response to previous CRS audit recommendations. In accordance with the CRS Risk-based Audit Plan for Fiscal Year 2013/14 to 2015/16, the 2008 Internal Audit of the LCSS Contract was selected for follow-up. The LCSS is a highly integrated tactical command-and-control system that forms part of every Canadian Army vehicle and platform-based weapon system.

The objective of the 2008 audit was to assess if adequate governance, risk and management control frameworks were in place to administer the LCSS contract. The $291-million contract was from 2002 to 2009 for software and hardware engineering, integrated logistics support, configuration management, and component repair of the LCSS. It was managed by the Director Land Command Systems Program Management (DLCSPM). The following audit findings were reported:

Terms of Payment. Contract terms of payment were not fully exercised to provide an incentive for the vendor to perform tasks efficiently and to minimize task cost escalation.

Value for Money. Improvements in contract management could have reduced costs by $2.8 million, controlled cost escalation, and helped in the forecasting of future costs.

Materiel Management. Measures were not in place to ensure that vendor-held Department of National Defence (DND) assets with a reported value of $6.0 million were properly safeguarded. Lack of DND oversight of the vendor repair line could result in inefficiencies.

Payment Certification. Contract clauses and policies regarding vendor time verification and inter-company profit were not exercised to support certification of payments worth $134 million.

Governance. Metrics to monitor performance were not reported by the vendor. The lack of data integrity in vendor reports did not provide accurate information for decision making.

Risk Management. An internal DND Risk Management Plan does not exist, and the vendor was not required by DND to exercise best practices in risk management.

In order to address the above issues, six recommendations were put forward (see Annex C). To address these recommendations, the Assistant Deputy Minister (Materiel) (ADM(Mat)) developed a set of MAPs to address the audit findings specific to their area of responsibility. The MAPs were intended to be implemented through the follow-on long-term support contract called the Engineering and Integration Long-Term Support Contract (E&I LTSC).

2.0 Objective

The purpose of this follow-up is to determine the progress made on the implementation of the MAPs in response to the 2010 Audit of the LCSS contract.

3.0 Methodology

This audit follow-up is the outcome of a review of documentation and evidence to assess the progress made in implementing the MAPs, based on the assessment criteria in Annex A. The following methods were used:

  • review of the Audit and Evaluation Recommendation Management System (AERMS);2
  • interviews and inquiries with DND contract management staff in the ADM(Mat), the Director General Land Equipment Program Management (DGLEPM), DLCSPM, specifically the Technical Authority and the Requisitioning Authority, in the Director of Land Procurement (DLP), and other headquarters staff;
  • review of contract documentation that pertained to the MAPs; and
  • analysis of data from the Defence Resource Management Information System (DRMIS), and from the DLCSPM Program Management Information System (PMIS).

Statement of Conformance

The audit follow-up conclusions contained in this report are based on sufficient and appropriate audit evidence gathered in accordance with procedures that meet the Institute of Internal Auditors’ International Standards for the Professional Practice of Internal Auditing. The audit follow-up thus conforms to the Internal Auditing Standards for the Government of Canada, as supported by the results of the quality assurance and improvement program. The opinions expressed in this report are based on conditions as they existed at the time of the audit follow-up, and apply only to the entity examined.

Good Practices

A program management information system was created to streamline contractor requests and increase visibility to stakeholders.

Holdbacks are being applied to all payments until deliverables are received.

4.0 Overall Assessment

At the time of follow-up, all six recommendations and eighteen MAP items were designated completed by the Office of Primary Interest (OPI) in AERMS. Upon further examination, seven MAP items were fully implemented, three were deemed obsolete, and eight were not fully implemented. Two of the eight MAP items will remain open (MAPs 1.a) and 4.c)). Full implementation of the MAPs was largely contingent on the creation of Program Performance Assurance Sub-section (PPAS), but the positions were not staffed. DLCSPM has chosen to accept certain risks of not staffing these positions given the resource trade-off decisions that must be made.

Until all MAPs are fully implemented, there remain concerns in the following areas:

  • There are difficulties in challenging vendor performance on recurring-type work without a performance baseline.
  • There is a risk that the vendor made excess profit from subcontracting to its own subsidiaries at higher rates.

A scorecard of the MAP items can be found in Annex B, and a more detailed assessment of progress can be found at Annex C.

Annex A—Assessment Criteria

Line of Enquiry: Progress made on the 2008 Audit Recommendations

Progress has been made on the implementation of MAPs identified in the 2008 LCSS audit.

The following criteria were used to assess the level of completion for each MAP item.

  1. Obsolete or Superseded

Audit recommendations that are deemed to be obsolete or have been superseded by another recommendation.

  1. No Progress or Insignificant Progress (0-24% complete)

No action taken by management or insignificant progress. Actions such as striking a new committee, having meetings and generating informal plans are insignificant progress.

  1. Planning Stage (25-49% complete)

Formal plans for organizational changes have been created and approved by the appropriate level of management (at a sufficiently senior level, usually at the Executive Committee level or equivalent) with appropriate resources and a reasonable timetable.

  1. Preparation for Implementation (50-74% complete)

The entity has begun necessary preparation for implementation, such as hiring or training staff, or developing or acquiring the necessary resources to implement the recommendation.

  1. Substantial Implementation (75-99% complete)

Structures and processes are in place and integrated in some parts of the organization, and some achieved results have been identified. The entity has a short-term plan and timetable for full implementation.

  1. Full Implementation (100% complete)

Structures and processes are operating as intended and are implemented fully in all intended areas of the organization.

Annex B—Management Action Plan Scorecard

Table B-1. Management Action Plan Scorecard

Table Summary:

Six recommendation areas are listed in the left-hand column. For each recommendation, read across the row to determine the MAP item, the OPI and the CRS assessment of progress on the MAP item.

Recommendation # MAPs OPI CRS Assessment of Progress on MAPs

1. Terms of Payment

1.a) Apply suitable payment options and assess holdbacks.

ADM(Mat)

Planning Stage
New Target Date: July 2014

1.b) Track performance metrics to establish a baseline.

ADM(Mat)

No Progress or Insignificant Progress

2. Value for Money

2.a) Track task budget and earned value information, and provide to stakeholders.

ADM(Mat)

Substantial Implementation

2.b) Invoke 12-month repair and overhaul hardware warranty.

ADM(Mat)

Full Implementation

2.c) Extend 90-day software warranty.

ADM(Mat)

Full implementation

2.d) Strengthen control over spares economic order quantities.

ADM(Mat)

Obsolete or Superseded

2.e) Insert credit clause for inactive spares.

ADM(Mat)

Obsolete or Superseded

3. Materiel Management

3.a) Ensure inventory codes are accurate.

ADM(Mat)

Substantial Implementation

3.b) Oversee compliance with stocktaking, adjustments, and set maximum and minimum stock levels.

ADM(Mat)

Substantial Implementation

3.c) Establish Government Furnished Equipment stocktaking policy.

ADM(Mat)

Full Implementation

4. Payment Certification

4.a) Conduct time verifications.

ADM(Mat)

Obsolete or Superseded

4.b) Audit time verification system.

ADM(Mat)

Full Implementation

4.c) Request Audit Services Canada (ASC) cost audit.

ADM(Mat)

Substantial Implementation

4.d) Capitalize qualified expenditures as betterments.

ADM(Mat)

Full Implementation

5. Governance

5.a) Track performance metrics.

ADM(Mat)

Preparation for Implementation

5.b) Withhold payments for undelivered reports.

ADM(Mat)

Full Implementation

6. Risk Management

6.a) Implement internal risk management program.

ADM(Mat)

Substantial Implementation

6.b) Ensure risks are categorized, quantified, prioritized.

ADM(Mat)

Full Implementation

Annex C—Summary of Management Action Plan Status

Table C-1. Status of the Implementation of the MAP Items for Recommendation 1

Table Summary:

The MAPs are listed in the left-hand column. For each MAP item, read across the row to determine the OPI, the target date, the progress to date, and the status.

Recommendation 1—“Ensure that DLCSPM use the most appropriate basis of payment option and apply holdbacks to tasks that contain discrete deliverables. In the follow-on contract, introduce incentives for cost and schedule performance.”
MAPs OPI Target Date Progress to Date Status of Action Item

1.a) The E&I LTSC includes various payment options. DLCSPM, in conjunction with DLP 2 and Public Works Government Services Canada (PWGSC), will apply the suitable payment option to each task and will assess holdbacks to ceiling-priced tasks where this is deemed appropriate.

ADM(Mat)

April 2009

Although firm fixed price and ceiling price are available in the E&I LTSC, only the limitation of expenditure basis of payment has been invoked. A 10-percent payment holdback is now being applied to all tasks.

The majority of tasks are developmental engineering work, and do not suit a firm fixed price, as costs are difficult to estimate by both DND and the vendor. However, the contract management team has recently identified ongoing support tasks, such as management services, where a ceiling-price basis of payment is applicable.

The plan is to implement a ceiling-price basis of payment for these tasks by July 2014.

Planning Stage

1.b) The E&I LTSC includes various performance metrics that can be called up for tasks. DLCSPM’s PPAS will begin to track task performance metric results in order to establish a baseline against which incentives can be applied as appropriate in due course.

ADM(Mat)

April 2009

Two new performance metrics are now being reported to assist DND in managing tasks to budget: Estimate to Completion (expected additional cost), and Estimate at Completion (expected total cost). However, metrics were not tracked specifically to establish a performance baseline at either the task or program levels. PPAS positions were created to assist in the tracking of the performance metrics, but these positions were not staffed. The risk is currently accepted by DLCSPM, as most of the tasks do not contain recurring work that can be compared to a baseline.

CRS recommends that this MAP be closed.

No Progress or Insignificant Progress

Table C-2. Status of the Implementation of the MAP Items for Recommendation 2

Table Summary:

The MAPs are listed in the left-hand column. For each MAP item, read across the row to determine the OPI, the target date, the progress to date, and the status.

Recommendation 2—“Ensure that DLCSPM technical authorities have task budget and earned value information to enable them to manage the tasks better. Ensure that DLCSPM invokes the 12-month repair and overhaul (R&O) warranty clause and extends the software warranty to at least 12 months in the follow-on contract. Strengthen controls on spares economic order quantities, and include a credit clause in the follow-on contract for inactive spare parts procured on the vendor’s recommendation.”
MAPs OPI Target Date Progress to Date Status of Action Item

2.a) The E&I LTSC includes provisions for this data. DLCSPM’s PPAS will track this information and will ensure that access to it is provided to stakeholders for improved management of tasks.

ADM(Mat)

April 2009

The DLCSPM technical OPIs (authorities) have access to task budget information through the PMIS, which they use to help manage their tasks. There is a provision in the E&I LTSC for DND to request earned value reporting when required. It has not been requested as the contract management team indicates that earned value is difficult to implement for the highly developmental nature of this contract.

CRS recommends that this MAP be closed.

Substantial Implementation

2.b) DLCSPM, in conjunction with PWGSC, will invoke the 12-month R&O warranty clause for hardware, as appropriate, under the E&I LTSC.

ADM(Mat)

April 2009

The E&I LTSC includes a 12-month warranty for hardware. Items are tagged with the date of the warranty period to trigger the process for the vendor to make a repair. Six warranty claims have been honoured from April 2009 to March 2013.

Full Implementation

2.c) DLCSPM and DLP, in conjunction with PWGSC, will extend the 90-day Standard Acquisition Contract Clause (SACC) clause warranty on software to 12 months for E&I LTSC tasks, where it is deemed appropriate.

ADM(Mat)

April 2009

The SACC software warranty clause has been expanded to include a 12-month warranty for defined software issues.

Full Implementation

2.d) DLCSPM’s PPAS will oversee the application under the E&I LTSC of controls on spares economic order quantities.

2.e) DLCSPM, in conjunction with DLP, will assess the need to include a credit clause for suitable tasks concerning inactive spares procured at the vendor’s recommendation under the E&I LTSC.

ADM(Mat)

April 2009

The E&I LTSC specifies that the vendor must not stock spares in anticipation of a repair tasking. When the vendor does require spares for components of the land command system, approval is sought via PMIS from the Program’s technical, procurement and contract authorities. This negates the requirement for a credit clause in the contract and for the oversight of economic order quantities related to spare parts.

CRS recommends that this MAP be closed.

Obsolete or Superseded

Table C-3. Status of the Implementation of the MAP Items for Recommendation 3

Table Summary:

The MAPs are listed in the left-hand column. For each MAP item, read across the row to determine the OPI, the target date, the progress to date, and the status.

Recommendation 3—“Improve R&O efficiency by revising the Repair Priority Codes (RPC),3 Maximum Repair Costs4 and Standard Unit Price settings for line items repaired by the vendor. Enforce compliance with contractor repair and overhaul policy (A-LM-184-001/JS-001) with respect to Canadian Forces Supply System account stocktaking and adjustments, maximum and minimum settings, and establish a policy for the frequency of Government Furnished Equipment stocktaking.”
MAPs OPI Target Date Progress to Date Status of Action Item

3.a) DLCSPM will work with DLP 2 to ensure that RPCs, Maximum Repair Costs, and Standard Unit Price settings are correctly established in E&I LTSC tasks for line items repaired by the vendor.

ADM(Mat)

April 2009

RPCs and MRCs are reviewed for accuracy, either on an annual or an ad hoc basis, by life cycle materiel managers (LCMM) and supply managers associated with the E&I LTSC. A supply manager will generate a DRMIS report and send it to the LCMM for updating. Standard unit prices cannot be adjusted by LCMMs.

The percentage of repairable items with a Maximum Repair Cost exceeding the standard unit price improved from 25 percent in the original audit compared to 16 percent now.

The remaining overall risk is low as repair and overhaul service charges from April 2009 to March 2013 were $7.3 million, which is 2 percent of the $651-million E&I LTSC.

CRS recommends that this MAP be closed.

Substantial Implementation

3.b) DLCSPM’s PPAS will oversee vendor compliance under the E&I LTSC with A-LM-184-001/JS-001 concerning Canadian Forces Supply System account stocktaking and adjustments, and maximum/minimum settings.

ADM(Mat)

April 2009

Evidence was provided that stocktaking was completed every two years on all materiel accounts held by the contractor. Any adjustments to inventory accounts in the supply system made by the contractor are overseen and approved by the National Defence Quality Assurance Representative.

The maximum and minimum levels of spares were set for only 6 of 47 repairable line items. However, these triggers of automatic re-ordering were causing too many spares to be procured. Although these levels have not been set for all line items, average turnaround time for repair is eight days lower than in 2008, indicating that the contractor has not had issues with stock-outs.

CRS recommends that this MAP be closed.

Substantial Implementation

3.c) Director General Materiel Systems and Supply Chain will develop a policy to establish the frequency of Government Furnished Equipment stocktaking.

ADM(Mat)

November 2009

Government-furnished equipment is materiel loaned to the contractor to assist in the repair process. The repair and overhaul special instructions policy states that the contractor is responsible for initiating and completing stocktaking of loaned materiel every two years.

Full Implementation

Table C-4. Status of the Implementation of the MAP Items for Recommendation 4

Table Summary:

The MAPs are listed in the left-hand column. For each MAP item, read across the row to determine the OPI, the target date, the progress to date, and the status.

Recommendation 4—“In consultation with PWGSC, verify the accuracy of labour hours through time verification and ensure inter-company transfers are reasonable through an ASC cost audit. Ensure that expenditures that meet the definition of a ‘betterment’ are properly accounted for.”
MAPs OPI Target Date Progress to Date Status of Action Item

4.a) DLCSPM’s PPAS and DLP 2, in conjunction with PWGSC, will begin conducting time verifications for selected E&I LTSC tasks.

ADM(Mat)

October 2009

This management action item has been superseded by MAP item 4.b).

CRS recommends that this MAP be closed.

Obsolete or Superseded

4.b) DLCSPM and DLP 2 will request PWGSC to audit the vendor’s time verification system.

ADM(Mat)

October 2009

The PWGSC contracting authority reported conducting two verifications of the vendor’s time system in September and November 2010. This includes a review of a sample of time-sheets and the vendor’s time system to validate total labour hours charged. PWGSC cannot provide the results to DND but a follow-up letter was sent to the vendor.

Full Implementation

4.c) DLCSPM and DLP 2, through PWGSC, will request an ASC cost audit of inter-company transfers after the E&I LTSC has become established.

ADM(Mat)

October 2009

In April 2009, DLP 2 asked if the PWGSC contracting authority would request a PWGSC cost profit audit. Evidence of a response could not be located. However, the current contracting authority confirmed via DLP 3-3 that a request for a PWGSC audit has now been made regarding intercompany transfers. This audit could address if the company as a whole made more profit than allowed through subcontracting with its own subsidiary.

CRS recommends that this MAP remain open to track its status.

Substantial Implementation

4.d) DLCSPM and DLP 2, in consultation with the DGLEPM comptroller, will ensure that expenditures that meet the definition of betterment—as opposed to obsolescence mitigation or corrective, adaptive, perfective or preventive maintenance—are accounted for properly.

ADM(Mat)

October 2009

There is a regular process where the DGLEPM comptroller requests expenditures to be reviewed for capturing betterments to capital assets. The E&I LTSC fielding support expenditures have been captured as a betterment against assets delivered by the Intelligence, Surveillance, Target Acquisition and Reconnaissance capital project.

Full Implementation

Table C-5. Status of the Implementation of the MAP Items for Recommendation 5

Table Summary:

The MAPs are listed in the left-hand column. For each MAP item, read across the row to determine the OPI, the target date, the progress to date, and the status.

Recommendation 5—“Metrics need to be reported in accordance with the vendor measurement plan to facilitate task estimation and the follow-on contract performance baseline. The LCSS contract managers should follow up discrepancies in vendor reports. In the follow-on contract, delivery of key reports should be linked to payments.”
MAPs OPI Target Date Progress to Date Status of Action Item

5.a) Under the E&I LTSC, the vendor will be tasked to provide performance metric data for tasks. DLCSPM’s PPAS will track this information, carry out trend analysis for improved program management, and ensure that access to the data is provided to stakeholders for improved task management.

ADM(Mat)

April 2009

The E&I LTSC requires the vendor to report performance metrics in six categories of work. The metrics reported at the program level include headcount, staff utilization, repair and overhaul status, and planned vs. actual deliverables, with trends. The vendor is also now reporting Estimate to Completion, Estimate at Completion, and variance. PPAS was not staffed and any further trend analysis was deemed to not add any value either at the program or task levels for developmental and integration work. Stakeholders, including task-level technical authorities, have access to task budget, amendment notes, and other program-level management data through the new PMIS.

CRS recommends that this MAP be closed.

Preparation for Implementation

5.b) Under the E&I LTSC, payments will be withheld for undelivered reports identified in tasks as deliverables.

ADM(Mat)

April 2009

A 10-percent holdback has been applied to all payments to encourage progress against milestones. The final payment holdback is released only after the task closure report is received and validated.
A review of a sample of vendor reports indicates that the vendor was compliant with the required format.

Full Implementation

Table C-6. Status of the Implementation of the MAP Items for Recommendation 6

Table Summary:

The MAPs are listed in the left-hand column. For each MAP item, read across the row to determine the OPI, the target date, the progress to date, and the status.

Recommendation 6—“For the follow-on contract, develop an internal risk management plan and ensure that DND and vendor risks are categorized, quantified and prioritized.”
MAPs OPI Target Date Progress to Date Status of Action Item

6.a) DLCSPM’s PPAS will implement an internal DLCSPM risk management program.

ADM(Mat)

April 2009

There are regular operational risks briefed to DLCSPM on the E&I LTSC.
A risk management plan should be considered to document the current risk management program.

CRS recommends that this MAP be closed.

Substantial Implementation

6.b) DLCSPM’s PPAS will ensure that the risks tracked by the vendor are categorized, quantified and prioritized.

ADM(Mat)

April 2009

Risks are reported by the vendor in a risk register. Risk categories include operational, budget, and schedule. Probability is quantified as a percentage, and impacts are shown as a dollar amount, number or systems affected, or number of months’ delay to schedule. Risks are prioritized in the PMIS, with the highest risk shown at the top. Highest risks are also briefed during the monthly program management meetings.

Full Implementation

___________________________________________________________________________________________________________________________

Footnote 1 Policy on Internal Audit http://www.tbs-sct.gc.ca/pubs_pol/dcgpubs/ia-vi/ia-vi_e.asp.

Footnote 2 AERMS is a system that tracks the progress of the MAP items.

Footnote 3 The RPC is used to tell the contractor/DND facility which items should be repaired first. The RPC reflects the number of months of serviceable stock available (http://admmat.ottawa-hull.mil.ca/material/mmi/1500_e.pdf).

Footnote 4 The Maximum Repair Cost is a standard established by DND to guard against the possibility of an item being repaired at a cost that exceeds its replacement value to DND.

Page details

Date modified: