Audit of the C-130J In-Service Support Contract

December 2015

7050-11-51 (ADM(RS))

Reviewed by ADM(RS) in accordance with the Access to Information Act. Information UNCLASSIFIED.

Acronyms and Abbreviations

Results in Brief

The Canada First Defence Strategy emphasized the Government of Canada’s commitment to acquire 17 urgently needed new C-130J Hercules tactical lift aircraft to replace the previous C-130 models. The C-130J aircraft is a transport aircraft used to carry equipment, troops and supplies around the world, enabling troops to conduct safer and more effective operations at home and abroad. Compared to previous models, the C-130J aircraft requires fewer crew members, can carry heavier payloads, and has greater fuel efficiency.¹ Less than seven months after receiving the first new C-130J, the aircraft and its crews were deployed to Afghanistan to help with the mission. The C-130J is currently being used for various requirements such as to deliver aid around the world and to conduct training in northern Canada.

In December 2007, the Department of National Defence (DND) purchased 17 C-130J aircraft. The contract² to acquire the C-130J aircraft was amended in December 2009 and February 2010 to include in-service support³ (ISS) for approximately $800 million.⁴ ISS elements of the Contract refer to the activities required to keep the C-130J fleet in serviceable condition to meet operational requirements. In order to follow the DND In-Service Support Contracting Framework (ISSCF), the ISS for the C-130J fleet was awarded to the platform⁵ supplier through a performance-based contract. The Contract is set to expire on 30 June 2016, with provisions for a five-year option period. A proposal from the Contractor to extend the Contract was received in July 2015. DND indicated that over the first five years of the ISS, the Contractor had consistently provided the level of service expected and DND has been working collaboratively with the Contractor to ensure the objectives of the Contract are being met.

Assistant Deputy Minister (Review Services) (ADM(RS)) conducted an analysis of contracts⁶ in 2012 to identify higher-risk contracts across DND to be considered for audit or review. Based on risk attributes such as its materiality and duration, the C-130J ISS Contract was selected for inclusion in the Chief Review Services (CRS)⁷ Risk-Based Internal Audit Plan for fiscal years (FY) 2013/14 to (FY) 2015/16. The objective of this audit was to determine whether governance structures, risk management practices, and internal controls were in place to provide value for money and to effectively manage the ISS elements of the C-130J ISS Contract.

The recommendations presented in this report and the corresponding action plans will help improve the management of the Contract going forward.

Findings and Recommendations

Basis of Payment. The C-130J ISS Contract’s scalability for maintenance and training support needs to be expanded to be better aligned with the expected variability in RCAF operations; also, greater costing visibility must be achieved to help improve value for money. The fixed fee portion of the Contract is being paid, although less than half of the contracted flying hours are actually flown, and about half of the originally expected training capacity is being utilized. The contracted projections for flying hours have been unachievable for various reasons that are not under direct control of the Contract Management Team, such as a lack of pilots and maintenance crews. The current Contract also does not provide pricing visibility into the fixed fee. Better visibility would have offered information to assess value for money, which would have helped in the negotiation for the option period. The Crown’s request for a proposal related to the option period included scalability to account for changes in activity levels. Improved pricing visibility was also requested.

It is recommended that Assistant Deputy Minister (Materiel) (ADM(Mat)) ensures that appropriate scalable flying hour and training activity ranges are provided to Public Works and Government Services Canada (PWGSC) to be incorporated into the option period of the C-130J ISS Contract, and continues to pursue visibility and an understanding of the contract costs so the option period contract can potentially achieve better value for money.

Information for Decision Making. Performance measurement has not been fully implemented and not all required reports are submitted or reviewed in accordance with the Contract. The Contract is performance-based and DND is regularly tracking performance data. There are also guidance documents that are in place to help with the recording of performance data. DND highlighted that the Contractor was performing well based on the interim metrics; however, if the Contractor was not performing well, it could not be held accountable for its performance, since the performance data and related calculations on which it is assessed are not fully accurate and complete. In addition, some reports were not submitted by the Contractor or reviewed by the Crown in accordance with the Contract, limiting the information available to the Contract Management Team.

It is recommended that ADM(Mat) formalizes the process to track performance data accuracy, and provides support to PWGSC in obtaining a formal agreement with the Contractor for acceptable data standards, in order to continue working towards implementing the performance-based aspects of the C-130J ISS Contract.

It is recommended that ADM(Mat), in collaboration with the Commander of the RCAF and the Assistant Deputy Minister (Information Management) (ADM(IM)), develops a plan to improve performance measurement in order to allow for the enforcement of the C-130J ISS Contract terms and conditions.It is recommended that ADM(Mat) reassesses required reports to determine their continued relevance, frequency, review period, and resubmission cycle, and includes provisions in the option period of the C-130J ISS Contract to hold the Contractor accountable for these deliverables.

Risk Management. ISS-related risks are not being formally managed by the Contract Management Team when these risks become their responsibility after they are removed from the Project Management Office risk register. Formal risk management can help better identify, mitigate and monitor ISS risks, and potentially reduce their impact if they materialize. However, through project risk review meetings, the Weapon System Manager is made aware of project-related and ISS-related risks. Effectively managing ISS risks with a risk management process and risk register can help the Contract Management Team focus on the most important risks while managing the Contract.

It is recommended that ADM(Mat) ensures that a formal ISS risk management process is in place, including a risk register, and ensures that it is kept current throughout the duration of the C-130J ISS Contract.

Financial Management. Although generally in compliance, the approval of additional work and the certification of invoices are not always in accordance with the Contract terms and conditions and the Financial Administration Act. Some of the sampled requests for additional work with a price equal to or in excess of $100,000 were not approved by the PWGSC Contracting Authority as per the Contract. In addition, some of the invoices had been signed by the Technical Authority and certified by the Procurement Authority that work had been performed prior to the fixed fee portion of work being completed. Supporting documentation, necessary for the certification by the Procurement Authority, was available for most of the sampled invoices. Since part of the Contract’s basis of payment is fixed fee, the impact of approval of invoices prior to the completion of work is minimal. However, improvements are required for compliance with contract terms and conditions and with the Financial Administration Act.

It is recommended that ADM(Mat) ensures that contractual documents are approved by the proper authority, and that personnel with certification authority receive the necessary guidance to help ensure that invoices are certified after the invoice period, and that adequate supporting documentation is retained on file to substantiate that goods and services have been received in accordance with the C-130J ISS Contract.

Note: Please refer to Annex A—Management Action Plan for the management response to the ADM(RS) recommendations.

1.0 Introduction

1.1 Background

The C-130J⁸ aircraft, made by Lockheed Martin Aeronautics (the Contractor), is a military transport aircraft. Over 300 C-130J aircraft have been sold worldwide, and it is currently used by 16 countries.⁹ The C-130J is considered the lifeline of deployed forces and is used to transport equipment, troops and supplies around the world, enabling militaries to conduct safer and more effective operations at home and abroad.

In 2005, the RCAF initiated the C-130J project (officially known as the Airlift Capability Project – Tactical). To meet the increasing requirements for operations and the requirement to replace a number of aging C-130E model aircraft, this capability became urgent.¹⁰ The Canada First Defence Strategy emphasized the Government of Canada’s commitment to acquire 17 new C130J Hercules tactical lift aircraft.

The Crown entered into a contract with the Contractor on December 20, 2007, to buy 17 C-130J aircraft for approximately $1.45 billion.¹¹ The first aircraft was delivered in June 2010 and the 17th aircraft arrived in May 2012. All 17 of DND’s C-130J aircraft are based in 8 Wing Trenton. Less than seven months after receiving the first new C-130J, the aircraft and its crews were deployed to Afghanistan to help with the mission. The C-130J is currently being used for various requirements such as to deliver aid around the world and to conduct training in northern Canada. Key benefits of the C-130J over its predecessors include its ability to fly faster, higher and farther, and carry heavier loads while burning less fuel and requiring fewer crew members.

Subsequent to the acquisition of the aircraft, the Contract was amended in December 2009 and February 2010 to incorporate ISS and training for approximately $800 million¹² – these amendments are referred to as the “ISS Amendment” and the “Maintenance Training Suite Amendment.” The fundamental goal of an ISS program is to provide equipment support at a level that meets or exceeds the operational requirements¹³ while obtaining optimum value for money. While the acquisition of the aircraft and the amendments were made under the C-130J Project Management Office, the responsibility to manage the ISS and training aspects of the Contract rests with the Weapon System Manager and the Contract Management Team.

The contracting strategy to award the ISS portion of the Contract to the platform supplier was based on direction provided in the ISSCF.¹⁴ The ISSCF was implemented by DND in 2009 during a period of significant equipment reinvestment; a standardized approach to ISS contracting was therefore needed to streamline the procurement of platforms and associated ISS. Canadian allies had successfully implemented performance-based contracting and incentivized contracting approaches for platform ISS; therefore DND decided to adopt a similar approach to ensure optimum value for money. This approach is detailed in the ISSCF. Key principles of the ISSCF are that the ISS contract is established with the platform supplier, that the contract is long-term and performance-based, and that the contract is structured and managed to ensure clear and irrefutable contractor accountability for contracted results. At this time, the ISSCF is under review within ADM(Mat) to determine if the contracting approach in the ISSCF is the best contracting approach for all platforms in all situations going forward. The Contract and its amendments expire on June 30, 2016. However, the Crown has the ability to extend the ISS and training aspects of the Contract through option periods. A proposal from the Contractor to extend the Contract was received in July 2015, which has initiated negotiations to potentially exercise a five-year option period.

1.2 Rationale for Audit

ADM(RS) conducted the Analysis of Contracts in March 2012 to determine DND contracts that could be included in the Risk-Based Internal Audit Plan. Risk criteria were applied to active DND contracts (tendered by PWGSC) greater than $1 million to select contracts that could benefit from audit coverage. The nature of the business conducted by DND resulted in some of the contract risks identified in the analysis as being unavoidable or inherent. Based on risk attributes such as the materiality and long duration of the C130J Contract, it was included in the CRS Risk-Based Internal Audit Plan for FY 2013/14 to FY 2015/16.

1.3 Objective

The objective of this audit was to determine whether governance structures, risk management practices, and internal controls were in place to provide value for money and to effectively manage the ISS elements of the Contract.

1.4 Scope

The audit scope included the ISS elements of the contract from the ISS Amendment dated December 2009 and from the Maintenance Training Suite Amendment dated February 2010. The audit did not include the acquisition of the 17 C-130J aircraft or the acquisition of the Maintenance Training Suite.

1.5 Methodology

The approach included the following:

• interviews with staff from ADM(Mat) and the RCAF;
• examination of DND and PWGSC policies, procedures, and guidance documents;
• examination of contract documents such as the Articles of Agreement, the Statement of Work, and other Annexes;
• examination of contract costs captured in the Defence Resource Management Information System (DRMIS);
• examination of a judgemental sample of 36 Contract Data Requirements Lists (CDRL), 13 contract invoices valued at approximately $87.78 million, and 27 requests for additional work valued at approximately $5.16 million; and
• site visit to 8 Wing Trenton.

1.6 Audit Criteria

The audit criteria can be found at Annex B.

1.7 Statement of Conformance

The audit findings and conclusions contained in this report are based on sufficient and appropriate audit evidence gathered in accordance with procedures that meet the Institute of Internal Auditors’ International Standards for the Professional Practice of Internal Auditing. The audit thus conforms to the Internal Auditing Standards for the Government of Canada, as supported by the results of the quality assurance and improvement program. The opinions expressed in this report are based on conditions as they existed at the time of the audit and apply only to the entity examined.

2.0 Findings and Recommendations

2.1 Basis of Payment

The C-130J ISS Contract’s scalability for maintenance and training support needs to be expanded to be better aligned with the expected variability in RCAF operations; also, greater costing visibility must be achieved to help improve value for money.

2.1.1 Costing Visibility

During the audit, DND highlighted they were satisfied with the training and engineering support that the Contractor was providing. Some improvements in the Contract could benefit the Contract Management Team’s ability to assess value for money for these services.

The C-130J ISS Contract has fixed and variable fees. According to the Contract, payment for maintenance support consists of an average fixed monthly fee of $4.73 million from 2010 to 2016. In addition, the variable monthly fee is based on the actual monthly flying hours at a fixed rate per flying hour.

Payment for training support consists of a fixed monthly fee as well as a fixed quarterly fee. The fixed monthly fee averages about $1 million from 2012 to 2016, and the fixed quarterly fee averages about $0.28 million from July 2013 to June 2016.

Prior to the 2009 ISS contract amendment, the Contract pricing breakdown was reviewed at the Contractor’s premises by a team from PWGSC and DND, but they were not permitted to retain copies of the documentation. Due to a lack of access to contractor-owned costing data, as well as staff turnover, there is an absence of a complete understanding of costing details. According to the Weapon System Manager, the ISS maintenance program was put in place to support 11,900 flying hours per year for the fleet of 17 aircraft, and the training program was based on the number of students to be trained. The PWGSC Contracting Authority stated that the fixed fee for the maintenance support includes all the fixed expenses that the Contractor had for the C-130J program, while the monthly fee for the training support was based on a training school being operational 16 hours per day, five days per week, regardless of the number of courses offered and students trained. Although the PWGSC and DND team reviewed the pricing structure prior to the ISS Amendment, it is not clear what cost drivers make up the fixed fee basis of payment.

Contractually, pricing visibility cannot be obtained from the Contractor for the original ISS Amendment because the Contract does not have an audit clause. However, for the ISS option period, the Contractor has been requested to “provide sufficient visibility of cost elements and cost drivers in order to allow for future negotiations of new performance periods. … Detailed cost breakdown and price certification must therefore be provided as part of [the] proposal.”¹⁵ To date, for the option period, the Contractor has demonstrated a willingness to provide greater pricing and costing information than it did for the original ISS Amendment. If greater visibility and understanding of the costs that support the fixed fee basis of payment are made available, this would provide better information to help assess whether value for money is being achieved.

2.1.2 Scalability and Value for Money

The expected yearly flying hours stated in the Contract were established based on the RCAF’s projection that the C-130J fleet would be flying 11,900 hours per year (700 flying hours per aircraft). The RCAF has been unable to meet this projection for various reasons that are not directly related to the Contractor or Contract Management Team, such as a lack of pilots and maintenance crews. The C-130J fleet flew on average 5,215 hours annually (based on calendar years 2012 to 2014). With the assumption that the maintenance support fixed fee takes into account the expected yearly flying rate of 11,900 hours, the fixed fee is not providing optimal value for money as the fleet is flying less than 44 percent of the expected yearly flying rate. Therefore, the fixed fee is being paid for at a level of contracted service that is not being fully utilized, and there is no provision in the Contract to adjust the fixed fee portion of the basis of payment to reflect the required levels of maintenance and training support.

On the other hand, as a result of the lower-than-planned yearly flying rate, the variable payment for the maintenance support is lower than anticipated. The basis of payment for monthly variable flying hours is based on a set rate per flying hour and the monthly invoice is affected proportionally by each additional flying hour. This has led to lower DND costs for the variable portion of the basis of payment. This is a form of scalability, as costs are reduced by flying less than 11,900 hours.

If the fixed fee portion of the basis of payment was scalable to be able to adjust the flying hours, the risk of not flying to the maximum hours that are included in the Contract could have been partially mitigated. Scalability was not included for the fixed fee basis of payment because the RCAF was concerned that if the Contractor’s level of service was decreased to meet the maintenance and training needs of the RCAF, then the Contractor might not be able to promptly increase its service levels back up if needed. Scalability for flying hours and training activities is being requested for the option period.

The training support fixed fee is also not providing optimal value for money. Based on the planned training schedule provided in the Contractor's Training Needs Analysis Report of March 2009, it is assumed that the training facility’s operational hours requirement was based on the planned training schedule of about 929 training days per year. Additionally, according to the Weapon System Manager, only half of the originally expected number of training courses were held as the demand for training was lower than expected.

Under the Department’s ISSCF, the basis of payment for ISS contracts is directed to be mostly fixed fee. Fixing the fee based on flying hour and training requirements that could not be changed or scaled to a different service level, compounded by the departmental directive to follow the ISSCF, resulted in the fixed fee portion of the basis of payment having to be paid even though less than half of the contracted flying hours are actually flown, and approximately half of the planned training capacity is being utilized.

For the option period, an enhanced scalability for the maintenance and training support is being considered as an addition to the Contract. As per the PWGSC Notification letter to the Contractor requesting the option period proposal, “An essential condition of success for this initiative is … an affordable proposal that will be scalable to meet DND’s operational and budgetary requirements.”¹⁶ Improved scalability would provide DND and the RCAF “the ability, at a known cost, to increase or decrease activity levels consistent with the operational requirement.” ¹⁷

Along with the scalability for the maintenance and training support, an annual operating plan¹⁸ is anticipated to be implemented, as per the ISSCF proposed process to manage the ISS contract. An annual operating plan works well with scalability in a contract as it will help to identify the expected work associated with a particular scalable level of service ranges (tiers) for the maintenance and training support required by DND. Annually, DND would notify the Contractor of its forecasted yearly flying rate and planned training support needed, and the Contractor would develop an annual operating plan based on these figures. The annual operating plan would provide visibility into the work that is to be performed, which would help to assess value for money. At this time, the scalable approach being considered for the option period is better than the current fixed fee basis of payment, but it is important that the ranges for the tiered approach are aligned with the RCAF projected flying hours and training requirements. Since the selected scale for flying hours and training activities is determined annually, this would allow the RCAF to adapt its projections more regularly, and in those cases where operational and budgetary requirements change frequently. As a result, better value for money could be achieved.

2.1.3 Summary

In general the Contract followed the direction of the ISSCF for the basis of payment to be fixed fee; however, the Contract did not include scalability for the fixed fee portion of the basis of payment so that it could annually adjust its costs for maintenance and training support. Enhanced scalability and an annual operating plan could result in lower overall costs and would help ensure that DND is only paying for contracted services that it annually expects to require. Additionally, there is no documentation in the contract file to provide visibility into the fixed fee portion of the Contract. Visibility into the fixed fee pricing would have provided information to assess value for money, which would have been useful for the option period negotiation. Pricing visibility has been requested for the option period, which is a step in the right direction.

2.2 Information for Decision Making

Performance measurement has not been fully implemented and not all required reports are submitted or reviewed in accordance with the Contract.

The Contract is a performance-based contract, where complete and accurate information is important to the Contract Management Team for decision making. Performance data are used to assess how well contract requirements are being met by the Contractor. The Contractor’s performance is calculated in the Contractor’s Performance Measurement System using three contractually enforceable performance metrics.

2.2.1 Performance Data

The use of the Performance Measurement System¹⁹ for the Contract was dependent on the implementation of DND’s DRMIS maintenance module for the management of the C-130J fleet. According to the Contract, performance data were to become contractually enforceable twelve months after DRMIS implementation for fleet management; however, the implementation of DRMIS for the C-130J fleet was delayed. In the meantime, the Project Management Office used an interim performance measurement system to record C-130J maintenance data. The interim performance measurement system allowed stakeholders to become accustomed to a performance-based environment; however, it was not able to provide all of the required information to the Contract Management Team, which made it difficult to determine if a performance issue was attributable to the Contractor or to DND. If the maintenance module for DRMIS had been implemented as scheduled for the C-130J fleet, the performance metrics information could have been useful in implementing the performance-based aspects of the Contract. There could also have been better information available to help negotiate the option period of the Contract.

The implementation of DRMIS for the C-130J fleet in April 2014 has required an ongoing change in culture by the RCAF C-130J maintenance staff at 8 Wing Trenton. According to the RCAF, DRMIS is not as user-friendly as anticipated, and since it is relatively new to the RCAF C-130J maintenance staff and it represents a change in how maintenance activity is recorded, the data being entered into DRMIS are not fully accurate or complete. To address this issue, the Project Management Office regularly tracks the data entered in DRMIS and produces data quality reports, which are submitted to the RCAF C-130J maintenance staff at 8 Wing Trenton to advise on the data fields that are incorrect or missing. The reports list the data that were checked, the accuracy of the data, the required improvements, and the procedure documents to follow for the recording of data. In addition to the data quality reports, the Project Management Office intends on providing training to the maintenance staff to help increase the accuracy and completeness of the data.

The integrity of the performance data is also dependent on DND’s system (currently DRMIS), and the Contractor’s system (Performance Measurement System), as well as the interface between the two systems. There are system-related technical issues that have yet to be resolved for the performance calculation to be accurate. Until the performance data in DRMIS are accurate and the technical issues are resolved, performance calculations will not be completely accurate. This is hindering the C-130J ISS Contract from being an enforceable performance-based contract.

The verification of the performance information that is being done by the Project Management Office is important to improving data accuracy. However, the process to verify the accuracy of the data in DRMIS as well as the process to verify the accuracy of the systems are not documented. Currently, one person from the Project Management Office is responsible for tracking the DRMIS data, and the process used to verify performance-related data has not been formalized.

For DRMIS information to be contractually used for performance measurement, the Contractor must formally agree to a standard for what are considered reliable data. In December 2014, the Contractor agreed to a framework for how accurate the data must be in order to be considered acceptable; however, there is no formal record to support this agreement. A documented verification process as well as a formally agreed-upon DRMIS data accuracy standard for the performance data would provide guidance and consistency in case of a change in personnel. This would also substantiate the validity of the performance calculation.

2.2.2 Contract Deliverables

The C-130J ISS Contract includes mandatory contractor deliverables called CDRLs. These deliverables specify the data and information, such as performance calculations and risk information, to be submitted by the Contractor, along with the required level of detail to be provided to DND in the form of reports. The Contract also specifies the required time frame and frequency for the reports to be provided by the Contractor, and the time that the Crown has to review and accept or reject the information in these reports. Some of these CDRL documents provide important information for decision making and could also be used for performance monitoring.

While most CDRLs have been received by DND in accordance with the Contract, there are areas for improvement regarding CDRL management. For example, some CDRLs have not been submitted to DND and some have been rejected and never resubmitted. These instances were due to DND not verifying if a CDRL was submitted when required, as well as the Contract not containing a resubmission clause. There were also CDRLs that DND reviewed late, CDRLs that had no evidence of being reviewed and CDRLs that were approved but were not in accordance with the Contract in relation to the level of detail required.

The majority of CDRLs have been included under the fixed fee portion of the basis of payment. Therefore, even if CDRLs are not satisfactory, the complete monthly fixed fee portion of an invoice will be paid to the Contractor as fixed fee payments are not linked to the acceptance of CDRLs.

2.2.3 Summary

The C-130J ISS Contract is a performance-based contract; however, the performance data in DRMIS related to the Contract are not yet considered accurate or complete so the data must be improved. This issue has led to the performance-based aspects of the Contract not being enforceable if contractor performance issues are present. In addition, in some instances, information required by the Contract was not submitted or reviewed. The Contract contains no mechanism to require the Contractor to provide satisfactory reports as payments are not linked to the acceptance of CDRLs. Improvements need to be made going forward so that information is available to make key decisions.

2.3 Risk Management

ISS-related risks are not being formally managed by the Contract Management Team.

2.3.1 Informal Risk Management

Within ADM(Mat), the Project Management Office is responsible for the acquisition of the C-130J fleet, while the Contract Management Team is responsible for the ISS of the C-130J fleet. During the acquisition of the C-130J aircraft, the Project Management Office had instituted a risk management plan and a risk register to document project risks. Once those project risks develop into ISS risks, they are removed from the Project Management Office’s risk register and the ownership of the risk is transferred to the Contract Management Team, including the Weapon System Manager. Therefore, project risks that become ISS risks as well as ongoing risks that arise throughout the ISS time frame of the equipment life cycle are to be managed by the Contract Management Team.

Based on the Project Management Office risk register, many ISS-related risks have been removed from the risk register as these risks are no longer the responsibility of the Project Management Office. For example, the risk to implement the performance metric framework, the risk to fulfilling the Weapon System Manager workload, and the risk of being unable to establish a baseline for the level of ISS needed prior to the option period negotiation have all been closed. To help ensure that the Contract’s objectives are achieved, those risks that should have been transferred to the Contract Management Team and all other ISS risks should have then been included in the Contract Management Team’s risk register.

Major departmental policies and guidelines such as the Procurement Administration Manual and the Integrated Risk Management policy and guidelines do not provide specific requirements for risk management within ISS activities.²⁰ However, to manage contract risks, it has been a Director General Aerospace Equipment Program Management²¹ procedure²² since 2008 to develop a risk register for all repair and overhaul contracts with a cash flow of greater than or equal to $30 million per year.

Since the transition of many contract responsibilities from the Project Management Office to the Weapon System Manager began in April 2014, the Weapon System Manager has expressed an interest in formalizing the risk management process for ISS risks. However, due to other priorities, neither a formal ISS risk management plan nor a risk register has been developed. Formal risk management could help better identify, quantify, prioritize, mitigate, and monitor ISS risks, and potentially reduce their impact on ISS-related costs and schedules. Effectively managing ISS risks can help the Contract Management Team focus on the most important risks while managing the Contract. Although there is no formal risk management, the Weapon System Manager remains aware of relevant ISS issues and risks by actively managing the Contract, by participating in quarterly risk review meetings held by the Project Management Office, and by receiving Contractor-provided progress reports containing risk information.

2.3.2 Summary

Due to unclear departmental guidance on risk management requirements by contract management teams, a formal risk management plan and a risk register are not yet in place for the C-130J ISS-related risks. While the Project Management Office has removed ISS-related risks from its risk register, the Contract Management Team has not yet documented these risks in its own risk register. Though the Contract Management Team is aware of these ISS-related risks, a formal risk management plan and risk register could have helped the Contract Management Team achieve better value for money by mitigating ISS risks, including risks related to unknown and evolving requirements.

2.4 Financial Management

Although generally in compliance, the approval of additional work and the certification of invoices are not always in accordance with the contract terms and conditions and the Financial Administration Act.

2.4.1 Approval of Additional Work

At times throughout the duration of a contract, there is a need for additional work that is within the contract scope but is not covered by the contract price. As per the C-130J ISS Contract, additional work with a price equal to or in excess of $100,000 has to be approved by the PWGSC Contracting Authority. From a sample of 27 approved requests for additional work valued at a total of $5.16 million, eight requests for additional work valued at $4.66 million had a price equal to or in excess of $100,000. Of those eight requests, only three valued at $3.85 million were approved by the PWGSC Contracting Authority, while the other five were approved by the DND Procurement Authority. While all of the additional work was for legitimate requirements, controls need to be improved in this area to promote compliance with the Contract and to improve oversight.

2.4.2 Certification of Invoices

Certification in accordance with the Financial Administration Act by a DND procurement authority is the control that attests that goods or services have been received in accordance with the contract before an invoice can be paid. Prior to the procurement authority certification, a technical authority must confirm that goods and services have been received in accordance with the contract. The certifications for both the technical authority and the procurement authority are supposed to be completed after the completion of work.

As part of this audit, 13 invoices dated between April 2014 and February 2015 were sampled with a total value of $87.78 million. The verification of those invoices demonstrated that they were all signed by DND procurement staff with appropriate delegation of authority. In seven out of 13 invoices, valued at $43.10 million, the Technical Authority had confirmed that goods and services had been received²³ prior to the fixed fee portion of work being completed. Five out of the seven invoices, valued at $30.60 million, also had the Procurement Authority certification prior to the fixed fee portion of work being completed. Certification by a procurement authority includes the requirement to ensure appropriate documentation is retained on file to support the receipt of goods and services. Of the 13 invoices examined, 11 invoices valued at $87.74 million had the necessary documentation to support the amounts invoiced. DND guidelines for invoice certification in accordance with the Financial Administration Act state the requirement to maintain appropriate supporting documentation.

2.4.3 Summary

The impact of the lack of oversight for the approval of additional work requests is partly mitigated as this Contract has a well-defined scope of work. Similarly, since the incorrect invoice certification is for the fixed fee portion of the invoice, the potential financial consequence of this issue is minor. However, oversight and controls in these areas need to improve to be in compliance with contract terms and conditions and with government acts, policies, and guidelines.

3.0 General Conclusion

The C-130J ISS Contract supports an essential capability for the RCAF and helps to fulfill the mandate of the Canadian Armed Forces. DND has been satisfied with the service provided over the first five years of the ISS and is working collaboratively with the Contractor to continually improve the partnership. The Contract Management Team has implemented good practices—such as tracking performance data on a regular basis, and requesting improved scalability and pricing visibility in anticipation of negotiations for the upcoming option period. To help enhance contract management and to continue to pursue value for money, improvements are needed in the governance processes, risk management practices, and controls related to the C-130J ISS Contract.

The basis of payment in the Contract should take into consideration expected variability in operational requirements, and the Contract should ensure that its objective of being performance-based is achieved. As well, mechanisms should be put in place so that satisfactory reports are obtained in accordance with the Contract’s terms and conditions. The Contract Management Team also needs to develop formal risk processes to help mitigate ISS risks. Additionally, improvements are needed in the areas of approving additional work and certifying contract invoices.

While this Contract is benefitting from the good practices mentioned above, the recommendations are provided to improve the contract management and value for money for the C-130J ISS Contract.

Annex A—Management Action Plan

ADM(RS) uses recommendation significance criteria as follows:

Very High—Controls are not in place. Important issues have been identified and will have a significant negative impact on operations.

High—Controls are inadequate. Important issues are identified that could negatively impact the achievement of program/operational objectives.

Moderate—Controls are in place but are not being sufficiently complied with. Issues are identified that could negatively impact the efficiency and effectiveness of operations.

Low—Controls are in place but the level of compliance varies.

Very Low—Controls are in place with no level of variance.

Basis of Payment

Management Action

As part of the Contract discussions for the next Period of Performance starting in July 2016, the method by which ISS will be provided focuses heavily on the inclusion of tiered service levels that can be adjusted to align with RCAF utilization of the C-130J. The ISS proposal that was delivered on July 29, 2015, includes a three-tiered approach to both aircraft support and technical training. For aircraft support, the tiered approach covers annual flying rates ranging from 7,000 to 11,900 hours. The technical training tiered approach, or number of courses offered, will be predicated not on the number of hours flown during the year but simply on the number of technicians at the Maintenance Unit who require training, increasing from what is currently offered (Tier 1) to the maximum identified within the Contract (Tier 3). Selection and agreement of the appropriate tier will be accomplished using an annual operating plan. The proposal also provides increased visibility into the Contractor’s cost structure during the next period of performance.

OPI: ADM(Mat)

Target Date: July 2016

Information for Decision Making

Management Action

Interim processes governing data integrity and quality will be formalized with Weapon System Management work instructions that will provide the necessary direction to ensure a performance-based accountable framework can be implemented and sustained. These final/formalized processes are planned to be in place pOPI: rior to the next contract period of performance but no later than June 2016. Additionally, the Weapon System Management staff will provide support as required to PWGSC in obtaining a formal agreement with the Contractor for acceptable data standards.

OPI: ADM(Mat)

Target Date: July 2016

Management Action

Given performance measurement data are solely dependent on DRMIS and associated programs, assurance of the quality of performance measurement data will require that the Weapon System Management staff continue to work directly with maintenance technicians to provide monthly data integrity and accuracy targets. A detailed plan will be elaborated to identify how efforts will be undertaken by the C-130J Weapon System Management staff, in collaboration with ADM(IM) – Director DRMIS and the RCAF, to prioritize and address system data, training and cultural challenges that affect ISS performance measurement. The plan will include the conduct of informal audits to ensure the information entered into and received from the system is accurate, logical, and meets the overall intent of the mandated performance requirements/targets.

OPI: ADM(Mat)

OCI: Commander of the RCAF and ADM(IM)

Target Date: July 2016

Management Action

As part of the recent contract negotiations with the Contractor, the need was identified to review the current deliverable schedule, frequency and approval timelines. Amendments relating to these issues were incorporated into the Statement of Work that supports the ISS proposal delivered by the Contractor on July 29, 2015. Furthermore, PWGSC will be consulted to determine the possibility/benefit of adding a contract clause to hold the Contractor accountable for future deliverables.

OPI: ADM(Mat)

Target Date: July 2016

Risk Management

Management Action

In support of the ongoing transition between Project Management Office Airlift Capability Project – Tactical (C-130J project) and Director General Aerospace Equipment Program Management, the C-130J Weapon System Manager has reviewed the appropriate documentation, including the project risk register, and is preparing a formalized transition plan that will be endorsed at the Director General level. The transition plan will include the necessary documentation in support of a formalized transition that identifies ISS risks and how they will be addressed or mitigated. The risk mitigation activities identified within the transition plan will be maintained throughout the duration of the C-130J ISS Contract to ensure risks post project closure are addressed or sufficiently mitigated, and that newly identified risks due to changes in the Weapon System may be captured and tracked in a risk register as per an existing Director General Aerospace Equipment Program Management, Air Force 9000 procedure Equipment Management Team 06.003.

OPI: ADM(Mat)

Target Date: July 2016

Financial Management

Management Action

The invoice certification process has been reviewed with contract, procurement and technical authorities, and supplemented with training where necessary. Future signatures will only be provided after invoice periods are complete and/or in accordance with authorized timelines. Additional Work Requirement forms were amended and processes revised and documented to conform to policy. Supporting documentation will be retained on file to validate that goods and services were provided in accordance with the C-130J ISS Contract.

OPI: ADM(Mat)

Target Date: Completed

Annex B—Audit Criteria

Criteria Assessment

The audit criteria were assessed using the following levels:

Assessment Level and Description

Level 1: Satisfactory

Level 2: Needs Minor Improvement

Level 3: Needs Moderate Improvement

Level 4: Needs Significant Improvement

Level 5: Unsatisfactory

Governance

1. Governance structures are in place to help manage the ISS elements of the Contract.

Assessment Level 1 – Performance data are regularly tracked in DRMIS and ways to increase the accuracy of the information are identified; however, further improvements are required to help support decision making. Performance measurement needs to be fully in place to implement the performance-based aspects of the Contract that will support the Contract Management Team’s ability to assess the Contractor’s performance.

Risk Management

2. Risks related to the ISS elements of the Contract are managed in accordance with departmental policy and guidelines.

Assessment Level 2 – The Contract Management Team is made aware of relevant ISS-related risks through project risk review meetings and other methods. To improve risk management, a formal risk management process, including a risk register, should be developed to help address and mitigate ISS risks when these risks are closed by the Project Management Office and new ISS risks are identified.

Controls

3. Controls over the ISS elements of the Contract provide value for money.

Assessment Level 3 – The Contract contains terms and conditions that help minimize risks to the Crown and provide value for money and the Contract is generally being administered in accordance with its terms and conditions. However, the Contract’s scalability for maintenance and training support needs to be expanded to be better aligned with the expected variability in RCAF operations. To improve value for money, greater visibility into the Contractor’s costing and pricing information is needed. Additionally, financial management needs improvement to be fully in accordance with the Financial Administration Act, government policies, and the contract terms and conditions.

Source of Criteria

Treasury Board Secretariat, Audit Criteria Related to the Management Accountability Framework: A Tool for Internal Auditors, March 2011.Governance:

• Reference to: ST-18, ST-20, ST-22, RP-2, RP-3, AC-1

Risk Management:

• Reference to: RM-1, RM-2, RM-4, RM-5, RM-6

Controls:

• Reference to: ST-5, ST-20, ST-22, RP-2

___________________________________________________________________________________________________________________________

Footnote 1 Royal Canadian Air Force (RCAF) website http://www.rcaf-arc.forces.gc.ca/en/aircraft-current/cc-130j.page. Last consulted on July 10, 2015.

Footnote 2 Referred to as “the Contract.”

Footnote 3 “ISS means all activities, including, but not limited to, engineering services (such as maintenance, repair, test and upgrade), logistics (such as parts supply, documentation and training) and related management functions, necessary to maintain a Canadian Forces platform throughout its service life.” Defence Administrative Orders and Directives (DAOD) 3022-0, Definitions, August 13, 2010.

Footnote 4 The Contract is based in United States Dollars (US$); however, all dollar values in this report are converted to Canadian Dollars (CA$) using the Bank of Canada posted exchange rate in effect on that date. These two amendments which total US$750 million are approximately CA$800 million.

Footnote 5 Defence Administrative Orders and Directives (DAOD) 3022-0, Procurement of In-Service Support for Canadian Forces Platforms, Definitions. “Canadian Forces platform means a major equipment system used by the Canadian Forces in the conduct of operations (e.g., aircraft, ship, tank).”

Footnote 6 Analysis of Contracts, March 2012: http://www.crs-csex.forces.gc.ca/reports-rapports/2012/179P0960-eng.aspx. Last consulted on June 16, 2015.

Footnote 7 Effective May 13, 2015, CRS has been renamed to ADM(RS).

Footnote 8 The C-130J aircraft is the label provided to the aircraft by the manufacturer Lockheed Martin Aeronautics. The C means “Cargo” and J is the latest model.

Footnote 9 http://www.lockheedmartin.com/us/products/c130.html. Last consulted on August 26, 2015.

Footnote 10 Capability Investment Database, Airlift Capability Project – Tactical, Capability Deficiency.

Footnote 11 CA$1.45 billion is approximately US$1.45 billion.

Footnote 12 CA$800 million is approximately US$750 million. The ISS Amendment was approximately CA$747 million or US$700 million. The Maintenance Training Suite Amendment was approximately CA$218 million or US$204 million; where CA$53 million or US$50 million was associated with ISS activities for training (CA$800 million = CA$747 million + CA$53 million).

Footnote 13 In-Service Support Contracting Framework (ISSCF), July 2009.

Footnote 14 ISSCF (2009), In-Service Support Guidance. “ISS contracts will normally be awarded to the platform suppliers, which may often be foreign-based companies.”

Footnote 15 PWGSC Notification letter for the Contract Option, February23, 2015.

Footnote 16 PWGSC Notification letter for the Contract Option, February23, 2015.

Footnote 17 ISSCF, July 2009, In-Service Support Contract Guidance.

Footnote 18 ISSCF, July 2009, “The Annual Operating Plan (AOP) is a contractor-prepared, Government of Canada-accepted plan that identifies the expected outcomes for the year (e.g. forecast yearly operating rate, schedules, equipment modifications, one-time engineering tasks, etc.). The purpose of the AOP is to identify contractor activities that could impact operations. For fixed-price contracts, the AOP is a coordination document, not a funding approval document. … If applicable, the AOP will detail the schedule for contractor-controlled maintenance, which will allow the DND to plan for scheduled equipment downtime as required. The AOP will also indicate which product improvements (usually not funded separately by the Government of Canada) and capability enhancements (usually funded separately by the Government of Canada) will be embodied over the coming year. If the impact on operations is acceptable, then the Government of Canada will accept the AOP.”

Footnote 19 Statement of Work 3.6.3.1, In-Service Support Performance Management.

Footnote 20 CRS Audit of the Weapons Effect Simulation (WES) Contract, December 2014, Recommendation 5a requires ADM(Mat) to enhance the [Procurement Administration Manual] PAM to clarify the requirement for risk management throughout the procurement life cycle.

Footnote 21 Director General Aerospace Equipment Program Management is a division within ADM(Mat).

Footnote 22 Director General Aerospace Equipment Program Management Air Force 9000 Equipment Management Team 06.003, October 2008.

Footnote 23 Technical authority certifies that “all goods/services have been received” in accordance with the “current [Statement of Work] SOW and have been found acceptable.” (Technical authority signature stamp on invoice.)