Analysis of Contracts

March 2012

7050-9-4-14 (CRS)

Reviewed by CRS in accordance with the Access to Information Act (AIA). Information UNCLASSIFIED.

Caveat

This analysis was conducted in order to determine which audits to include in the annual Chief Review Services (CRS) internal audit work plan. The analysis conclusions do not have the weight of an audit. It should also be noted that the analysis is not intended to assess the performance of contractors. Contractors have not been interviewed or otherwise asked to provide comment or feedback.

Acronyms and Abbreviations

Results in Brief

CRS developed a risk analysis methodology in 2003 to help identify potential contract audits to include in the annual audit work plan. This risk analysis was included in the fiscal year 2011/12 CRS Audit Work Plan.

The purpose of this analysis was to identify those higher-risk contracts that exhibit attributes that warrant audit or further review.

The analysis of goods and services contracts, tendered by Public Works and Government Services Canada (PWGSC), included a series of computer-assisted and physical documentation audit tests. These tests were developed to analyze the 157 active Department of National Defence (DND) contracts, with a contract value greater than $1 million and an expiry date on or after September 30, 2013 worth $15.1 billion. A risk scoring system was developed using six automated and 11 document review attributes (Annex A) to identify and rank those contracts that may require audit attention.

Outcome

In the 25 DND contracts with the highest cumulative risk attribute score

• 22 had significant subcontract work;
• 8 were with vendors on the PWGSC high-risk vendor list;
• 18 contained at least one high-risk method of payment;
• 13 contained the highest-risk basis of payment; and
• 5 had cost increases in excess of 50 percent.

In addition to serving as a means to rank the contracts, the documentation review also resulted in the identification of issues that will either be communicated through management letters or forwarded to other audit teams for follow up.

Introduction

Background

In compliance with government internal audit policy, CRS uses risk-based audit work plans in order to focus audit resources where the Department would realize the greatest benefit. Since 2003, CRS has applied an automated risk analysis process to help identify procurement contracts that may warrant closer audit scrutiny.¹ In the case of this analysis of contracts, the automated review phase was followed by a secondary file analysis, in order to further refine the risk ranking.

Objective

To identify those higher-risk contracts that exhibit attributes that warrant audit or further review.

Scope

The scope included active DND contracts tendered by PWGSC that expired on or after September 30, 2013 and were greater than $1 million in value. These parameters reduced the contract population subject to our analysis to 157. The value of these contracts totalled $15.1 billion.

A scope limitation in this analysis was the inability to conduct an analysis of contracts tendered by DND. The database containing this information, the Contract Data Management System, had too many incomplete fields to conduct a thorough analysis. A management letter will be sent to the appropriate office(s) to raise awareness of this issue.

Methodology

Sources of Data

• July 31, 2011 PWGSC Automated Buyer Environment (ABE) database extract of DND contracts.
• Audit Services Canada (ASC) audits of DND contracts completed since April 2003.

Filters

Two filters were applied in order to arrive at a manageable sample size of contracts:

• Filter 1. The PWGSC ABE database included 170,249 contracts from 1997 to present worth $79.6 billion. However, only 796 contracts worth $21.8 billion were found to be active with an expiry date on or after September 30, 2013.
• Filter 2. In order to further reduce the sample size, only contracts worth more than $1 million were included in this analysis. The sample was thereby reduced in size from 796 to 157 contracts. This sample represents $15.1 billion, or 69 percent of the $21.8 billion in contracts remaining after the first filter.

Risk Attributes

Six automated risk attributes² were applied to each of the 157 contracts. The highest 25 risk-ranked contracts, valued at $9.9 billion, were selected for secondary analysis.

1. Materiality
2. Cost Increase
3. Contract Award Process
4. Basis of Payment
5. Method of Payment
6. Duration

The following secondary analysis attributes³ were then applied to the remaining 25 contracts:

1. Statement of Work clearly defines deliverables
2. Complexity of the deliverables
3. Subcontractor and offshore supplier involvement
4. Cost/schedule increase with no scope change
5. Sufficient supporting documentation to verify the receipt of goods/services
6. Linkage of payments to the deliverables
7. Contract terms of payment and audit clauses as required
8. Segregation of duties in work authorization and approval process
9. Contract Manager/Project Manager status as a contractor versus government employee
10. Alignment with departmental outcomes
11. High-risk vendors as per the ASC high-risk vendors list

Outcomes

Twelve of the 25 contracts sampled each exhibited a high-risk score in eight of the 17 risk attributes. The top five high-risk contracts in the sample each received a high-risk rating in at least eight of the 17 risk attributes tested.

Subcontract Visibility. A recent CRS internal audit report on subcontractor visibility tested 57 contracts and determined that on average, 30 percent of the value of these contracts went to subcontractors. This subcontractor involvement can result in increased revenue-sharing risk, especially if suppliers are subsidiaries of the prime contractor. Additionally, offshore supplier involvement can result in increased foreign exchange risk.

Seven contracts received a high-risk rating due to the involvement of offshore subcontractors while 15 contracts received a medium-risk rating because of non-offshore subcontractor involvement. Without detailed subcontractor supporting documentation to accompany invoices, DND is exposed to significant risks as it is not in a position to determine validity or reasonableness of costs related to subcontractor work.

High-Risk Vendor. DND maintains contracting relationships with some vendors that have a history of over-billing or excess profits, and are considered higher risk.⁴ Six contracts tested were with vendors identified as previously having excess profits greater than $350,000 or 4 percent and therefore received a high-risk rating while two other contracts tested received a medium-risk rating. Overall, five of the seven highest-risk rated contracts, and eight of the 25 contracts tested involved vendors which exceeded contractual profit limits. In the absence of additional contract clauses or audits, contracting with high-risk vendors may increase profit risk exposure.

Contract Award Process. DND uses as many as 11 contract award processes with various levels of inherent risk. In addition to these risks, whether a contract was sole-sourced or competitively bid may increase the risk to the Department. Seventeen of the 25 contracts sampled received a high-risk rating based on their contract award process. All 14 sole-sourced contracts in our sample fell within this high-risk rating. Sole-source processes are generally considered less open, fair and transparent than competitive processes. Sole-source contracting may not be perceived as the best value for money irrespective of which contract award process is utilized.

Method of Payment. Each method of payment has a different amount of inherent risk. Contracts can have multiple terms of payment given the various deliverables within one comprehensive contract. There were 38 methods of payment in the 25 contracts sampled. See Appendix 4 of Annex A for method of payment breakdown. The review team allocated the percentage of each method of payment within a contract based on their respective proportion of the total payments.

Higher-risk contract methods of payment are evident in 18 of the contracts sampled. Advance and progress payments are considered a higher risk as they represent payments made for work not yet completed. Only three contracts had both advance and progress payments identified in the same contract. These methods of payment create elevated contract risk because payments are not linked to specific deliverables or completed products at the time of partial payment.

Basis of Payment. The terms of each contract may stipulate a variety of methods by which a supplier can be compensated for delivering various products or services. These methods have varying risk attributes such as poor cost transparency or little incentive for vendor efficiency. All of the risk attributes are not mutually exclusive. Whether a contract is sole-sourced or competitively bid can affect the risk level of a basis of payment in a contract. Firm fixed prices appear to be lower risk because they give maximum incentive to the vendor to control costs because the contractor assumes full responsibility for all costs under or over the firm price. A firm fixed-price contract that is sole-sourced does allow vendors to set their own price with a significant margin to reduce their cost-control incentive.

The top three bases of payment within each contract were determined for each of the 25 contracts. The highest-risk basis of payment, cost reimbursable with fee based on actual costs, was identified in 13 of the contracts tested. Cost reimbursable with a fixed-time rate is also high-risk and was found in 14 of the contracts sampled. See Appendix 5 of Annex A for a complete basis of payment breakdown. In the case of contracts with cost reimbursable elements, the price is not specified in the contract and is not determined until completion of the work. This increases risk to the Department, especially when adequate controls are not in place to ensure that the contractor is not using inefficient or wasteful methods.

Cost Increases. Unanticipated cost increases in contracts can have a serious impact on the delivery of a product or service or lead to poor value for money. Cost increases can be a symptom of poor planning, an opportunity to expand the scope of a project without going through another competitive bidding process or a contractor attempting to maximize revenue after winning the contract at a lower price.

Eleven of the 25 contracts tested had cost increases greater than 10 percent of the original value of the contract. Eight of these contracts had cost increases greater than 50 percent of the original contract value. The contract value increases in three of these eight contracts were a result of option years being exercised in the original contract while in two and increase in scope was negotiated. There was no increase in scope for the remaining three contracts that had a contract value increase of greater than 50 percent of the original contract value. Without a diligent independent amendment approval process, cost control risks and fair competition may be perceived to be in jeopardy.

Summary

All contracts are subject to varied levels of risk. The nature of business conducted by DND may render some of these risks unavoidable. Increasing risk awareness and understanding of the potential impacts of these risks enhance the implementation of risk mitigation strategies and thus reduce the level of risk to an acceptable level.

Annex A—Automated and Secondary Risk Analysis

Automated Risk Attributes

Six automated risk attributes were applied in the first phase of the analysis of the PWGSC ABE database. This database contains contract information on all DND contracts tendered by PWGSC. Eligible contracts in the ABE database had to meet the following criteria:

• All contracts must have had an expiry date after September 30, 2013;
• Each contract had to have a value greater than $1 million; and
• Only contracts with complete information were included in the analysis.

The automated risk attributes were applied to 157 eligible contracts. These 157 qualifying contracts, worth $15.1 billion, were ranked according to the automated analysis and the 25 highest-risk contracts, not previously audited, were selected for the secondary analysis.

The results of the six risk attributes are provided at Appendix 1. For each attribute, each contract was assigned a risk score. Contracts that scored higher were deemed to be higher risk than their lower-scoring counterparts.

Materiality. Higher-value contracts result in a potential higher-risk impact if poorly managed. Contract values were divided into seven risk ranges. The lowest range was $1 million to less than $5 million, with the highest range being greater than $1 billion.

Cost Increase. The escalation in the contract’s value was determined by comparing the original contract value to the most current value as per the most recent contract amendment. Options exercised by way of a signed amendment were included for the purpose this analysis. The resulting cost increase was then compared to the contract duration elapsed to date. Duration elapsed was divided into two categories: a 0-50 percent duration elapsed to date versus the contract duration and 51-100 percent duration elapsed to date versus the contract duration. A risk rating was assigned depending on the relevant cost increase relative to the duration elapsed.

Contract Award Process. There are a number of circumstances that result in a contract not being competitively tendered. Certain requirements such as combat systems with unique designs often result in sole-source in-services support contracts with higher-risk contract award processes due to exclusive rights. Contracts with a higher risk contract award process such as national security, which was assigned the highest-risk rating, were rated higher risk than those with the lower award processes such a best overall proposal, which rated in at the lowest possible risk rating.

Basis of Payment. It is more difficult to ensure value for money in the case of a cost-plus-profit basis of payment, which thus was assigned the highest possible risk rating. Those contracts with payments based on a vendor’s time and material do not provide full assurance of vendor efficiency and were also rated relatively high on the risk-rating scale. Firm price rated lowest, as the final price is known at the beginning of the contract, and therefore was assigned the lowest risk rating.

Method of Payment. Verifying receipt of services can be difficult for certain payment types. Advance payment was assigned the highest risk rating because no deliverables are received before payment. Monthly progress claims for services that could potentially be provided over many years can pose verification challenges unless specific milestones are set with related acceptance analysis. Payment upon delivery and acceptance had the lowest risk rating as the verification is straightforward once the service has been delivered.

Duration. Contract durations are correlated with risk levels. Although there could be some efficiency gains achieved by the vendor, longer-term contract obligations could also reduce the flexibility of the Department to seek out other vendors when the delivery of goods and services is unsatisfactory. Issues such as staff turnover, retention of knowledge, economic risks and unforeseeable circumstances also become a greater concern as the duration of a contract increases. Contracts lasting eight years or more were rated highest risk, while those with a duration of less than a year were assigned the lowest risk rating.

Secondary Analysis

In the second phase of the analysis, 11 secondary attributes were applied to the top 25 contracts, worth $9.9 billion or 45 percent of the value of the contracts eligible for the automated analysis. In this exercise the previous rankings were removed and the automated attributes were reexamined along with the secondary analysis using document review. The contract and annexes, amendments, statement of work and three most recent claims, along with a questionnaire completed by the requisition authority, were reviewed to complete this phase. This resulted in a new risk ranking of the top 25 contracts.

The results of the application of the 11 secondary attributes are at Appendix 1. Similar to the first phase of the analysis, in the second phase, all contracts were assigned a risk score, and contracts that scored the highest were considered to be the riskiest.

Statement of Work clearly defines deliverables. Section 34 of the Financial Administration Act (FAA) requires that goods and services received must agree with contract specifications and requirements. It is important, therefore, that the statement of work would clearly identify a contract’s deliverables. Risk ratings were given from low to high depending on the clarity and measurability of the deliverables.

Complexity of the deliverables. The complexity of the deliverables associated with a given contract influence the riskiness of a contract as these higher complexity contracts are more challenging to manage, monitor and evaluate. Those contracts with deliverables that are highly complex in nature, such as weapon systems, repair and overhaul (R&O), developmental projects or a complex mix of services received a higher risk rating than contracts with simple deliverables.

Subcontractor and offshore supplier involvement. The Departmentmay not always have visibility of subcontract-related charges when certifying invoices for payments because subcontractor supporting documentation is not required. The ability to substantiate subcontractor cost is particularly important for certain basis of payment types. The situation is further complicated when there are offshore suppliers involved. Contracts with vendors outside Canada are subjected to increased risk due to currency fluctuations and legal jurisdictions. Contracts with no subcontractor or offshore involvement were rated least risky, while those with subcontractors and more than one offshore supplier received the highest risk rating.

Cost/schedule increase with no scope change. In the case of both cost and timeline increases, the highest risk rating was assigned. However, adjustments were made to the risk score depending on the particular circumstances of the increase. If the increase was due to the exercise of an option, the rating was adjusted to low risk. Alternatively, if the cost or schedule increase was the result of a scope increase not related to the exercise of an option, the score was lowered to medium risk. This mid-range risk rating was applied in the case of a scope increase as this situation may indicate that there were issues/barriers in clearly identifying the deliverables within the statement of work.

Sufficient supporting documentation to verify the receipt of goods/services. When exercising Section 34 of the FAA, verification of receipt of goods and services, it is essential that sufficient supporting documentation be available to confirm receipt of goods and/or services as per the requirements of the contract. Invoices, claims and supporting documentation were verified to make this determination for the latest three claims. In the case of sufficient documentation, a low-risk rating was assigned, while a high-risk rating was scored when sufficient documentation was not available.

Linkage of payments to the deliverables. The Crown puts itself at risk when contract dollars are spent before receiving the deliverables. Therefore, payments to the vendor should be linked to the receipt of goods and services as required by the contract. In cases where this linkage does not occur, the Department may have less leverage to resolve downstream cost, schedule and performance issues. If the payments are linked to deliverables, a low-risk rating was assigned, whereas unclear linkages between payments and deliverables resulted in a higher-risk rating.

Contract terms of payment and audit clauses as required. To ensure that value for money is achieved for Canada, the PWGSC Supply Manual provides PWGSC contracting authorities with specific Standard Acquisition Clauses and Conditions that must be in place given the nature and selected bases and method of payment utilized within the contract. In the case of this analysis, if a clause was required, a low-risk rating was assigned if the clause was present in the contract. If the required clause was not present, it was assigned high-risk rating. Note that if there was a ceiling price or limitation of expenditure clause, then the risk rating for each basis of payment was reduced.

Segregation of duties in work authorization and approval process. Duties related to Section 34, certification of receipt of goods and services, and Section 33, payment authority, should be segregated. This is an essential control measure to ensure that only goods and services received by DND are paid for by the Department. A lack of segregation leads to concerns in relation to a greater risk exposure to the misuse of funds. A proper segregation was rated as low risk, while the absence of segregation of Sections 33 and 34 led to a high-risk score.

Contract Manager/Project Manager status as a contractor versus government employee. Contractors may be relatively less experienced in managing DND contracts, and may not possess the same level of corporate knowledge as an employee of the Department. In addition, contractors may have an incentive to prolong the timelines of projects in order to maximize the duration of their contract. Therefore, contracts managed by employees were scored at a lower risk level than contracts overseen by contractors.

Alignment with departmental outcomes. Contracts that are clearly linked to the core capabilities of the Canadian Forces and the organization’s ability to successfully deliver its mission are deemed to be higher risk. For example, from this perspective, a crucial complex weapons system contract would be higher risk than a contract for software licences and maintenance for systems unrelated to the core capabilities of the organization. In this instance, a contract in alignment with departmental outcomes was assigned a higher-risk score than contracts no so aligned.

High-risk vendors as per the ASC high-risk vendors list. In reviewing the PWGSC high-risk vendors list, CRS was able to identify vendors with a history of over-claims or excess profits on past contracts. Vendors with a history of at least 4 percent excess profit or over-claims equal to or greater than $350,000 were scored the highest risk rating.

Appendix 1 to Annex A—Automated Risk Attributes Ratings and Results

Table 1. Automated Risk Attributes Ratings and Results. Six automated review criteria were used to rank 157 eligible contracts so that 25 contracts could be selected for the secondary attribute analysis.

Appendix 2 to Annex A—Secondary Analysis Ratings and Results

Table 2. Secondary Analysis Ratings and Results. The top 25 contracts identified using the six automated risk attributes were subjected to 11 secondary attributes and ranked according to each contracts total risk score.

Appendix 3 to Annex A—Final Analysis Rankings

Table 3. Final Analysis Rankings. The highlighted contracts in the last two rows ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ and ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ have expiry dates before September 30, 2013 because ABE was incorrect. ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ has amendments likely to be exercised beyond September 30, 2013, but not the other contract.

Appendix 4 to Annex A—Method of Payment Attribute Analysis

Table 4. Method of Payment Attribute Analysis. The top 25 contracts identified included a total of 38 methods of payment. This table shows the number of times each method of payment was used.

Appendix 5 to Annex A—Basis of Payment Attribute Analysis

Table 5. Basis of Payment Attribute Analysis. The top 25 contracts identified included a total of 53 bases of payment. This table shows basis of payment breakdown for all 25 contracts.

___________________________________________________________________________________________________________________________

Footnote 1 Example of CRS report is Risk Analysis of Operations and Maintenance Contracts, April 2007 (http://www.crs-csex.forces.gc.ca/reports-rapports/pdf/2007/P0714om-eng.pdf).

Footnote 2 See Annex A for the application and a detailed description of each of the automated risk attributes, and Appendix 1 to Annex A for their associated scoring ranges and results.

Footnote 3 See Annex A for the application and a detailed description of each of the secondary risk analysis, and Appendix 2 to Annex A, for their associated scoring ranges and results.

Footnote 4 PWGSC has created a list of high- and medium-risk vendors, based on audits and investigations undertaken since April 2003. These investigations identified vendors with a history of over-billing exceeding $350,000 or excess profits exceeding 4 percent as high risk, and over-billing below $350,000 or excess profits below 4 percent as medium risk.