Internal Audit of Capital Project Cost Estimation

May 2013

7050-11-42 (CRS)

Reviewed by CRS in accordance with the Access to Information Act (AIA). Information UNCLASSIFIED.

Acronyms and Abbreviations

Results in Brief

Since 2008, Chief Review Services (CRS) has identified certain concerns regarding the accuracy of cost estimation in five capital acquisition internal audits. The Office of the Auditor General (OAG) has also raised concerns regarding project cost growth in a recent audit of DND acquisitions.¹ Given the magnitude of the capital program in the 2008 Canada First Defence Strategy (CFDS), an audit of project cost estimation was included in the CRS Risk-based Audit Plan. The Departmental Audit Committee directed that this audit begin in fiscal year (FY) 2011/12. The objective of the audit was to assess the reliability and rigour of the costing methodology used to estimate capital project costs. The audit focussed on project value estimates and the independent challenges carried out at three main project decision points that occur at approximately two year intervals, to establish the project budget before a contract is awarded to the defence industry: a rough estimate at the project identification phase; an indicative² estimate at the Project Approval (Definition) phase; and a substantive³ estimate at the Project Approval (Implementation) phase. Three major initiatives that should contribute to improved cost estimation were taking place concurrently with this audit: the Vice Chief of the Defence Staff Project Approval Process Redesign, related Defence Renewal Team initiatives, and a Treasury Board independent review of the life cycle cost framework for major military acquisitions. Lastly, as part of the Chief Financial Officer (CFO) transformation, it was planned that the cost estimation staff capacity would double in size.

Findings and Recommendations

Cost Estimation Accuracy. Insufficient rigour was applied to the cost estimation assumptions in the project identification phase due to insufficient standards for original cost estimates. The audit team’s review of seven identification phase project estimates found that long-term escalation rates were not standardized and contingency was not included in four of the projects. Although the DND Economic Model escalation rates are not intended to include technology improvements, some project sponsors took into account the technology enhancement costs for replacing equipment while others did not. The new mandate of the Defence Capability Board to ensure project capabilities are necessary and affordable may resolve this issue. Additionally, the original cost estimates will be challenged by the Director Costing Services before identification phase projects are included in the 20-year Long Range Planning Tool (LRPT).

In subsequent project phases, schedule delays, scope increases, unpredictable inflation and delays in engaging industry have contributed to cost growth. Over the last 7.5 years, the value of projects in the definition phase increased by 3.6 percent and the value of projects in the implementation phase increased by 1.6 percent. Cost growth associated with project delays was due to shortfalls in project management staff and the lengthy project approval process.⁴ The VCDS Project Approval Process Redesign project was on-going at the time of the audit, with the intention of streamlining the capital acquisition cycle time by 50 percent and doubling the throughput of project approvals. While recent cost validations in the options analysis and definition phases have not extended the project approval process, too much validation effort was found to be devoted to low-value items.

Higher confidence estimates would have been possible if industry had been engaged earlier for price and availability quotes before Project Approval (Definition) and if contracts had been tendered earlier. Unlike Major Crown Projects,⁵ lower-value projects were tendered to industry after Project Approval (Implementation). Delays in tendering contracts were due to a misunderstanding of Section 32 of the Financial Administration Act (FAA), which only requires certification of the availability of funds prior to contract award, not at the initiation of the tendering process.

It is recommended that clear standards for escalation rates, capability improvement factors, infrastructure estimates and contingency funding be developed for original project cost assumptions in the identification phase. In addition, cost validation activities should be risk-based to focus limited resources in areas of greatest benefit. The Project Approval Directive (PAD) should be revised to advance the timing of price and availability activities. It is also necessary to verify with Public Works and Government Services Canada (PWGSC) that lower-value projects can be tendered before Project Approval (Implementation) without FAA Section 32 certification.

Expected Confidence Level of Estimates. For complex military acquisitions, accurate project estimates early in a project life cycle are rarely possible given the insufficient maturity level of requirements and design. DND policy requires identification phase life cycle indicative cost estimates to have a high confidence level of +/- 25 percent even though the range could be +/-50 percent according to industry guidelines.⁶ Therefore, some project budgets were based on low confidence estimates that resulted in the use of capability trade-offs to offset cost increases and to remain within budget. There are project briefs required at the three main decision gates that have the option of discussing the confidence level of the cost information. This option was not exercised in a sample of recent project briefs. DND plans to adapt its existing costing tools and procedures once an updated Treasury Board procurement and project approval policy is promulgated.

It is recommended that the DND Costing Handbook and PAD be revised to address a sufficient range of cost variance in each project phase in accordance with industry standards and that the discussion of the confidence level of cost information in project briefs be mandatory.

Project approval policy requires precise cost estimates too early in a project life cycle. By not engaging industry earlier, project estimates have lower confidence levels. This results in some capital projects with budget ceilings that require capability trade-offs to control costs while remaining within acceptable tolerance levels of operational risk.

Note: For a more detailed list of CRS recommendations and management response, please refer to Annex A—Management Action Plan.

Introduction

Rationale for Audit

Since 2008, CRS has completed five audits of capital acquisition projects that identified certain concerns regarding the accuracy of cost estimates. A recent OAG report has also raised concerns regarding the cost growth in original project estimates. Given the magnitude of the capital program outlined in the 2008 CFDS, these audit observations warranted the inclusion of this audit in the CRS Risk-based Audit Plan for FY 2011/12 to FY 2013/14. The Departmental Audit Committee directed that this audit commence in FY 2011/12.

Background

Recent Initiatives. During the audit conduct phase, the following three initiatives were under way, and this should positively influence the DND cost estimation framework:

• In June 2012, the VCDS initiated the Project Approval Process Redesign project. The desired outcome of the project is the effective management of DND expenditures with project approval processes that are proportional to complexity, cost, urgency, importance and risk. The aim is to reduce project acquisition cycle time by 50 percent while doubling the throughput of projects. Co-chaired by the VCDS, the Project Approval Process Redesign council includes Canadian Armed Forces/DND stakeholders and representatives from the Treasury Board Secretariat, PWGSC, and Defence Construction Canada. Given the extensive work that CRS has done in the past⁷ on streamlining the project approval process, it was agreed that CRS would act as an observer on the Project Approval Process Redesign working group. In this capacity, the audit team provided information to the Project Approval Process Redesign project staff on capital program slippage.
• The Defence Renewal Team was established in August 2012 to support key activities relating to spending reduction and process reform by serving as a central point of contact on change initiatives for a three-year period. During the audit reporting phase, the cost avoidance opportunity associated with project approval streamlining was briefed to members of the Defence Renewal Team.8
• An independent review of the life cycle cost framework for large military acquisition projects was initiated by Treasury Board and completed in January 2013. The report highlighted the challenges of estimating precise life cycle costs in the early phases of a project. It stated that the “Government and the public often seek a higher degree of precision than can be justified with the information available.”⁹ A related report stated that the April 2006 DND Costing Handbook life cycle costing policy was comparable to the leading international best practices.¹⁰ Based on the independent review recommendations, it is the intention of Treasury Board to promulgate a new policy on procurement and project approval policy that will encompass a cost estimation framework.

Cost Estimation Framework. At the three major decision gates in a major capital project’s¹¹ nine-year life cycle, project costs are estimated then followed by an independent cost validation by the Director Costing Services as portrayed in Figure 1. The process details are as follows:

• At the outset of a project, known as the identification phase, a rough life cycle cost estimate is proposed by a Level 1 project sponsor. Prior to inclusion in the 20-year LRPT and the five-year Investment Plan, a review is performed by Director Costing Services and the project is approved by the Defence Finance Committee (DFC). All projects that require definition or implementation phase funding are to be included in the Investment Plan. Therefore, an identification phase project estimate could be required four years in advance of the project definition phase that will be approved in the fifth year of the Investment Plan.

Figure 1. Project Approval Process. To reduce the life cycle time between the identification approval and the close-out phase from thirteen years to nine, the milestones for each phase were set by the Project Management Board in December 2003. Normal two-year definition phases prescribed in the PAD may be extended for complex developmental projects and the implementation phase may also be longer. For construction projects, the implementation phase is usually less than three years.

• After a two-year options analysis phase, and depending on the risk profile of the project, expenditure approval by the Minister of National Defence (MND) or the Treasury Board will be required for the definition phase costs based on a substantive estimate. An indicative cost of the total project acquisition value is also estimated. Known as Project Approval (Definition), this two-year phase amounts to an average of 6.9 percent of the total project value. During this phase, the project office provides greater refinement to the indicative project estimate through increased engagement with industry.
• Once the two-year definition phase is complete, expenditure authority, known as Project Approval (Implementation), is sought for the five-year implementation phase – the complete project acquisition budget based on a substantive estimate. Life cycle support costs are acknowledged, but only the acquisition costs are approved.
• Should the expenditure authority for the definition or implementation phase be exceeded, the project must seek re-approval by the MND or Treasury Board. In these situations, all revised expenditure approvals greater than 10 percent or $10 million must be reviewed by the DFC.¹²

Objective

The objective of this audit was to assess the reliability and rigour of costing methodology used to estimate capital project costs.

Scope

The scope of the audit included the following:

• major capital equipment, infrastructure and Information Management (IM) projects only – minor projects less than $5 million were excluded;
• estimates in all phases of a project’s life cycle: identification, options analysis, definition and implementation.

The audit had the following scope limitations:

• Given the ongoing work of the VCDS Project Approval Process Redesign project taking place concurrently with the audit, a detailed process analysis to determine the root causes of schedule delay was excluded.
• Although it is standard practice in DND to consider life cycle costs during the options analysis phase of a project, it was decided at the outset of the audit that life cycle support costs would be excluded from the scope and be treated as a separate audit.¹³

Methodology

The audit results are based on evidence from the following sources:

• interviews with key DND, PWGSC, and Treasury Board staff;
• an examination of policies, project documentation, the Capability Investment Database, the corporate submission database, and the Investment Plan;
• a comparison with United Kingdom (UK) and United States (US) defence project cost estimation results for similar types of combat systems;
• a sample of 14 current IM, infrastructure, and equipment projects in different phases representing 33 percent of the major capital program’s value. (Each project office completed one questionnaire and participated in focus group discussions. Annex B contains the list of the projects reviewed.);
• from January 2005 to July 2012, 348 projects endorsed by the Project Management Board for definition or implementation phase funding; and
• a case study of the Canadian Surface Combatant (CSC) project cost estimates in the identification phase (See Annex C for detailed results)

Criteria

The audit criteria are outlined in Annex D.

Statement of Conformance

The audit findings and conclusions contained in this report are based on sufficient and appropriate audit evidence gathered in accordance with procedures that meet the Institute of Internal Auditors’ International Standards for the Professional Practice of Internal Auditing. The audit thus conforms with the Internal Auditing Standards for the Government of Canada, as supported by the results of the quality assurance and improvement program. The opinions expressed in this report are based on conditions as they existed at the time of the audit and apply only to the entity examined.

Findings and Recommendations

Cost Estimation Accuracy

Shortfalls exist in the cost estimation assumptions for projects in the identification phase. Project delay, scope increase, unpredictable inflation and delayed engagement with industry contributed to cost growth from the options analysis phase to the implementation phase.

Original Project Estimates

Although projects in the identification phase have limited information to create high quality estimates, shortfalls were evident in some cost assumptions made by project sponsors for long-term projects. As the approval authority for project briefs in the identification phase, the Defence Capability Board was recently mandated in December 2012 to challenge the capability and affordability of all future projects. This new mandate may address the audit observations on a sample of seven projects in the identification phase that aim to replace or upgrade major combat systems beyond the year 2022. Although these projects have not yet been subject to review by Director Costing Services prior to inclusion in the LRPT, the following observations were made:

• Contingency was not included in four project estimates, which normally range from 10 to 15 percent for Major Crown Projects and may be as high as 20 percent for complex projects.
• Long-term cost escalation factors were not in accordance with the DND Economic Model. Tailored escalation models are only applied to Major Crown Projects once they reach the options analysis phase.
• In some cases, cost escalation assumptions were based on replacing the same combat system without technology improvements. According to a firm that specializes in research and development, the technology/capability improvements alone for military aircraft amounted to 4.6 percent per year over a 30-year period.¹⁴ These escalation rates account for capability/technology enhancements driven by such factors as interoperability requirements and current defence industry norms.

Construction in Support of Equipment Projects. Defence policy requires equipment projects to include the associated infrastructure such as jetties, training facilities, and hangars. Initial infrastructure costs were underestimated for an audit sample of 12 equipment Major Crown Projects. A shortfall of construction engineering staff to provide early infrastructure cost estimates is a major cause. While the initial infrastructure estimates were 1.6 percent of the indicative costs of the 12 equipment projects, these estimates grew to an average of 6.2 percent over time. The new challenge function provided by the Defence Capability Board and the Director Costing Services, before inclusion in the LRPT, should alleviate this issue that has been observed by CRS in a past audit.¹⁵

Project Cost Growth. There are a number of cost growth factors in the project phases that follow the first identification phase estimate. From the review of 165 projects that began the definition phase since January 2005, the value of these projects increased by 3.6 percent. In the same time frame, the value of 183 projects in the implementation phase increased by 1.6 percent. Although some projects were de-scoped to remain within budget, our analysis of the 63 revised expenditure approvals identified the following root causes for cost growth: project delay, scope increase, inflation, and delayed engagement with industry.

Project Delay. Although the project delay costs related to the lengthy project approval process may be addressed by the VCDS Project Approval Process Redesign project, shortfalls in project management capacity¹⁶ are also a significant cause for project delay. Over the last three fiscal years, 17 percent of the capital acquisition budget allocations were postponed to future years. Reviews of the capital program¹⁷ indicate most delays occur in the options analysis and definition phases.

One element of the lengthy capital acqusition cycle time is the 170-working-day corporate submissions process currently required for both the definition and implementation phase expenditure approvals. Although a target submission cycle time of 113 days was set by the VCDS in July 2011, there have been significant obstacles in achieving this goal.¹⁸ The VCDS Project Approval Process Redesign project intends to streamline the capital acquisition process for lower-risk projects, which represented 89 percent of all projects at the time of the audit.

Focus groups with project management personnel from the audit sample of 14 projects have indicated that too much cost validation time was spent on low-value items such as travel and rounding errors. Although the DND Costing Handbook states that items under three percent of the project value do not require as much cost validation effort, all cost information is routinely validated.

Scope Increase. Known as the “triple constraint” in project management, project cost is interdependent with the schedule and the scope of the requirements. Since January 2005, there were 19 revised expenditure approvals for projects that involved an increase in scope. For construction projects, this was due to changes in the national building code and understated requirements in preliminary design. For equipment projects, the number of deliverables increased due to the new capabilities required in the Canadian Forces Transformation.

Inflation and Foreign Exchange. Volatile construction costs, particularly in Western Canada, have affected 16 construction projects. Unpredictable foreign exchange was also a factor in some off-shore procurements. Some countries manage foreign exchange risk at the national level rather than having their defence departments absorb this pressure.

Early Engagement with Industry. Projects are not obtaining price and availability estimates from industry sufficiently early to provide higher confidence levels. The PAD states that price and availability activity should take place in the definition phase. This has occurred in six of the seven projects reviewed. Often, however, this activity takes place too late to obtain indicative project costs for Project Approval (Definition). As well, if contract terms and conditions are communicated early to industry, then price and availability quotes would include the risk premiums associated with contractor liability or liquidated damages.

In accordance with the PAD, tendering prior to the Project Approval (Implementation) is the most concrete means of obtaining substantive costs for capital equipment acquisitions and is the normal practice for high-valueMajor Crown Projects.¹⁹ However, an audit sample of 59 lower-value projects found that 46 projects were tendered after Project Approval (Implementation). Due to the lower confidence in these implementation phase estimates, the projects included more contingency funding that is often not spent. In a sample of 63 completed projects, it was determined that 68 percent of the projects that were lower in value did not spend any contingency funds. Some commodities have a bid price that expires before expenditure approval can be obtained. Whenever possible, requests for proposals should include a bid validity period that is sufficient for the duration of the approval process.

The intent of FAA Section 32 certification is to provide assurance that funds are available prior to a contract award, not at the initiation of the tendering process. Substantive project costs are not usually known until after the tendering process. However, with the exception of Major Crown Projects, PWGSC requires Section 32 certification to initiate the tendering process as a convenience to make it unnecessary to verify the availability of funds with DND just prior to contract award. This Section 32 requirement acts as an obstacle for DND staff to initiate the tendering process without expenditure authority. Although there is a risk of tendering a contract that may ultimately not be awarded, the projects are in the DND five-year Investment Plan and the project offices are already established in the definition phase. Following a tendering process similar to Major Crown Projects for lower-value projects would add certainty to cost estimates prior to implementation expenditure approval.

Summary. Recent controls have been put in place to challenge original project sponsor estimates in the identification phase and provide a consistent approach to estimating project cost escalation rates, technology improvement factors, infrastructure requirements, and contingency funding. Clear criteria for cost assumptions in the identification phase would reduce the amount of validation effort needed before a project is endorsed for inclusion in the LRPT. Engagement with industry for price and availability did not occur for some high and low-value projects until the definition phase. The tendering process did not take place for most lower-value projects until after expenditure approval for the implementation phase. Therefore, the estimates were lower in confidence and required more project contingency funds, and were later found to be unnecessary for those lower-value projects.

Confidence Level of Estimates

High confidence project estimates early in the project life-cycle are rarely possible due to the insufficient maturity level of requirements and design.

Confidence Level Policy. Projects in the identification phase that require development or significant modification of existing products are expected to estimate complete life cycle indicative costs up to six or more years before a contract award. In this phase, project requirements have not been refined and a design does not even exist. Although the industry guidelines relating to the Project Management Body of Knowledge for projects in the identification phase could be a variance estimate of +/- 50 percent, the PAD requires identification phase life cycle indicative cost estimates to have a range of +/- 25 percent, as defined in the DND Costing Handbook. According to nine of the fourteen focus groups, there is insufficient staff in the identification phase to support the costing exercise. As an example, the estimation challenges faced by the CSC project are in Annex C.

For approval of identification, definition, and implementation phases, the PAD requires a project brief to discuss how the scope, schedule and costs are established as a performance baseline. It is optional to discuss the strength and confidence of cost data, an option that was not implemented in a sample of 10 recent project briefs.

Identification Phase Requirements. Project requirements are not sufficiently developed in the identification phase to enable high confidence estimates. High-level mandatory capabilities are a general description of an operational requirement, but they do not take into account the type of equipment or a specific solution. Even if a preliminary statement of requirement was completed, the confidence level of the estimate would still be less than 60 percent according to the 14 questionnaires received from the audit sample of projects.

Summary. The DND policy of a +/- 25 percent variance for indicative costs in a project’s identification phase does not recognize the wide range of variance associated with developmental projects. As well, the confidence level of cost information is not mandatory in project briefs. This policy has led to some constrained project budgets that require capability trade-offs with acceptable operational risk to remain within expenditure approval ceilings.

Cost Estimation Training, Tools and Staff

Current shortfalls in training, tools, and staff will hamper the CFO’s new expanded cost estimation role if left unaddressed.

CFO Transformation. Since June 2011, ADM(Fin CS), as the CFO, has had an expanded responsibility for financial resource allocation, costing, in-year financial management, maintenance, and reporting of the Investment Plan, as well as for Treasury Board submissions. To increase the capacity of the 12 staff in the Director Costing Services cost validation section, six personnel were recently hired to commence cost estimation training. As the Director Costing Services staff will more than double in size by September 2013, it is intended to have their cost estimators embedded in high-value project offices and provide a centralized resource to assist smaller projects.

Cost Estimation Training Certification. While Director Costing Services cost validators have extensive experience, educational backgrounds and accounting designations, there are few certified cost estimators at DND. The Society of Cost Estimation and Analysis and the Association for the Advancement of Cost Engineering International are two organizations that provide certification for which the added value may outweigh the associated costs. With a four-year university degree, the Society’s core certifications consist of two exams and three years of experience; whereas the Association’s Cost Engineer certification requires four years of experience and the completion of a research paper. DND currently relies on informal on-the-job training and does not offer training related to cost estimation. Director Costing Services is currently evaluating costing capability requirements for the purposes of introducing life cycle costing with existing certification and training available in Canada and other countries.

Cost Templates. With the exception of Royal Canadian Navy and construction projects, there are no standard detailed cost templates for project office estimates. However, from our 14 focus groups, the main needs identified were as follows:

• a comprehensive listing of all potential cost items to be included in an estimate;
• flowcharts describing the numerous steps and considerations in a cost estimate as templates change frequently; and
• a central repository of best practices or current examples available to facilitate projects in developing quality cost estimates.

Tools. There are requirements for improved DND costing tools. Since 2008, ADM(Mat) has had a requirement for a costing tool to make costing data more easily replicated. However, this requirement is currently deemed a low priority – number 18 of 20 on the list of Materiel, Acquisition and Support priorities. As part of the CFO transformation, ADM(Fin CS) has determined that a requirement exists for costing tools, such as the Automated Cost Estimating Integrated Tools, to produce better cost estimates. A review of tools that projects have used in the past is planned by Director Costing Services.

The DND Costing Handbook could be improved with additional examples and guidelines. For example, the current cost model for Integrated Logistics Support (ILS)²⁰ acquisition costs does not take into account various procurement types. Annex E is a CRS analysis of past ship, vehicle, and aircraft projects performed to assess the different ILS breakouts in acquisition and operations and maintenance (O&M) costs. The audit focus groups also suggested a greater need for increased use of tailored escalation models for individual projects. Currently, there is only one tailored model for a Navy project in the DND Economic Model.

Shortfalls in Cost Estimation Staff. There are shortfalls in the capacity of ADM(Fin CS) to fulfill the new role of project cost estimation. The historical average of completed cost validations was 46 per year; however, the demand for cost estimation/validation will likely triple in future years. As well, for some Major Crown Projects, there will be an increase in cost estimation workload associated with the new Memorandum to Cabinet format that requires life cycle cost analysis for all alternative options.

Summary. There are plans to address the current limitations of the independent cost estimation capacity in the Department to accommodate the expanded role of ADM(Fin CS) in project cost estimation. The shortfall in standard cost templates, tailored escalation models, and forecasting tools reduces the confidence level of project estimates.

Performance Measurement

Measurement of cost estimation performance is not taking place to compare actual costs to original estimates and help improve cost estimation accuracy.

Historical Cost Comparisons. There are no standard project cost breakdowns for all types of projects seeking expenditure approval. Although every project is required to complete a post-completion report, there is no standard cost breakdown template. Therefore, it is not possible to compare actual costs to original estimates. From our examination of 95 completed projects, 72 percent of the post-completion report cost breakdowns could not be matched to the implementation expenditure approval cost breakdowns. As well, it was not possible to determine the original identification and definition phase estimates due to the nine-year life cycle of projects.

Benchmarking with Allies. Currently, DND does not compare cost estimation performance with our allies. Todetermine if DND’s cost estimation accuracy resembled that of allies, CRS compared 20 DND high-value projects in the implementation phase to the cost growth of similar projects reported in US and UK audit reports. While the median unit cost increase of deliverables of the Canadian projects from Project Approval (Definition) was 12.4 percent,²¹ the cost growth was higher for both the US projects at 15.1 percent and the UK projects at 13.5 percent.

Summary. With the exception of construction projects, expenditure approval documents and post-completion reports do not have a standard cost breakdown format to enable performance measurement to improve cost estimation. Without cost estimation benchmarking, it is difficult to establish reasonable cost estimation performance targets.

Conclusion

While some improvements to the cost estimation framework are necessary, efforts currently being made by senior management to improve the reliability and rigour of costing methodology should move DND forward in its ability to estimate capital project costs. It is clear that, as stated in a Treasury Board-funded review of the life cycle cost framework, the government and the public often expect greater cost estimation precision than is possible with the available information. Within this government context, the Treasury Board is currently revisiting its policy framework on procurement and project approvals, an initiative that is intended to address some of the system-wide issues that have surfaced in recent years.

Annex A—Management Action Plan

CRS uses recommendation significance criteria as follows:

High—Controls are not in place or are inadequate. Important issues are identified that could negatively impact the achievement of program/operational objectives.

Moderate—Controls are in place but are not being sufficiently complied with. Issues are identified that could negatively impact the efficiency and effectiveness of operations.

Low—Controls are in place but the level of compliance varies.

Cost Estimation Accuracy

Management Action

Develop a standardized approach and cost estimating tools to cost and validate major capital projects before their inception into the Investment Plan.

ADM(Fin CS) is transitioning from its cost validation process to a full costing process, which includes having an embedded coster in the Project Management Office. To ensure a challenge function, a new standard and evaluation cell will be created within Director Costing Services.

OPI: ADM(Fin CS)/D Cost S/DFPA/DRDC/CORA

Target Date: December 2013

Management Action

Once ADM(Fin CS) has developed the new standards, VCDS/Chief of Programme (C Prog) will review the changes and revise the PAD to reflect the changes as appropriate. The PAD changes should be fully implemented within six months of receipt of the new standards from ADM(Fin CS).

OPI: VCDS

Target Date: June 2014

Management Action

Notwithstanding the constraint of the bid validity period stated in the tendering document, which may limit the time required to obtain approval that necessitates expenditure authority, ADM(Mat) will lead an initiative, in conjunction with ADM(Fin CS) and PWGSC, to validate whether tendering of contracts can proceed without expenditure authority and FAA Section 32 certification.

OPI: ADM(Mat)/DGMSSC/DMPPOCI: ADM(Fin CS), ADM(Mat)/DCOS/DMG Compt

Target Date: November 2013

Confidence Level of Estimates

Management Action

D Cost S will update the Costing Handbook and develop a costing methodology and framework for projects at the identification phase before inclusion into the LRPT.

OPI: ADM(Fin CS)/D Cost S/DRDC/CORA

Target Date: December 2013

Management Action

Once ADM(Fin CS) has revised the Costing Handbook, VCDS/C Prog will review the changes and revise the PAD to reflect the changes as appropriate. The PAD changes should be fully implemented within six months of receipt of the new Costing Handbook from ADM(Fin CS).

OPI: VCDS

Target Date: June 2014

Cost Estimation Training, Tools and Staff

Management Action

Costing capability requirements will be assessed. Standard qualifications will be developed. A training program will be implemented for D Cost S personnel.

Standardized templates will be developed at the macro level in accordance with Treasury Board Secretariat guidelines.

D Cost S and the Centre for Operational Research and Analysis (CORA) are currently assessing different tools (software) to facilitate cost estimation.

OPI: ADM(Fin CS)/D Cost S

Target Date: March 2014

Performance Measurement

Management Action

As per the second bullet of 6, above, standardized templates (cost breakdown structure level 1) will be developed at the macro level in accordance with Treasury Board Secretariat guidelines.

D Cost S will institute a project closure benchmarking and reporting initiative in order to capture performance data.

OPI: ADM(Fin CS)/D Cost S

Target Date: March 2014

Management Action

Once ADM(Fin CS) has developed the new standards, VCDS/C Prog will review the changes and revise the PAD to reflect the changes as appropriate. The PAD changes should be fully implemented within six months of receipt of the new standards from ADM(Fin CS).

OPI: VCDS

Target Date: September 2014

Annex B—Audit Sample of Current Projects

Table 1. CRS Audit Sample of Current Projects. A mixture of equipment, IM and construction projects in different phases was reviewed by way of focus group questions and survey.

Annex C—Canadian Surface Combatant Case Study

As the CSC project will be the most significant capital expenditure in DND, it was used as a case study to determine the cost estimate challenges facing a developmental project in the identification phase. As stated in the 2008 CFDS, the CSC project will replace the three Tribal-class destroyers and 12 Halifax-class frigates. During the four-year identification phase that commenced in 2005, the CSC project had five estimates ranging ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ ²² using the DND Economic Model escalation rates. Revised estimates were necessary as benchmarks from our allies became available. In accordance with the capital acquisition process, the CSC project did not have the benefit of a preliminary statement of requirement which was delivered in the options analysis phase in November 2010.

These estimates were made at an early stage in the identification phase and resulted in a ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ It was acknowledged by the Department in March 2010 that there would ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ ²³ Based on current Halifax-class annual Personnel, Operations and Maintenance costs, for ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ ²⁴ ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓

A number of good practices were observed when reviewing the various cost estimates performed in the identification phase of the CSC project.

• In a 2006 estimate, a range of costs were considered: +50 percent / -25 percent and, if applied, ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓
• Costs were benchmarked with allies.
• Transferable modules for high-readiness ships were considered in 2006 to reduce costs.
• Automation studies were done in 2006 for a number of ship subsystems that had the potential of reducing a ship crew by 50 percent.
• In a 2008 estimate, sensitivity analysis was done with two annual escalation factors—1.8 percent and 3.5 percent. This resulted in an estimate range from ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓
• The 2008 estimate had 31 cost elements and 49 cost assumptions—significantly more than the earlier estimates.

A better project estimate would have resulted if not for the following practices:

• In one of the estimates, the learning curve based on the Canadian Patrol Frigate project was applied to both the ship construction and program costs.
• The incremental cost difference between a frigate and area air defence command ship was underestimated as 10.9 percent versus 46 percent.
• ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ This definition funding was split into two phases ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ ²⁵ ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓

Annex D—Audit Criteria

Objective

To assess the reliability and rigour of costing methodology used to estimate capital project costs.

Criteria

• Risk-based guidelines exist to assist with the preparation of project estimates.
• A formal and effective process is in place to challenge the assumptions and related resource allocations in a project estimate.
• A project estimate is developed at the appropriate level of detail with the assistance of costing models/applications.
• Reporting of actual results compared to the project estimates is done on a periodic basis to facilitate decision making with explanations for significant variances.
• Staff are provided with the necessary training, resources, and information to support their cost estimation responsibilities and have adequate experience to complete the tasks.

Source

Audit Criteria Related to the Management Accountability Framework: A Tool for Internal Auditors. Treasury Board of Canada Secretariat, March 2011.

Annex E—Integrated Logistics Support Cost Model

Table 2. Comparison of Life Cycle Cost Breakout for Past Projects to the Costing Handbook Model. There are significant acquisition, ILS and O&M cost differences between aircraft, ships and armoured vehicles.

___________________________________________________________________________________________________________________________

Footnote 1 Report of the Auditor General of Canada to the House of Commons. Chapter 6, Fall 2010.

Footnote 2 An indicative estimate has a variance of +/- 25 percent according to the DND Costing Handbook.

Footnote 3 substantive estimate has a variance of +/- 15 percent according to the DND Costing Handbook.

Footnote 4 In December 2003, the Program Management Board directed a 10-year life cycle from the time of the Synopsis Sheet (Identification) SS(ID) to project close-out in order to reduce the project life cycle from 13 years. The timeframe for the options analysis and definition phases would be four years. In September 2004, Assistant Deputy Minister (Materiel) (ADM(Mat)) adopted an acquisition cycle performance target of nine years, including a two-year options analysis phase, a two-year definition phase and a five-year implementation phase. The September 2011 PAD also provides for a maximum of two years for the options analysis phase (paragraph 1.1.23) and an average of two years for the definition phase (paragraph 2.6.5).

Footnote 5 Major Crown Projects are high-value or high-risk projects that require Treasury Board approval.

Footnote 6 Project Management Body of Knowledge, 4th Edition, Chapter 7.

Footnote 7 The CRS Treasury Board/Minister of National Defence Submission Process Review, July 2009, recommended a critical path to reduce the submission cycle time by 30 percent.

http://www.crs-csex.forces.gc.ca/reports-rapports/pdf/2009/134P0894-eng.pdf

Footnote 8 14 March 2013 CRS audit staff meeting with Defence Renewal Team staff.

Footnote 9 KPMG Large Military Acquisitions: Life Cycle Cost Framework, January 2013.

Footnote 10 KPMG Next Generation Fighter Capability: Life Cycle Cost Framework, November 2012.

Footnote 11 Major capital projects are projects greater than $5.0 million in value.

Footnote 12 The DFC is chaired by the Deputy Minister and supported by the CFO and the VCDS. These members sign off all expenditure approval submissions.

Footnote 13 Major audit stakeholders agreed on 23 March 12 that life cycle sustainment costs could be treated as a separate audit. The CRS Risk Based Audit Plan for FY 2013/14 to 2015/16 currently includes an audit of Capital Project Life Cycle Support Estimates.

Footnote 14 Why has the cost of fixed-wing aircraft risen? RAND Corporation, 2008. The lowest annual aircraft escalation rate of 6.7 percent was for an electronic warfare aircraft of which 2.1 percent accounted for material and labour cost escalation and 4.6 percent for technology/capability improvements.

Footnote 15 CRS Audit of Construction in Support of Capital Equipment Projects, September 2005.

Footnote 16 At the time of the audit, equipment project offices in ADM(Mat) had a vacancy rate of 23 percent that would increase to 29 percent in three years – minutes of Program Management Board, 5 July 2012.

Footnote 17 May 2011 DND Annual Performance Review and December 2010 VCDS review of 47 high-value projects both concluded that projects are on average four years behind schedule prior to the project implementation phase.

Footnote 18 The VCDS target was set based on the CRS 2009 Treasury Board/MND Submission Process Review.

Footnote 19 A Major Crown Project requires a Memo to Cabinet due to high-dollar value/risk in the options analysis.

Footnote 20 ILS includes training, spare parts, publications, simulators, technical data packages, special tools, and test equipment.

Footnote 21 Not accounting for fewer deliverables, the total net cost growth for the 20 DND projects was only 6.7 percent.

Footnote 22 All estimates reviewed were converted into a 15-ship estimate for comparison purposes.

Footnote 23 A briefing note to the Deputy Minister ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ may be delivered with a common hull design.

Footnote 24 2011 annual SRB presentation.

Footnote 25 ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓