Thematic Evaluation of Equipment Acquisition Effectiveness
Acronyms
Executive Summary
The evaluation focuses on the extent to which equipment acquisition projects remain aligned with the needs of the Canadian Armed Forces (CAF). This evaluation was launched at the same time as both the Chief of Force Development (CFD) and the Independent Review Panel for Defence Acquisition Office (IRPDAO) expressed an interest in examining the effectiveness of High Level Mandatory Requirements (HLMR) in the acquisition process. As such, this evaluation narrowed its focus to examine the role of HLMRs during the Implementation phase of acquisition projects. The evaluation was conducted in compliance with the Departmental Evaluation Plan, approved by the Performance Measurement and Evaluation Committee (PMEC), and with the Treasury Board of Canada Secretariat (TBS) Policy on Results (2016).
Program Description
Three DND/CAF Programs were included in this evaluation: Maritime; Land; and Aerospace Equipment Acquisition. All three programs seek to acquire new or modernized equipment required by the CAF through the Definition and Implementation phases of approved capital equipment projects.
Scope
The objective of this evaluation is to examine the extent to which defence equipment acquisition projects consistently align with the needs of the CAF. Ensuring this alignment ensures the CAF can carry out its intended capabilities in the field and contribute to Force Readiness. In 2015, HLMRs were introduced to clarify the main objectives of a project, underpin the project’s option’s analysis and better align projects with the DND/CAF Capability Based Planning outcomes.Footnote 1 Given the alignment role HLMRs are designed to play, the evaluation focused its scope to examine if HLMRs play their intended role in ensuring that the final piece of equipment resolves CAF capability deficiency (gaps). The evaluation also sought to examine how equipment acquisition projects contribute to Force Readiness.
Results
HLMRs play their intended role during project planning, but clear traceability between HLMRs and subsequent requirements was not well established or maintained during project implementation. When projects change scope past the planning phase, there is no evidence these changes are compared against the original equipment needs of the CAF or the HLMRs to ensure they still align. For example, three case study projects made changes to project scope following engagement with industry, and one project noted scope changes due, in part, to costs and schedule changes. A check against HLMRs was not noted. There is often a reluctance to change the HLMRs during the project implementation stages due to a perceived lengthy amendment process. The impact of this unclear trail back to the original expression of CAF needs increases the risk that a delivered capability would not be operationally useful and/or may provide an insufficient improvement over the status quo, thus prolonging a CAF capabilities gap. In turn, a lack of alignment between the project and CAF equipment needs could put general Force Readiness at risk.
Overall Conclusions
Clear traceability between HLMRs and subsequent project scope was not well-established and/or consistently identified. The weaknesses in documentation makes it difficult to conclude if and how the equipment acquired will meet the previously identified needs of the CAF and, once delivered, contribute to Force Readiness.
Recommendations
- Clarify the roles and responsibilities of those responsible for tracing HLMRs, as well as improve training and resources for proper tracing.
- Clarify how project scope, and changes to it, should be aligned with the original CAF needs and HLMRs throughout the project’s lifetime, including the role of HLMRs particularly during the Definition and Implementation phases.
Evaluation Scope
Coverage and Responsibilities
The objective of this evaluation is to examine the extent to which defence equipment acquisition projects consistently align with the needs of the CAF. Specifically, the evaluation sought to determine if acquired equipment would meet the needs of CAF stakeholders and to what extent the equipment would contribute to operational readiness. The evaluation also examined the role HLMRs play in ensuring traceability (or alignment) between the capability deficiencies originally defined by the CAF and the acquired equipment. This evaluation was launched at the same time both CFD and the IRPDAO expressed an interest in examining the effectiveness of HLMRs in the acquisition process. As such, this evaluation narrowed its focus to examine the role of HLMRs during implementation phases of acquisition projects. Information about the programs included in this evaluation can be found in Annex C.
Noting that a number of audits and evaluations of the acquisition process were taking place concurrently, and other studies were being planned, this evaluation examined six equipment acquisition projects as current examples of how acquisition guidance was applied in practice. While the evaluation did not provide an exhaustive examination of all acquisition projects, it sought to identify best practices and challenges that are occurring in the current context.
Evaluation Questions
The evaluation sought to answer the following two core questions:
- To what extent did the acquired equipment (or will the latest version of the equipment) fill the original capability deficiencies?
- To what extent does the Acquisition Process influence the alignment between the originally identified capability deficiency and the final product?
- To what extent did HLMRs play a role in aligning the equipment capabilities with the original capability deficiency?
- To what extent did/will the acquired equipment contribute to operational readiness?
Methodology
A process-based approach was used to examine the effectiveness (or anticipated effectiveness) of DND/CAF in acquiring equipment that meets CAF needs, using six acquisition case study projects as examples. The evaluation also used the Causal Link Monitoring methodology.
Case Study Projects
Since 2015, DND policies require equipment acquisition projects to include HLMRs; only projects with HLMRs were included in the case studies. The selected projects were among those furthest along in the Project Approval Process (PAP) (see Annex D - F) with HLMRs.
| Environment | Case Study Project | Annex |
|---|---|---|
| Aerospace Equipment Acquisition | Griffon Limited Life Extension (GLLE) | G |
| Aerospace Equipment Acquisition | Multi-Fleet Air Traffic Management Avionics (MFATMA) | H |
| Maritime Equipment Acquisition | Multi-Role Boat (MRB) | I |
| Maritime Equipment Acquisition | Naval Large Tugs (NLT) | J |
| Land Equipment Acquisition | Chemical Agent Sensor (CAS) | K |
| Land Equipment Acquisition | Light Weight Towed Howitzer (LWTH) | L |
Table 1 Summary
This table depicts the association between the project, its environment, and annex where the project summary can be found. This table has three columns. The left-hand column lists the environment, the middle column lists the case study project, and the right-hand column lists the associated annex. GLLE is hyperlinked to Annex G. MFATMA is hyperlinked to Annex H. MRB is hyperlinked to Annex I. NLT is hyperlinked to Annex J. CAS is hyperlinked to Annex K. LWTH is hyperlinked to annex L. Read across each row for the project and its corresponding environment and annex.
Detailed information about the evaluation methodology, limitations and the selection of case study projects can be found in Annex M.
FINDING 1: HLMRs generally played their intended role during project planning
HLMRs play a role in project planning phases.
HLMRs are foundational statements of specific CAF capability deficiencies and outline the objectives of a project.Footnote 2 Guidance documents indicate that HLMRs are used during the project planning phases to guide the development of the project scope, in particular the specific operational requirements for which the equipment must be used in the field.Footnote 3 HLMRs are also used to determine which project option would best meet CAF needs.Footnote 4 In practice, the evaluation’s group of Subject Matter Experts (SME) noted that HLMRs are useful during the project planning phases because they provide a clear project objective, help design the subsequent project scope, and assist with assessing project feasibility when selecting project options. However, HLMRs were also criticized for often being poorly written, making it difficult to use them to further design the project scope.
“HLMRs cannot be developed, nor understood, in isolation from other elements of a project.”
Most case study projects used the HLMRs for their intended purpose during the project planning stages.Footnote 5 Four of the six case study projects (MRB, NLT, GLLE and MFATMA) used HLMRs to develop the project scope, demonstrating clear alignment between the HLMRs and subsequent project details. In addition, five case study projects used HLMRs to select the most cost-effective project option that would respond to CAF equipment needs. The CAS did not have HLMRs but used System Effectiveness Requirements (SER) in a similar manner. While CAS used HLMRs to further describe the needs of the CAF, they did not use SERs to select the most cost effective project option. They did link the SERs to the project scope (specifically the equipment’s technical requirements) later in the project.
LWTH: Early Breakdown in HLMR alignment with Project Scope
The LWTH project did not demonstrate a link between the HLMRs and the rest of the project scope during the planning phases of the project.Footnote 6 While the project was effectively completed in 2018, this early breakdown in the alignment of project scope with HLMRs poses a risk that the acquired equipment may not meet the project’s original objectives or CAF needs once delivered.
FINDING 2: There is a breakdown in alignment between HLMRs and project scope during the implementation phases
HLMRs’ role during implementation stages is unclear.
Both the 2019 Vice Chief of the Defence Staff (VCDS) HLMR Directive and the Project Approval Directive (PAD) guide outline that projects should be traced back to HLMRs through the project lifecycle, using measures of the HLMRs.Footnote 7 The guidance documents also indicate that HLMR measures should have a clear pass/fail criteria. The use of “tracing” tools is also recommended to track alignment between the project scope and the HLMRs throughout the project.Footnote 8 The guidance documents do not, however, provide any additional direction on how to develop a methodology for measuring or tracing HLMRs throughout the project’s implementation stages.
The responsibility for ensuring project alignment with HLMRs during implementation stages is also unclear.
The PAD states “maintaining clear traceability between [HLMRs] and subsequent requirements, is a key task for project teams.” While the Project Team includes a Project Director (i.e., Sponsor) and a Project Manager (i.e., Assistant Deputy Minister (Materiel) (ADM(Mat)), the PAD does not clarify which team member is responsible for tracing HLMRs throughout the acquisition process. ADM(Mat) SMEs indicated tracing HLMRs was the responsibility of Project Sponsors. C Prog SMEs concurred. In contrast, Project Sponsors reported the responsibility for tracing was that of ADM(Mat).
Sponsors for the case study projects reported that only one training course was available to them before they took on their role. In addition, they noted the resources provided to them through the training and from the previous Sponsors were insufficient to effectively take over the management of the project.Footnote 9 If Project Sponsors are responsible for tracing HLMRs during project implementation, the lack of clarity around responsibilities could be due to a lack of training and resources.
Recommendation
Clarify the roles and responsibilities of those responsible for tracing HLMRs, as well as improve training and resources for proper tracing.
None of the case study projects track HLMRs during the project implementation phases.
As a result of the lack of clarity around the role of HLMRs during the implementation phases, none of the case study project teams measure or track HLMRs during the implementation phases.
Figure 1 Summary
This figure depicts the relationship and alignment between project scope and HLMRs during each project phase. This flowchart contains two main columns and three main rows. Column 1 (i.e., the left-most column) depicts Planning Phases and Column 2 (i.e., the right-most column) depicts the Implementation Phases. Row 1 (i.e., the top row) depicts the extent to which alignment exists between the project scope and HLMRs. Row 2 (i.e., middle row) depicts four project phases: ID, OA, DEF, and IMP. Row 3 (i.e., bottom row) depicts the two high-level project phase categories: Planning and Implementation. Read column by column to determine the extent to which alignment exists within each project phases.
MFATMA and GLLE attempted to link HLMRs to the performance measures but the measures speak to the success of the project (e.g., meeting timelines), not the HLMRs themselves.Footnote 10 While all the projects use a tool to track the project scope,Footnote 11 none of the projects link HLMRs to the rest of the project scope in these tools. Instead, the tools demonstrated the alignment between other aspects of the project scope, such as between the operational and technical requirements. See Annex N for more information on traceability practices.
Only two projects presented risks to the project scope, due to project changes, to oversight committees.Footnote 12
Only two case study projects [GLLE and MFATMA] regularly presented risks to project scope, HLMRs, and impact on CAF operations to oversight committees due to changes to the project. Only rarely were risks to scope addressed in the committees’ records of decision. The other case study projects (CAS, LWTH, NLT and MRB) rarely mentioned project scope, HLMRs or the impact on CAF operations in their presentations. Many factors can result in a change to project scope.
GLLE: Tracing Tools
While GLLE was the only project to include HLMRs in their tracing tool during the implementation phases, the HLMRs were not linked to the other aspects of the project scopeFootnote 13 in a way that would allow the tracking tool software to signal a need to re-examine the HLMRs if the project scope were to change. There is currently no evidence of whether changes to the project scope impact the project’s ability to deliver the HLMRs.
Project scope can change during the implementation phases.
All six case study projects made significant changes to the project scope during implementation phases of the project. The reasons for these changes were most often due to budgetFootnote 14 and industry capacity to deliver the equipment as originally designed.Footnote 15
| Case Study Project | Change |
|---|---|
| CAS | Removed 27 requirements from the project scope.Footnote 16 |
| LWTH | Made four changes to the equipment’s operational requirements for the field.Footnote 17 |
| MRB | Forty-eight changes to the equipment’s technical requirements were recorded.Footnote 18 |
| NLT | Made 331 changes to the equipment’s technical requirements.Footnote 19 |
| GLLE | Created a Change Request for four changes to the equipment’s intended operational requirements in the field.Footnote 20 |
| MFATMA | Recorded 14 changes to project scope in its tracking documents but did not specify what aspect of the project scope was changed.Footnote 21 |
Table 2 Summary
This table describes the changes to project scope for each case study project. This table has two columns. The left-hand column lists the case study project acronym and the right-hand column details the change(s) to the project scope. Read across each row for the case study project and its corresponding change(s) to scope.
As none of these projects linked its project scope (including operational and technical requirements) to the HLMRs,Footnote 22 it is unclear if the acquired equipment will align with the project’s original objectives or meet CAF needs.
Changes to project scope were not verified against the HLMRs.
Project tracing documents do not provide evidence that changes to project scope were compared against the HLMRs to ensure the project would still meet its original needs. Instead, when tracing was conducted, the documents focused on whether changes to the equipment’s technical requirements would impact the equipment’s anticipated operational requirement in the field.Footnote 23 Despite the fact that HLMRs are used to guide the development of operational requirements, there was no evidence that changes to the projects’ operational requirements were compared against the HLMRs.
LWTH: Scope changes were not compared against HLMRs.
In June 2013, the LWTH Senior Review Board agreed to increase the minimum temperature of the CCF/PKG fuze from -46 C to –32 C.Footnote 24 This would restrict the use of the LWTH in the Arctic to the short summer months, as average temperatures in the winter is two degrees colder (-34 C). The HLMR originally called for the LWTH to withstand temperatures of -46 C; however, there are no records of this change escalating to the Defence Capabilities Board (DCB) to examine the impact of the change on the HLMRs nor is there evidence the HLMRs were updated. As such, the final product may not achieve all the HLMRs.
Recommendation
Clarify how capability requirements and changes to them should be traced back to the capability deficiency throughout the PAP, including the role of HLMRs during the Definition and Implementation phases.
HLMRs are not updated when project context changes.
Not only are changes within the project scope not tracked back to HLMRs, but changes to the project’s external pressures are also not compared against HLMRs. Such external pressures could include the ability of industry to deliver equipment that meets the HLMRs or changes to the threat environment in which the equipment would be used. Project teams noted that HLMRs seldom change during the project implementation phases as the process for changing the HLMRs is understood to take a very long time, which would risk delaying the project.Footnote 25 As a result, HLMRs are not updated to reflect changing external pressures such as the changing threat environment.
TAPV: Did Not Modernize Project Objectives.
The concurrent Evaluation of Land Equipment Acquisition Program (2022)Footnote 26 found that the Tactical Armoured Patrol Vehicle (TAPV) equipment acquisition project does not meet the long-term strategic needs of the CAF, and is falling behind evolving global strategic circumstances. As a result, many TAPVs are not being used. If the objectives of the project had been revisited during implementation, the project could have been updated to ensure it was remaining relevant to the changing threat environment.
Equipment risks being unaligned with CAF current needs.
The Evaluation of Ready Land Forces (2022)Footnote 27 heard from CAF members that the long procurement process results in equipment being out of date by the time it is delivered. As a result, CAF equipment may not keep pace with that of Canada’s allies or stay ahead of the equipment of our adversaries.
FINDING 3: As project alignment with CAF needs is not tracked at a strategic level, the CAF cannot demonstrate its progress towards meeting capability requirements or achieving Force Readiness
There is a risk that projects may not meet CAF needs at either the individual project level or at a strategic level.
If Project Teams are not consistently assessing the scope of an acquisition project against the original CAF needs and project objectives (i.e., HLMRs), there is a risk that CAF equipment needs will not be met. In addition, there is no requirement for Project Teams to report their achievement of HLMRs back to an oversight body at the end of a project.Footnote 28 In the absence of such a reporting system, it is difficult to demonstrate how individual equipment acquisition projects are meeting the CAF’s higher level strategic capability objectives.
A system to track progress towards high-level capability requirements does not exist.
The Force Capability Plan outlines future force capability requirements. At this time, no system exists to track progress towards the requirements outlined in this plan.Footnote 29 This means that the progress of individual equipment acquisition projects towards addressing the CAF’s strategic level capability needs is not being done. If such a system is to exist in the future, Project Teams will need to consistently track their progress towards CAF needs and HLMRs using a standard method so their results can be collectively analyzed at the strategic level.
The Project Sponsors agree they will support Force Posture and Readiness; however, this is not reflected in the project documentation.
Five of the six Project Sponsors agreed or strongly agreed the project equipment will support Force Posture and Readiness requirements.Footnote 30 However, due to the lack of project traceability information, it is difficult to understand how the projects will contribute to Force Readiness.
Furthermore, three of the six projects are scoped to receive fewer components than originally requested in early project documentation.Footnote 31 Finally, the original timelines for all six projects have been extended.Footnote 32 Such changes and delays could negatively impact the CAF’s overall readiness.
GLLE: Project delays may result in equipment being obsolete upon delivery.
The GLLE project is intended to upgrade the Griffon helicopters so they may continue flying into the 2030s.Footnote 33 The project was launched in 2013 but did not receive funding until 2017.Footnote 34 Due to this initial delay, the project will now only complete the upgrades in 2029,Footnote 35 meaning the upgrades will be in place for only six years before the helicopters are replaced. Deciding whether funding for this project should have been directed to the purchasing of new helicopters instead is neither simple nor straightforward. Information that can establish and maintain a link between capability gap and project outcomes can support this decision.
Evidence from other evaluations has indicated that the equipment acquisition process is hindering Force Readiness.
The 2016 Evaluation of Land ReadinessFootnote 36 indicates that the lack of replacement vehicles for the foreseeable future would have a negative effect on the Army’s Force Readiness, particularly in the Arctic. The current Readiness Integrated Strategic Analysis (2022)Footnote 37 has also found challenges with the procurement of equipment due to the amount of equipment produced being less than originally planned. There is, therefore, a perceived risk that the current acquisition projects may not be sufficient to effectively support Force Readiness, at either the individual project level or at the higher strategic level.
The lack of HLMRs tracking at the project level and the lack of capability tracking at the strategic level puts the CAF’s Force Readiness at risk.
If projects do not track changing project scopes back to the original HLMRs, it is difficult to ensure the project will deliver on its original objectives. Furthermore, if HLMRs are not continuously re-examined and updated to ensure they remain relevant to the changing threat environment, the project may be obsolete once delivered. At a strategic level, if the progress of all acquisition projects towards the CAF’s overarching capability requirements is not tracked, it is difficult to determine the CAF’s state of readiness. Combined with timeline delays and changes to the number of equipment units being delivered, it is difficult for the CAF to demonstrate the extent to which acquisition projects will address the CAF’s capability needs and contribute to Force Readiness.
MFATMA: Project delays may result in fleets being grounded.
The MFATMA project is seeking to upgrade navigation equipment for a number of fleets to allow them to be compliant with current international flight regulations.Footnote 38 The fleets received a waiver for these regulations in 2017. The MFATMA project has requested a four month extension to the project closure date to December 2027.Footnote 39 &Footnote 40 At least one waiver is due to expire in 2025,Footnote 41 so there is a risk that this project delay may result in some fleets experiencing flying restrictions.
Compliance with standards and regulations are limitations applied to the project and do not come from the operational end user (unlike HLMRs). Compliance can be a key driver in a project; for this reason it is recognized they may need to be captured in HLMRs. Documenting the impact of key external drivers on the attainment of HLMRs supports timely communication between stakeholders. Improved HLMR tracking at a strategic level would in turn contribute to improved understanding of the impact on CAF Force Posture and Readiness through the delivery of capabilities by a project (e.g., Initial Operating Capability (IOC) and Final Operating Capability (FOC)). Strategic readiness calculations include IOC and FOC. As such, this evaluation supports the Readiness Evaluation Integrated Strategic Analysis (2022) Recommendation 1: Develop a departmental strategic approach to prioritize and streamline departmental efforts to support holistic CAF readiness. Holistic CAF readiness planning should be revisited on a cyclical basis to ensure alignment with broader objectives and changes in the future threat landscape.
Conclusion
HLMRs are intended to be aligned with the project scope throughout an equipment acquisition project, using a measureable and clear pass/fail criteria, in order to ensure the final product meets the capability needs of the CAF. In practice, the HLMRs play their intended role during the planning phases of a project by providing clear objectives, despite needing some refinement in how they are written. The role of HLMRs is, however, less clear during the project implementation phases, where HLMRs are not consistently measured or tracked to ensure the project remains in line with its original objectives.
The role of HLMRs in ensuring project alignment is particularly relevant when changes are made to the project scope during the implementation phases. However, there was no evidence in the case study projects that the HLMRs were examined after a change to the project scope to ensure project alignment. There appears to be resistance, in practice, to returning to the HLMRs in case this triggers a need to change the HLMRs themselves; a process that is poorly understood and perceived to be very time consuming.
Guidance documents should clarify how project scope, and changes to it, should be traced to the HLMRs throughout the project’s lifetime, particularly during the project’s implementation phases. The roles and responsibilities for tracing HLMRs should also be clarified. In addition, more training and resources for proper tracing practices should be developed. There is a risk that if these issues are not addressed, traceability of HLMRs will be lost in the later stages of the acquisition process and, as a result, CAF needs may not be met and Force Readiness may not be achieved.
Annex A — Findings and Recommendations
ADM(RS) Recommendation
1. Clarify the roles and responsibilities of those responsible for tracing HLMRs, as well as improve training and resources for proper tracing.
Action 1.1 - Terms of Reference
Initiate PAD amendments to applicable PAD Terms of Reference, in order to reinforce and clarify L1 roles and responsibilities related to HLMR development, validation and alignment, in order to improve capability and Force Development focus past Gateway 3.
OPI: CFD/VCDS
OCI: C Prog/VCDS, DCB
Target Date: Q3 22/23
Action 1.2 - Capability Launch Meeting
Conceive, design, build and manage a Capability Launch Meeting, comprised of a multidisciplinary team, in order to develop HLMRs, constraints and assumptions, once Gateway 1 has been achieved.
OPI: CFD/VCDS
OCI: C Prog/VCDS, DCB
Target Date: Q1 23/24
Action 1.3 - Digitization
Investigate and institutionalize, where feasible, digitization tools, Artificial Intelligence, software and resources that enable a digitized Force Development capability.
OPI: CFD/VCDS
OCI: DEA/CFD/VCDS, DRDC
Target Date: Q3 23/24
Action 1.4 - Training and Standards
Initiate the resourcing and business planning of Force Development Training and Standards Staff. Refine current courseware. Conceive, design, build and manage Force Development and OA phase courseware.
OPI: CFD/VCDS
OCI: VCDS
Target Date: Q3 23/24
Action 1.5 - Force Development and Design Doctrine
Conceive, design, build and manage Force Development and Design Doctrine.
OPI: CFD/VCDS
OCI: DCapA /DGCSI/CFD/VCDS
Target Date: Q3 23/24
ADM(RS) Recommendation
2. Clarify how project scope, and changes to it, should be aligned with the original CAF capability deficiency and HLMRs throughout the project’s lifetime, including the role of HLMRs, particularly during the DEF and IMP phases.
Action 2.1 - Force Capability Plan (FCP)
Concurrent to FCP release in the fall of 2022, introduce a requirement for all L1 sponsors to provide periodic FCP progress updates to DCB and IRPDA.
OPI: CFD/VCDS
OCI: VCDS, DCB
Target Date: March 2023
Action 2.2 - Documentation
Amend current PAP direction and templates to clarify HLMR role within the Statement of Operational Requirements (SOR).
OPI: CFD/VCDS
OCI: C Prog/VCDS, DCB
Target Date: Q4 22/23
Action 2.3 - HLMR Measurability
Conceive, design, build and manage HLMR Measurability throughout the PAP, amending current direction and applicable templates as required.
OPI: CFD/VCDS
OCI: C Prog/VCDS, DCB
Target Date: Q4 22/23
Action 2.4 - Measures of Capability
Amend current PAP direction and applicable templates to manage Measures of Capability throughout the PAP.
OPI: CFD/VCDS
OCI: C Prog/VCDS, DCB
Target Date: Q1 23/24
Action 2.5 - Capability Ladder
Conceive, design, and build Capability Ladder guidance, and incorporate within the PAP direction and applicable templates.
OPI: CFD/VCDS
OCI: C Prog/VCDS, DCB, PMB
Target Date: Q2 23/24
Annex C — Program Profile
DND/CAF Programs
This evaluation examined the alignment of acquisition projects with CAF needs and HLMRs across three DND/CAF Programs: Maritime Equipment Acquisition, Land Equipment Acquisition and Aerospace Equipment Acquisition.
Programs’ Objectives
The Equipment Acquisition Programs seek to acquire new or modernized equipment required by the CAF through Definition and Implementation of approved capital equipment projects. These programs aim to respond to evolving Defence requirements.
Equipment Acquisition Process
Defence equipment acquisition projects follow the PAP, “which ensures the delivery of Defence Procurement requirements”.Footnote 42 The PAP is composed of five distinct phases and two transition phases, which are outlined in Annex E.
| Groups of Primary Interest | Groups of Secondary Interest |
|---|---|
|
|
Project Sponsor ADM(Mat) |
Table 3 Summary
This table provides descriptions of the key program stakeholders. This table has two major columns. The column on the left is ‘Groups of Primary Interest,’ and nested underneath are the Project Sponsor and ADM(Mat). The column on the right is ‘Groups of Secondary Interest.’ Read down each column to identify the key stakeholders and their descriptions.
Core Responsibility
All three programs fall within DND/CAF Core Responsibility 5: “Procure advanced capabilities to maintain an advantage over potential adversaries and to keep pace with allies, while fully leveraging defence innovation and technology. Streamlined and flexible procurement arrangements ensure Defence is equipped to conduct missions.”
Performance Indicator Results
Percentage of projects that remain in approved scope, schedule and expenditure authority:
| TARGET: 100% | |||
|---|---|---|---|
| YEAR | LAND | MARITIME | AEROSPACE |
| 2020-2021 | 100% | 100% | 88.9% |
| 2019-2020 | 100% | 100% | 100% |
| 2018-2019 | 100% | 77.8% | 95.8% |
Table 4 Summary
This table depicts program results by program type. This table has four columns and four rows. The first column (i.e., the far left-hand column) lists the fiscal year, the second column lists Land program, the third column lists Maritime program, and the fourth column lists Aerospace program. The first row (top row) depicts 2020-2021. The second row (middle row) depicts 2019-2020). The third row (bottom row) depicts 2018-2019). Read across each row for the year and corresponding program indicator result by program type.
NOTE: The performance indicators aim to measure end-year performance at the aggregate level. Therefore, they use the most recent version of the scope, schedule and budget as a baseline. They do not measure the alignment of the project with the original capability deficiency (gap) identified by the CAF, timeline or budget.
Annex D — Capabilities Overview
Capabilities are defined by the Statement of Capabilities Deficiency Guide in the PAD as “the ability to deal with risks identified in scenarios or the risks associated with actual operations.” This includes the availability of personnel and materiel as well as quantitative assessment. In the context of equipment acquisition, capabilities are the abilities of the equipment to deal with risk in operational scenarios.
When an equipment acquisition project begins, a capability deficiency is defined by the Project Sponsor which identifies the capability gap or problem that the equipment will aim to address, either in full or in combination with other equipment projects.
The capability deficiency is then broken down into HLMRs, which are defined in the PAD as “Foundational statements of specific capabilities required by the CAF to meet government defence policy objectives, and they thereby set out the core objectives of the project.” In short, the HLMRs state the high-level capability objectives the equipment is meant to accomplish in operational contexts.
The Defence Terminology defines operational requirements as “an established need justifying the timely allocation of resources to achieve a capability to accomplish approved military or civil objectives, operations, missions, or actions.” In the Statement of Operational Requirements, the HLMRs are then further broken down into more specific Operational Requirements, which come in the form of stated performance objectives for the equipment acquisition project, expressed in operational or mission terms. In other words, operational requirements outline what the equipment will be able to accomplish in the context of an operation or mission.
In the System Requirements Document, operational requirements are translated into technical requirements. While a specific definition does not exist for this term, for the purpose of this evaluation “Technical Requirements” were defined as the qualities the equipment must have or what the equipment must be able to do.
For the purpose of this evaluation, a general term was required to group the capability deficiency, HLMRs, operational requirements and technical requirements together. CFD recommended the use of the term “project scope” for this purpose.
The PAD states that as projects progress through the PAP, each of the project scope requirements should be able to be traceable to (or aligned with) HLMRs. This process of ensuring alignment with HLMRs is know as tracing and the extent of the alignment is know as traceability.
Additional definitions can be found in Annex O.
Figure 2 Summary
This figure depicts the decomposition of the project scope. The project scope contains four elements. From left to right, they are: Capability Deficiency, HLMRs, Operational Requirements, and Technical Requirements. Equipment (i.e., the delivery of the equipment) is outside of the project scope. Read from left to right to understand how one element evolves into another element.
Annex E — Project Approval Process for Defence Equipment Acquisition
Figure 3 Summary
This figure depicts phases of the project approval process using the acronym for each phase. There are 8 phases. From left to right, these phases are: Pre-ID, ID, OA, T-DEF, DEF, T-IMP, IMP, and CL. Read from left to right to understand the order of phases.
Pre-Identification (Pre-ID) Phase
DND/CAF identifies capability deficiencies (or gaps) that need to be addressed.
Identification (ID) Phase
The Project Sponsor identifies the capability deficiencies in consultation with DND/CAF stakeholders. The Sponsor develops the Strategic Context (i.e., the business need, HLMRs and operational objectives that the equipment will achieve to address the deficiency) and screens the Preliminary Options.
Options Analysis (OA) Phase
The Project Sponsor presents project options, identifying the preferred option that will achieve the HLMRs and best address the capability deficiencies. An option will be endorsed by the SRB and approved by the DCB.
Transition to Definition (T-DEF) Phase
The Project Team receives departmental approval at the PMB, and receives project approval and expenditure authority from the Minister of National Defence (MND) or Treasury Board (TB).
Definition (DEF) Phase
The Project Leader transitions to a member from ADM(Mat) to determine how the preferred option will be implemented. The project scope and schedule will identify the activities and timelines required to accomplish project objectives.
Transition to Implementation (T-IMP) Phase
The Project Team develops and updates documents as required, receives departmental approval at PMB, and receives project approval, expenditure authority, and contracting authority from the MND or TB.
Implementation (IMP) Phase
A contractor is hired to carry out the project under the supervision of the Project Team. DND/CAF tests and verifies complex equipment and requests modifications if needed. When the project is complete, the equipment is delivered to the CAF, along with any necessary training.
Closeout (CL) Phase
The Project Team gives formal notice that the operational capabilities and project conditions has been achieved within the project limitations and that lessons learned have been recorded.
Project Evolution
All projects are assigned a Project Leader to oversee the equipment acquisition project. The Project Leader is from the Project Sponsor organization during the ID and OA phases and then transitions to a representative from ADM(Mat) at the DEF stage. Various stakeholders and governance bodies develop and approve the projects’ scope, ensuring that the capability deficiencies are translated to HLMRs, as well as operational and technical requirements. A description of the process for developing and approving equipment acquisition projects is located in Annex F.
Annex F — Detailed Project Approval Process
This map provides an overview of how the project scope is developed and approved throughout the Equipment Acquisition Process. It also identifies the Project Leader at different phases of the process.
| Project Leader | Project Phase | Steps |
|---|---|---|
| Project Leader is from the Sponsor Organization | Pre-ID | CAF CDs identified through the CBP, GoC priorities, Operational Community, & Business Support Community |
| ID | Gate 1: CD need funding source identified (by IRMC or CDM); CD created in SOCD or similar; HLMRs created in SCD; CD included in SCD; Gate 2: CD & HLMRs (in SCD) approved by DCB1; CD & HLMRs (in SCD) by IRP1 | |
| As Project Director, the Sponsor defines the capability requirements and confirms that the delivered capability satisfies the specified requirements | OA | CD and HLMRs included in BCA; Gate 3: CD & HLMRs (in BCA) approved by DCB2; ORs created in the SOR; CD, HLMRs & ORs included in SOR; CD, HLMRs & ORs (in SOR) endorsed by SRB; CD, HLMRs, & ORs (in BCA and SOR) reviewed by IRP2 |
| T-DEF | CD, HLMRs & ORs (in SOR) updated as required; Gate 4: PC & PMP approved by PMB | |
| Project Leader is from ADM(Mat) | DEF | CD, HLMRs & ORs (in SOR) updated as required; HLMRs & ORs (in SOR) endorsed by SRB; TRs created in SRD; ORs & TRs included in RFP; ORs & TRs (in RFP) approved by Sponsor, PSPC, & ISED |
| As Project Manager, ADM(Mat) defines and implements the project to obtain the required capability | T-IMP | CD, HLMRs & ORs (in SOR) updated as required; CD, HLMRs & ORs (in SOR) endorsed by SRB; Gate 5: PA(IMP) approved by PMB |
| IMP | ORs & TRs included in contract; ORs & TRs (in contract) approved by PSPC; Equipment created by contractor; Equipment tested by ADM(Mat), Contractor, & CAF; Gate 6: Equipment (in IOCC) approved by SRB; Gate 7: Equipment (in FOCC) approved by SRB |
Table 5 Summary
This table depicts the detailed stages of the project approval process by project phase as well as the Project Leader for each phase. There are three major columns. The first column (i.e., the left-hand column) describes who the Project Leader is and how the responsibility moves from one individual to another through the process. Read top to bottom to understand the changes in the Project Leader responsibility. The second column (i.e., the centre column) lists the project phases using the acronym for each phase. Read top to bottom to understand the flow of the phases. The third column (i.e., the right-hand column) depicts the steps associated with each phase. Each of the rows in the column should be read left to right to understand the progression of the steps in each phase. Read the full table across each row to gain an understanding of the project leader responsible for each project phase and corresponding steps in each phase.
Annex G — Griffon Limited Life Extension (GLLE)
| Years | Event | Phase |
|---|---|---|
| 2013 | Project Start | ID; OA |
| 2014 | - | OA |
| 2015 | VCDS HLMR Directive 2015 | OA |
| 2016 | - | OA |
| 2017 | BCA Completed; SOR Completed | OA |
| 2018 | - | OA |
| 2019 | - | DEF |
| 2020 | - | DEF |
| 2021 | - | DEF |
| 2022 | Issued RFP/Contract Awarded | IMP |
| 2023 | - | IMP |
| 2024 | - | IMP |
| 2025 | - | IMP |
| 2026 | - | IMP |
| 2027 | - | IMP |
| 2028 | - | IMP |
| 2029 | Project End | CLOSE |
Table 6 Summary
This table depicts the timeline of the GLLE project. The table has five major columns identifying the project phases using the acronym for each phase. From left to right, these are: ID, OA, DEF, IMP, and CL. This table has three major rows: key activity (i.e., the top row), the year (i.e., the middle row), and the project phase (i.e., the bottom row). Read column by column to identify what year the activity and project phase took place.
| Traceability Tools | Key Traceability Documents | Traceability Information Inclusion | Traceability Information |
|---|---|---|---|
Traceability Matrix
IBM DOORS |
BCA | Included | Links Capability Deficiency to HLMRs |
| Included | Uses HLMRs to select options | ||
| SOR | Included | Links Capability Deficiency to HLMRs | |
| Reflected | Links HLMRs to performance measures | ||
| Included | Links HLMRs to operational requirements | ||
| Included | HLMRs and performance measures reflected in IOC and FOC | ||
| IBM DOORS | Included | Includes HLMRs | |
| Not Included | Does not link HLMRs to operational and/or technical requirements | ||
| Included | Links operational and technical requirements |
Table 7 Summary
This table identifies what traceability information was included in each traceability document for the GLLE project. This table has four major columns and three major rows. The first column (i.e., left-most column) lists the traceability tools used by the project. The second column lists key traceability documents. The third column indicates whether traceability information was included in the document, partially reflected in the document, or not included in the document. The fourth column indicates the traceability information. The first row in column 2 (i.e., the top row) reads BCA. The second row in column 2 (i.e., middle row) reads SOR. The third row in column 2 (i.e., bottom row) reads IBM DOORS. Read across the rows to determine what traceability information was included in each document.
Case Study Summary
The GLLE, initiated in 2013, is meant to extend helicopters’ lives to 2030 until replacement crafts are introduced. While the initial estimate for project close-out was 2022, only shortly after its expected end of life, it has since been extended to March 2029.
The Business Case Analysis (BCA) provided a link between the capability deficiency and the HLMRs. HLMRs were also used to select a project option during OA. In the SOR, the Requirements Table identifies to which HLMR each operational requirement is linked. The first HLMR is reflected in one performance measure but no measurement methodology is provided. GLLE included the HLMRs in its DOORS system but does not link the HLMRs to individual operational or technical requirements.
Annex H — Multi-Fleet Air Traffic Management Avionics (MFATMA)
| Years | Event | Phase |
|---|---|---|
| 2015 | Project Start; VCDS HLMR Directive 2015 | ID |
| 2016 | - | ID |
| 2017 | - | OA |
| 2018 | - | OA |
| 2019 | BCA Completed | OA |
| 2020 | SOR Completed | OA; DEF; IMP Group 1 |
| 2021 | - | DEF; IMP Group1 |
| 2022 | - | DEF; IMP Group1 |
| 2023 | - | DEF; IMP Group1 |
| 2024 | - | IMP Group 1; IMP Group 2 |
| 2025 | - | IMP Group 1; IMP Group 2 |
| 2026 | - | IMP Group 1; IMP Group 2 |
| 2027 | - | IMP Group 2; CLOSE |
| 2028 | Project End | CLOSE |
Table 8 Summary
This table depicts the timeline of the MFATMA project. The table has five major columns identifying the project phases using the acronym for each phase. From left to right, these are: ID, OA, DEF, IMP, and CL. This table has three major rows: key activity (i.e., the top row), the year (i.e., the middle row), and the project phase (i.e., the bottom row). Read column by column to identify what year the activity and project phase took place.
| Traceability Tools | Key Traceability Documents | Traceability Information Inclusion | Traceability Information |
|---|---|---|---|
Traceability Matrix
Briefing Note
Transmittal Sheet
Change Register Worksheet |
BCA | Included | Links Capability Deficiency to HLMRs |
| Included | Uses HLMRs to select options | ||
| SOR | Included | Links Capability Deficiency to HLMRs | |
| Reflected | HLMRs reflected in performance measures | ||
| Included | Links HLMRs to operational Requirements | ||
| Included | Links performance measures to IOC and FOC | ||
| Change Register | Not Included | Does not include HLMRs | |
| Not Included | Does not link HLMRs to operational and/or technical requirements | ||
| Not Included | Does not link operational and technical requirements |
Table 9 Summary
This table identifies what traceability information was included in each traceability document for the MFATMA project. This table has four major columns and three major rows. The first column (i.e., left-most column) lists the traceability tools used by the project. The second column lists key traceability documents. The third column indicates whether traceability information was included in the document, partially reflected in the document, or not included in the document. The fourth column indicates the traceability information. The first row in column 2 (i.e., the top row) reads BCA. The second row in column 2 (i.e., middle row) reads SOR. The third row in column 2 (i.e., bottom row) reads Change Register. Read across the rows to determine what traceability information was included in each document.
Case Study Summary
The MFATMA project seeks to upgrade the avionics of multiple RCAF fleets found to be non-compliant with air control regulations introduced in 2019. Without the upgrade, RCAF fleets would be unable to conduct operations globally with allied forces. Project close-out has recently been delayed until 2029.
The MFATMA BCA links the HLMRs to the Capability Deficiency and uses the HLMRs to select options during OA. The SOR links the HLMRs to the different systems that need to be upgraded. The performance measure requires 100% of all fleets to be upgraded. No other measure is provided for the HLMRs. Traceability documents provided did not demonstrate a link between the HLMRs and the operational and technical requirements.
Annex I — Multi-Role Boat (MRB)
| Years | Event | Phase |
|---|---|---|
| 2012 | Project Start | ID; OA |
| 2013 | - | OA |
| 2014 | BCA Completed | OA |
| 2015 | VCDS HLMR Directive 2015 | OA |
| 2016 | - | OA |
| 2017 | - | OA; DEF |
| 2018 | SOR Completed | DEF |
| 2019 | - | DEF |
| 2020 | Contract Awarded | DEF; IMP |
| 2021 | - | IMP |
| 2022 | - | IMP |
| 2023 | - | IMP |
| 2024 | - | IMP |
| 2025 | - | IMP |
| 2026 | - | IMP |
| 2027 | - | IMP |
| 2028 | - | IMP |
| 2029 | Project End | CLOSE |
Table 10 Summary
This table depicts the timeline of the MRB project. The table has five major columns identifying the project phases using the acronym for each phase. From left to right, these are: ID, OA, DEF, IMP, and CL. This table has three major rows: key activity (i.e., the top row), the year (i.e., the middle row), and the project phase (i.e., the bottom row). Read column by column to identify what year the activity and project phase took place.
| Traceability Tools | Key Traceability Documents | Traceability Information Inclusion | Traceability Information |
|---|---|---|---|
Traceability Matrix
IBM DOORS |
BCA | Included | Links Capability Deficiency to HLMRs |
| Included | Uses HLMRs to select options | ||
| SOR | Included | Links Capability Deficiency to HLMRs | |
| Included | Links HLRMs to tests and trials | ||
| Included | Links HLMRs to operational requirements | ||
| Included | Links tests and trials to IOC and FOC | ||
| IBM DOORS | Not Included | Does not include HLMRs | |
| Not Included | Does not link HLMRs to operational and/or technical requirements | ||
| Included | Links operational and technical requirements |
Table 11 Summary
This table identifies what traceability information was included in each traceability document for the MRB project. This table has four major columns and three major rows. The first column (i.e., left-most column) lists the traceability tools used by the project. The second column lists key traceability documents. The third column indicates whether traceability information was included in the document, partially reflected in the document, or not included in the document. The fourth column indicates the traceability information. The first row in column 2 (i.e., the top row) reads BCA. The second row in column 2 (i.e., middle row) reads SOR. The third row in column 2 (i.e., bottom row) reads IBM DOORS. Read across the rows to determine what traceability information was included in each document.
Case Study Summary
The MRB project seeks to replace the existing Halifax-class ships boat system, which is no longer serviceable and not designed for current operations. While the initial project close-out was August 2020, it has since been extended to June 2029.
The BCA links the HLMRs to the capability deficiency, and HLMRs were used to select the project option during OA. While there are no direct measures of HLMRs, they are linked to tests and trials of equipment in the SOR. The DOORS system does not include HLMRs, so the impact of changes to the operational and technical requirements on the HLMRs would not be flagged. As such, HLMRs do not appear to be directly measured or tracked by the project.
Annex J — Naval Large Tugs (NLT)
| Years | Event | Phase |
|---|---|---|
| 2007 | Project Start | ID; OA |
| 2008 | - | OA |
| 2009 | - | OA |
| 2010 | - | OA |
| 2011 | - | OA |
| 2012 | - | OA |
| 2013 | - | OA |
| 2014 | - | OA |
| 2015 | VCDS HLMR Directive 2015 | OA |
| 2016 | BCA Completed | OA |
| 2017 | - | OA; DEF |
| 2018 | SOR Completed | DEF |
| 2019 | Contract Awarded | DEF; IMP |
| 2020 | - | IMP |
| 2021 | - | IMP |
| 2022 | - | IMP |
| 2023 | - | IMP |
| 2024 | Project End | IMP; CLOSE |
Table 12 Summary
This table depicts the timeline of the NLT project. The table has five major columns identifying the project phases using the acronym for each phase. From left to right, these are: ID, OA, DEF, IMP, and CL. This table has three major rows: key activity (i.e., the top row), the year (i.e., the middle row), and the project phase (i.e., the bottom row). Read column by column to identify what year the activity and project phase took place.
| Traceability Tools | Key Traceability Documents | Traceability Information Inclusion | Traceability Information |
|---|---|---|---|
Traceability Matrix
IBM DOORS |
BCA | Included | Links Capability Deficiency to HLMRs |
| Included | Uses HLMRs to select options | ||
| SOR | Included | Links Capability Deficiency to HLMRs | |
| Not Included | Links HLRMs to performance measures | ||
| Included | Links HLMRs to operational requirements | ||
| Included | Performance measures linked to IOC and FOC | ||
| IBM DOORS | Not Included | Does not include HLMRs | |
| Not Included | Does not link HLMRs to operational and/or technical requirements | ||
| Included | Links operational and technical requirements |
Table 13 Summary
This table identifies what traceability information was included in each traceability document for the NLT project. This table has four major columns and three major rows. The first column (i.e., left-most column) lists the traceability tools used by the project. The second column lists key traceability documents. The third column indicates whether traceability information was included in the document, partially reflected in the document, or not included in the document. The fourth column indicates the traceability information. The first row in column 2 (i.e., the top row) reads BCA. The second row in column 2 (i.e., middle row) reads SOR. The third row in column 2 (i.e., bottom row) reads IBM DOORS. Read across the rows to determine what traceability information was included in each document.
Case Study Summary
The NLT project seeks to replace existing tug boats that did not have enough power to tow bigger and larger RCN ships. The new tugs will also have firefighting capabilities in order to replace the existing fire-class tug boats. The project will acquire four new tug boats. While the initial project close-out was May 2016, it has since been extended to October 2024.
HLMRs are linked to the capability deficiency, as well as operational and technical requirements in the BCA and SOR. The HLMRs do not have specific measures and are not linked to the performance measures. In the Mandatory Requirements table, the HLMRs are linked to operational requirements. The HLMRs are also not included in the tracing documents provided.
Annex K — Chemical Agent Sensor (CAS)
| Years | Event | Phase |
|---|---|---|
| 2003 | Project Start; BCA | ID; OA |
| 2004 | - | OA; DEF |
| 2005 | SOR Phase 1 | DEF; IMP |
| 2006 | SOR Phase 2 | IMP Phase 1 |
| 2007 | Contract Phase 1 | IMP Phase 1; IMP Phase 2 |
| 2008 | - | IMP Phase 1; IMP Phase 2 |
| 2009 | - | IMP Phase 1; IMP Phase 2 |
| 2010 | - | IMP Phase 1; IMP Phase 2 |
| 2011 | IOCC Phase 2; SOR Phase 3; FOCC Phase 2 | IMP Phase 1; IMP Phase 2 |
| 2012 | - | IMP Phase 1; IMP Phase 3 |
| 2013 | - | IMP Phase 1; IMP Phase 3 |
| 2014 | - | IMP Phase 1; IMP Phase 3 |
| 2015 | VCDS HLMR Directive 2015 | IMP Phase 1; IMP Phase 3 |
| 2016 | - | IMP Phase 1; IMP Phase 3 |
| 2017 | - | IMP Phase 1; IMP Phase 3 |
| 2018 | IOCC Phase 1 | IMP Phase 1; IMP Phase 3 |
| 2019 | FOCC Phase 1 | IMP Phase 1; IMP Phase 3 |
| 2020 | - | IMP Phase 1; IMP Phase 3 |
| 2021 | - | IMP Phase 1; IMP Phase 3 |
| 2022 | - | IMP Phase 1; IMP Phase 3 |
| 2023 | - | IMP Phase 1; IMP Phase 3 |
| 2024 | Project End | IMP Phase 1; CLOSE |
Table 14 Summary
This table depicts the timeline of the CAS project. The table has five major columns identifying the project phases using the acronym for each phase. From left to right, these are: ID, OA, DEF, IMP, and CL. This table has three major rows: key activity (i.e., the top row), the year (i.e., the middle row), and the project phase (i.e., the bottom row). Read column by column to identify what year the activity and project phase took place.
| Traceability Tools | Key Traceability Documents | Traceability Information Inclusion | Traceability Information |
|---|---|---|---|
Phase 1: Traceability and Gap Analyses and spreadsheet
Phase 2: Traceability and Gap Analysis
Phase 3: Access Database |
BCA | Included | Links Capability Deficiency to System Effectiveness Requirements |
| Not Included | Does not use System Effectiveness Requirements to select options | ||
| SOR | Included | Links Capability Deficiency to System Effectiveness Requirements | |
| Not Included | No direct measure of System Effectiveness Requirements | ||
| Reflected | System Effectiveness Requirements reflected in performance measures | ||
| Reflected | Links performance measures to IOC and FOC in phase 3 only | ||
| Traceability Documents | Included | Links System Effectiveness Requirements and technical requirements |
Table 15 Summary
This table identifies what traceability information was included in each traceability document for the CAS project. This table has four major columns and three major rows. The first column (i.e., left-most column) lists the traceability tools used by the project. The second column lists key traceability documents. The third column indicates whether traceability information was included in the document, partially reflected in the document, or not included in the document. The fourth column indicates the traceability information. The first row in column 2 (i.e., the top row) reads BCA. The second row in column 2 (i.e., middle row) reads SOR. The third row in column 2 (i.e., bottom row) reads Traceability Documents. Read across the rows to determine what traceability information was included in each document.
Case Study Summary
The CAS project seeks to provide the CAF with systems allowing for quicker detection and identification of a wider range of chemical warfare agents. These systems would also allow for sample collection. While the initial project close-out was September 2013, it has since been extended to July 2024.
CAS did not use HLMRs but sometimes used System Effectiveness Requirements in a similar way. System Effectiveness Requirements were linked to the capability deficiency in the BCA and SOR, but were not used to select options in the BCA. Performance measures required 100% of System Effectiveness Requirements to be completed. System Effectiveness Requirements and technical requirements were linked in tracing documents.
Annex L — Light Weight Towed Howitzer (LWTH)
| Years | Event | Phase |
|---|---|---|
| 2008 | Project Start; BCA Completed | ID; OA; DEF |
| 2009 | SOR Completed (Not signed); Contract Awarded | DEF; IMP |
| 2010 | - | IMP |
| 2011 | IOCC | IMP |
| 2012 | - | IMP |
| 2013 | - | IMP |
| 2014 | SOR Completed (signed) | IMP |
| 2015 | VCDS HLMR Directive 2015 | IMP |
| 2016 | - | IMP |
| 2017 | FOCC | IMP |
| 2018 | - | CLOSE |
| 2019 | - | CLOSE |
| 2020 | - | CLOSE |
| 2021 | - | CLOSE |
| 2022 | - | CLOSE |
| 2023 | Project End | CLOSE |
Table 16 Summary
This table depicts the timeline of the LWTH project. The table has five major columns identifying the project phases using the acronym for each phase. From left to right, these are: ID, OA, DEF, IMP, and CL. This table has three major rows: key activity (i.e., the top row), the year (i.e., the middle row), and the project phase (i.e., the bottom row). Read column by column to identify what year the activity and project phase took place.
| Traceability Tools | Key Traceability Documents | Traceability Information Inclusion | Traceability Information |
|---|---|---|---|
SOR Compliance Spreadsheet
Briefing Note |
BCA | Included | Links Capability Deficiency to HLMRs |
| Included | Uses HLMRs to select options | ||
| SOR | Included | Links Capability Deficiency to HLMRs | |
| Not Included | No direct measure of HLMRs | ||
| Reflected | Some HLMRs are reflected in performance measures | ||
| Reflected | HLMRs reflected in operational requirements | ||
| Reflected | HLMRS and performance measures reflected in IOC and FOC | ||
| SOR Compliancy Spreadsheet | Not Included | Does not include HLMRs | |
| Not Included | Does not link HLMRs to operational and/or technical requirements | ||
| Not Included | Does not link operational requirements to technical requirements (Project did not require technical requirements because it involved purchasing existing equipment) |
Table 17 Summary
This table identifies what traceability information was included in each traceability document for the LWTH project. This table has four major columns and three major rows. The first column (i.e., left-most column) lists the traceability tools used by the project. The second column lists key traceability documents. The third column indicates whether traceability information was included in the document, partially reflected in the document, or not included in the document. The fourth column indicates the traceability information. The first row in column 2 (i.e., the top row) reads BCA. The second row in column 2 (i.e., middle row) reads SOR. The third row in column 2 (i.e., bottom row) reads SOR Compliancy Spreadsheet. Read across the rows to determine what traceability information was included in each document.
Case Study Summary
The LWTH project seeks to provide the CAF with longer ranged, technology-assisted indirect fire support. While the initial project close-out was June 2013, it has since been extended to December 2023 in order to finalize the project’s finances. The equipment was delivered in 2018.
While the capability deficiency is linked to the HLMRs, the HLMRs were not clearly linked to operational requirements in the BCA or SOR. HLMRs are not directly measured but some HLMRs are mentioned in the performance measures. A compliancy spreadsheet was used for tracing but it only included operational requirements, not HLMRs or technical requirements. The LWTH was the only project to have reached CL. The interim project completion report does not include reference to HLMRs.
Annex M — Methodology and Limitations
Methodology
As the focus of the evaluation was the Equipment Acquisition Process, rather than a specific program, a process evaluation methodology was employed. The evaluation used the Causal Link Monitoring methodology, whereby the process required to achieve a desired result is mapped out and is then used as a baseline against which how the process is employed in practice is compared. In this context, the evaluation compared the acquisition process of six case study projects against the standard acquisition process, as described in various DND guidance documents.
Lines of Evidence
The evaluation used multiple lines of evidence to answer the evaluation questions:
- Document Review of Acquisition Process Guidance Documents and Case Study Documents
- Project Sponsors Questionnaire (N = 5)
- ADM(Mat) Project Teams Questionnaire (N = 6)
- Subject Matter Experts Group Discussions (N = 18)
- Key Informant Interviews (N = 19)
Limitations and Mitigation Strategy
As this evaluation used a case study approach and looked at each case at a specific point in time, the results of the case study projects are not a reflection of the situation for all equipment acquisition projects. These case study projects can only be taken as examples of how the policies and procedures can play out in practice. This limitation was mitigated by a review of the policy and interviews with other members of DND/CAF to determine whether the case study examples could play out for other projects due to limitations in policies, procedures and training.
Selection of Case Study Projects
The evaluation sought to identify cases that were developed under the current acquisition policies and procedures. These projects had to be far enough along in the acquisition process to have undergone changes to the project scope. The evaluation sought equal representation from the Maritime, Land and Aerospace Acquisition Programs.
Input from IRPDAO and CFD
Both IRPDA and CFD provided a list of potential projects that used HLMRs. Assistant Deputy Minister (Review Services) (ADM(RS)) removed some projects from this list as they were already being examined by other acquisition evaluations or audits currently underway
Input from ADM(Mat)
The reduced list was provided to ADM(Mat). Members noted that only two of all DND equipment projects with HLMRs had made it to Close Out (CL). One project was removed because evaluation and audit teams had unsuccessfully tried to engage this project. The other project at CL and two other projects on the list were included in the final list. ADM(Mat) advised the two additional projects were not at CL but were far enough along to provide sufficient information on capability evolution. The final three projects were not on the original list but were identified by ADM(Mat) as meeting the criteria for the evaluation.
During the document review phase of the project, it was discovered that one of these projects did not have HLMRs because it began prior to the VCDS HLMR Directive 2015. The project remained in the case study as it had high-level requirements that were used similarly to how HLMRs are used today.
Annex N — Traceability Summary
Traceability Tools
Each case study used different traceability documents during the implementation phases, and tracked information to various levels of details.
Figure 4 Summary
This figure illustrates the types of traceability tools that were used by projects. There are 5 pie slices in the chart. The area of the pie slices depicts the extent to which traceability tools were used across the six case study projects, using a percentage. Word documents reads 9%; excel spreadsheets reads 28%; PDFs, including briefing notes and transmittal sheets reads 27%; access database reads 9%; and IBM DOORS reads 27%.
All the projects used different traceability tools.
Figure 5 Summary
This figure depicts the number of traceability tools used by each project. There are 3 bars on the graph, each representing the number of traceability tools used by projects. The number of document types used runs along the x-axis and the number of projects runs along the y-axis. The first bar is ‘1 type of document’ and it reads 2, meaning 2 projects used one type of document for traceability the project change in scope. The second bar is ‘2 types of documents’ and it reads 3, meaning 3 projects used two types of documents for traceability the project change in scope. The third bar is ‘3 types of documents’ and it reads 1, meaning one project used three type of documents for tracing the project.
Four case study projects used at least two traceability tools.
Figure 6 Summary
This figure illustrates a timeline of four events as they relate to the improvement of traceability processes over time. The first event (i.e., the left-most item on the timeline) occurred in 2005. The second event occurred in 2009. The third event occurred in 2018. The fourth event occurred in 2020. Read across the timeline from left to right to uncover the events associated with traceability improvement.
Traceability did appear to have improved over time.
Annex O — Terminology
Page details
- Date modified: