Evaluation of Acquisition Project Management (Agile Acquisition, Innovation and GBA Plus)

December 2022

1258-3-057- ADM(RS)

Reviewed by ADM(RS) in accordance with the Access to Information Act.  Information UNCLASSIFIED.

Acronyms

Key Findings 

While a definition has been developed by Public Service and Procurement Canada (PSPC) and endorsed by the TBS, there is no common understanding and agreement on the definition of agility in relation to acquisition amongst the Defence Team. The Defence Team’s current guidance and documentation lack specific advice, methodologies or examples to promote agility and innovation in acquisition, and there is no agile-specific training offered. It is not clear how or when to apply agility and innovation in acquisition. Additional internal and external barriers to agility and innovation were found, such as a culture of risk aversion, staffing shortfalls, requirements around acquisition from both DND and external stakeholders, lessons learned are not timely and elements of agility and innovation are not tracked. Canada’s allies were found to have strategic commitments to agile acquisition, documentation and specific training. 

GBA Plus is needed to ensure procured equipment and materiel critical for DND/CAF operations match the needs of a diverse and gender-balanced workforce. Although tools and resources are available, GBA Plus is not fully implemented or tracked throughout all phases of the acquisition process and project lifecycles. Although conducting GBA Plus assessments is required by Strong, Secure, Engaged (SSE) and the Project Approval Directive (PAD), GBA Plus is often only implemented in the early phases of the acquisition process and not present in other phases.

Overall Conclusion

The evaluation concluded that a flexible and agile, rather than a “one size fits all”, approach to acquisition is needed, and that accountability and capacity for GBA Plus should be integrated into all phases of Acquisition Project Management.

Back to Table of Contents

Evaluation Context

Evaluation Overview

Defence acquisition is vital to the operational effectiveness of the CAF. SSE sets enhancing the Defence acquisition process to be more streamlined, innovative and flexible as a strategic priority. To ensure that the CAF is ready to fulfill its critical mandate in this new era of Defence and Security, a transformation of the acquisition process is necessary. 

Evaluation Objectives and Importance

In accordance with the Treasury Board (TB) Policy and Directive on Results, and with the endorsement of the PMEC, this evaluation examined questions related to agile and innovative acquisition practices and approaches as well as the application of GBA Plus in the acquisition project management. 

Audit of Defence Project Management 

The evaluation worked in conjunction with the audit of Defence Program Management. The objective of the audit is to determine if key internal controls are in place and functioning as intended, to support the successful delivery of major capital projects. The evaluation team worked closely with its Audit partners during the scoping, planning and conduct phases to ensure alignment of findings and cohesion amongst recommendations. The audit findings are presented under separate cover.

Program Spending (FY 20/21)

Evaluation Stakeholders

The following L1 responsibilities come from Defence Administrative Orders and Directives 3000-0 and the relevant Senior Designated Official letters.

Assistant Deputy Minister (Materiel) ((ADM(Mat))

• ADM(Mat) is responsible to ensure that the procurement of goods and services obtains the necessary assets and services that support the delivery of the Defence Services Program to Canadians. Additionally, ADM(Mat) is responsible for the management of Defence materiel. 

Assistant Deputy Minister (Information Management) ((ADM(IM)) 

• ADM(IM) is responsible to ensure that IM/Information Technology (IT) projects and programs are effectively planned, implemented, monitored and controlled, and closed. 

Vice Chief of the Defence Staff (VCDS)  

• VCDS is responsible to ensure that projects and programmes are effectively planned, implemented, monitored and controlled, and closed. Additionally, VCDS is responsible for the PAD. 

Director Gender Equality and Intersectional Analysis (DGEIA)

• DGEIA is responsible to support the Defence Team’s implementation of GBA Plus when developing policies, directives and programs, and when planning and carrying out operations. 

Other Organizations Consulted 

• Assistant Deputy Minister (Finance) ((ADM(Fin))
• Canadian Army  
• Corporate Secretariat  
• Defence Women’s Advisory Organization
• Public Service and Procurement Canada (PSPC)
• Treasury Board Secretariat (TBS)

Organizations Not Consulted 

The following organizations, while playing a role in DND/CAF acquisition, were excluded from the scope of this evaluation:

• Assistant Deputy Minister (Infrastructure and Environment) ((ADM(IE))
• Defence Research and Development Canada^{Footnote 1} 

Back to Table of Contents

Evaluation Themes

Back to Table of Contents

Agile and Innovative Acquisition 

Why it’s Important

• The performance of Defence acquisition has been a major concern for decades, as it is often portrayed in the media as slow, obsolete and costly.
• SSE commits to “implementing flexible new procurement mechanisms that allow Defence to develop and test ideas and the ability to follow through on the most promising ones with procurement.”
• The Independent Review Panel for Defence Acquisition noted several projects struggled with the lack of agility in the acquisition process. The panel supports further exploration and experimentation in more innovative and agile acquisition approaches.^{Footnote 2} 

What We Found

• The evaluation found that there are no defining characteristics of Agile acquisition in Defence within any of the documentation reviewed (e.g., the PAD and documentation of the three case study projects) and no training specific to agile acquisition offered.
• While PSPC’s definition of Agile often refers only to procurement, they have expanded their guidance towards using Agile throughout the acquisition process and to a broader set of projects beyond just IM/IT.
• Agile acquisition guidance and training is available in the United States and provided by their Defence Acquisition University with courses such as the Introduction to Agile Software Acquisition offered. The United Kingdom and Australia make strategic level commitments to agile acquisition (see Annex D for more international context).

Note: This report adopts the definitions of procurement and acquisition as established in DND’s Procurement Administration Manual (2022): “The acquisition process includes all activities related to the introduction of a new equipment or system (for example: procurement, contracting, project management, engineering, etc.)… Procurement is a sub-process of acquisition.” This contrasts with the TB Policy on the Planning and Management of Investments, which defines procurement as all activities related to acquiring goods, services or construction, from planning to completion, and for which acquisition is considered only one stage in the broader procurement life-cycle.

Back to Table of Contents

Why it’s Important

• A balanced approach to risk and flexibility is necessary when applying the principles of agility and innovation. 
• Limited agility and innovation within acquisition poses risks to operational capabilities. 
• Consistent and timely guidance and documentation is required for the successful implementation of agile and innovative acquisition practices.  

What We Found

• Limited evidence of agility was found in case studies and documents reviewed (e.g., ISSP did include elements of an iterative process, some team diversity and engagement with industry). This degree of agility was not found in the documentation reviewed for the other case study projects.
• Interviews and survey results also indicate limited agility and innovation in the acquisition process. 
• The following five key barriers were identified as root causes for not widely applying agility and innovation: culture and risk aversion, lack of performance information and lessons learned, human resources and training, governance and oversight, and guidance and documentation.

E. Guidance and Documentation 

• Acquisition guidance and documentation do not include sufficient advice or methodologies to enable agility and innovation. Defence guidance still encourages procurement officials towards a more traditional/waterfall approach, and the PAD does not provide a definition or specific examples to encourage the application of the principles of agility. 
• The PAD was updated in 2019 and the process was reduced from 100+ steps down to 50-60. While the PAD mentions four process paths, they are differentiated by cost and PCRA rating: Process A (Lower Risk: $30K-$10M), Process B (Lower Risk: $10M-$100M), Process C (Medium Risk: $100M+), and Process D (Higher Risk: PCRA 4). (See Annex F for details on the process paths). 
• Paths could be based on the type of acquisition instead of by cost or PCRA rating as is currently done. For example, the US Adaptive Acquisition Framework outlines paths based on acquisition type (e.g., Urgent Capability Acquisition, Major Capability Acquisition and Software Acquisition) and offers a breakdown of the different phases within each path.^{Footnote 4} 
• Senior management at DND are considering levels of funding authority and “right size” of projects, which could enable more agility and innovation. Further study or experimentation would be required to determine viability of options. 

Back to Table of Contents

Gender-based Analysis Plus

Why it’s Important

• GBA Plus is a priority of the Government of Canada. It is an important element of our international cooperation and interoperability with key partners and allies.
• The CAF is attempting to employ 25% women by 2026, as set out in SSE.
• Procured equipment and materiel need to match the needs of a diverse and gender-balanced CAF and workforce.
• Conducting GBA Plus assessments contributes to a more diverse, inclusive and operationally effective CAF and Defence Team.
• GBA Plus is part of our international commitments, such as United Nations Security Council Resolution (UNSCR) 1325 Women, Peace and Security, which states that women need to be equal participants in peacekeeping and conflict resolution.

What We Found

• Documentation, interviews and survey results indicate limited GBA Plus implementation beyond the corporate submission and the initial Identification phase of acquisition. GBA Plus should not only be in early phases, but when assessed to be relevant, should be an ongoing and iterative process applied to inform every aspect of the project, from Identification to Closeout.
• No follow-up assessments or KPIs exist to track the level of GBA Plus implementation in project lifecycles. 
• Lessons Learned do not include mandatory fields for GBA Plus, and this area is often not captured.

Back to Table of Contents

Why it’s Important

• As per the PAD, GBA Plus should be integrated throughout the acquisition process.
• Additional project and/or acquisition-specific GBA Plus expertise and resources will increase the knowledge of GBA Plus within teams and partners and increase the implementation of GBA Plus.

What We Found

• General mandatory training for GBA Plus has been offered by Women and Gender Equality Canada, and non-mandatory tailored training has been offered in the past by DGEIA. 
• Only a minority (16%) of survey respondents felt that the current mandatory GBA Plus training was effective or very effective for applying GBA Plus in Defence acquisition; this was corroborated through interviews. 65% of survey respondents reported not taking any additional courses related to GBA Plus.
• In addition to the tailored GBA Plus training, DGEIA also provides guidance and support for acquisition project teams. For example, DGEIA’s work in collaboration with the Human Systems Integration team resulted in changes to an RFP that challenged companies to re-think anthropometric measurements in consideration of women.
• In all three of the case studies, no evidence of a GBA Plus expertise was identified among the project teams. As outlined in the PAD: “Project Teams should include a member with expertise in Gender-based Analysis Plus (GBA Plus).”

Back to Table of Contents

Promising Practices

Back to Table of Contents

Overall Conclusion

While there is a growing interest within DND/CAF Project Management to explore more flexible, agile and innovative approaches to acquisition, limitations and barriers exist. For example, the current culture is risk-averse. Project Management Teams focus on preventing failure and minimizing risk, which keeps the acquisition system rigid. Furthermore, there is a lack of tailored resources and specific training program on the use of agile and innovative methodologies in procurement processes. Some of the additional barriers to agile and innovative acquisition are attributed to the multiple layers of financial requirements and oversight, lack of performance metrics for agile and innovation, and the outdated and inconsistent existence and use of Lessons Learned.  

While the Project Approval Process includes steps for incorporating GBA Plus in all phases of the acquisition process, the evaluation found that GBA Plus is only implemented in the early phases of the acquisition, and there is limited use of tools and resources tailored for conducting Defence-specific GBA Plus analysis. 

The evaluation concluded that a flexible and agile, rather than a “one size fits all”, approach to acquisition is needed, and that accountability and capacity for GBA Plus should be integrated into all phases of Acquisition Project Management

Back to Table of Contents

Annex A – Management Action Plan

Management Action

This Management Action Plan (MAP) will require interim milestones in order to fully address the recommendation. The VCDS, through the Policy Suite DG Steering Committee (PSDGSC), will coordinate with key stakeholders to:

• Review PSPC’s definition of ‘agility’ as well as the findings from ADM(RS)’ Evaluation of the Innovation for Defence Excellence and Security (IDEaS) Program to facilitate a discussion and resulting in the Departmental definition of innovative acquisition. 
  • Target - December 2023 – coordination and liaison is required across key stakeholders at PSPC, TBS, ADM(IM), ADM(Mat), ADM(IE), ADM(Fin), C Prog and Chief of Force Development (CFD) and followed by formal approval by the Programme Management Board through the PSDGSC
• Update applicable Departmental policy documents with definitions of ‘agility’ and ‘innovative acquisition’. 
  
  • Target - September 2024 – coordination and liaison is required across key stakeholders at PSPC, TBS, ADM(IM), ADM(Mat), ADM(IE), ADM(Fin), C Prog and CFD and followed by formal approval through the PSDGSC as a follow-on to the above bullet pertaining to the definition of agility.
• Identify and minimize internal and external barriers to agility and innovation by leveraging the Departmental Organizational Project Management Capacity Assessment authorities through recently approved increases in Delegations of Authority.
  
  • Target - September 2025 – coordination and liaison through the PSDGSC is required across key stakeholders internal and external to DND (PSPC, TBS, MNDO, DMO, CMP, ADM(HR-Civ) ADM(IM), ADM(Mat), ADM(IE), ADM(Fin), C Prog and CFD)

Closure Statement: This MAP will be considered closed once each of the three interim milestones have been completed.

Target Date: September 2025

OPIs: VCDS (PSDGSC)
OCIs: ADM(Mat), ADM(IM), ADM(IE)

Management Action

This MAP has a specific interdependency with the successful implementation of MAP #1. The VCDS, through the PSDGSC, and in close collaboration with OCIs, will ensure that applicable Departmental policy documents include the following:

• There is a requirement to identify KPIs through an agreed upon definition of ‘Agile’ and ‘Innovative Acquisition’ and implementing a monitoring and reporting mechanism/tool.
  • Target – June 2024 - Mechanisms for ensuring this task is completed will include the establishment of an agreed upon definition of ‘Agile’ and ‘Innovative Acquisition’ and the establishment of a monitoring and reporting tool. This requires consultation with CFD, C Prog, ADM(Mat), ADM(IM), ADM(IE) and other DND and OGD stakeholders as needed.
• There is a requirement to provide an update of lessons learned and KPIs by the Project Sponsor and/or Implementer during their briefing to the Senior Review Board (SRB) - minimum annually.
  • Target – September 2025 – Mechanisms for ensuring this task is completed will need to include amendments to the PAD and development of a template to gather and report on KPIs. In addition, this MAP has interdependencies with the successful implementation of MAP #1. This requires consultation with CFD, C Prog, ADM(Mat) [as the DND lessons learned authority], and other DND and OGD stakeholders as needed.

Closure Statement: This MAP will be considered closed once Departmental policy documents have been updated and communicated to all L1s and the respective monitoring and reporting mechanisms are implemented.

Target date: September 2025

OPIs: VCDS (PSDGSC)
OCIs: ADM(Mat), ADM(IM), ADM(IE)

Management Action 

This MAP requires collaboration across multiple stakeholders, including Chief Professional Conduct and Culture (CPCC). The VCDS, through the PSDGSC, will coordinate with the OCIs to ensure that applicable Departmental policy document(s) include the following:

• A requirement to provide an update on GBA Plus assessments including of lessons learned made by the Project Sponsor/Implementer during their briefing to the SRB - minimum annually. 
  • Target – June 2023 – Changes to the PAD and the Procurement Administration Manual (PAM) with the development of tools and tracking methods to support this requirement will require consultation with ADM(Mat), CFD, C Prog, CPCC (the DND GBA Plus authority), and other stakeholders as needed.

Closure Statement: This MAP will be considered closed once Departmental policy document(s) have been updated and communicated to all L1s.

Target date: June 2023

OPIs: VCDS (PSDGSC)
OCIs: ADM(Mat), ADM(IM), ADM(IE), CPCC

Management Action 

This MAP requires collaboration across multiple stakeholders, including CPCC. The VCDS, through the PSDGSC, will coordinate with the OCIs to ensure Project Sponsors/Implementers include the following:

• Identify all critical requirements, including knowledge gaps, needed to increase capability for GBA Plus within project management. 
  
  • Target December 2022 - This requirement will require consultation with CFD, C Prog, CPCC (the DND GBA Plus authority), ADM(Mat), ADM(IE) and other stakeholders as needed.
• Resource (training and people), identify and establish KPIs and enable mechanisms (process) to increase capability and visibility for GBA Plus within project management.
  • Target - September 2023 - This requirement will require consultation with CFD, C Prog, CPCC (the DND GBA Plus authority), ADM(Mat), ADM(IE) and other stakeholders as needed.
• Track and monitor GBA Plus lessons learned through annual briefing to SRB. 
  • Target - September 2024 - Changes to the PAD and the PAM with the development of tools and tracking methods to support this requirement will require consultation with ADM(Mat), CFD, C Prog, CPCC (the DND GBA Plus authority), ADM(IM), ADM(IE) and other stakeholders as needed.

Closure Statement: This MAP will be considered closed when the training and mechanisms have been implemented within project sponsors/implementers.

Target date: September 2024

OPIs: VCDS (PSDGSC)
OCIs: ADM(Mat), ADM(IM), ADM(IE), CPCC

Back to Table of Contents

Annex B – Methodology

The findings and recommendations of this report were informed by multiple lines of evidence collected throughout the conduct phase of the project. These lines of evidence were triangulated with each other and verified with program officials to ensure their validity. The research methodology used in the scoping and conduct of the project were as follows:

Document review

A preliminary review of federal, departmental and program policies, directives, plans, reports and other relevant documents was conducted to provide an in-depth understanding of the evaluation context. During the conduct phase, a detailed document review was undertaken with a broad approach given the unique nature of the evaluation. The evaluation team reviewed 282 documents, and 137 general and DND-specific documents were disaggregated by the following seven categories: internal defence, external domestic, previous evaluations and audits, literature review on agile acquisition, innovation,GBA Plus, and news report and media. The review also included 84 case-study specific and 60 international documents. 

Interviews

A total of 32 semi-structured interviews were conducted for this evaluation. This total represents a mix of informal/scoping interviews as well as more methodological interviews that followed a semi-structured format. In-depth key informant interviews with primary and secondary stakeholders of the Land, Air and Sea Program were conducted to gain insight from key leadership/senior management on the understanding and implementation of agile and innovative acquisition and also on the implementation of GBA Plus within the acquisition project management. This served as an important source of qualitative information and provided informed opinion and observations on evaluation questions and relevant indicators. Interviews were conducted using MS Teams or by telephone.   

Surveys

A survey was conducted to gain diverse insight from individuals who are involved in the Defence equipment acquisition process (e.g., technical expertise, project managers, procurement officers, etc.). The survey contained 19 questions and collected information and perspectives about Defence Acquisition Project Management and its focus on agile, innovative acquisition as well as consideration of GBA Plus throughout the project management cycle. The survey was administered across projects in ADM(Mat) or various L2 organizations within ADM(Mat), specific ally: DGLEPM, DGMEPM, DGAEPM, Air and Land DGMPD, Sea DGMPD and COS (Mat). Out of the 345 surveys returned, 235 were complete and 110 partially completed. 

Case studies

The evaluation employed a case study methodology by performing an in-depth examination of three example projects (two from ADM(Mat) and one from ADM(IM)) that met the following criteria: (1) Under Major Projects Division (MPD) and EPM as L2 Implementers; (2) Cover the three environments (Air, Land and Sea); (3) Are in level 3 and 4 Project Complexity and Risk Assessment (PCRA); (4) Are not under assessment by other Evaluation Teams or Audits; and (5) Have not been recently assessed. Document reviews and specific informal interviews were conducted for each of the three projects. 

International benchmarking

The evaluation consulted 60 documents from three other countries to assess how Canada performs on an international scale and to identify successful models to inform recommendations. International benchmarking provided an additional measure of how other countries are managing acquisition projects and using innovative and agile acquisition practices as well as taking GBA Plus into consideration. The international scan/benchmarking included documents from the United States, Australia, United Kingdom and NATO.

Back to Table of Contents

Annex C – Case Studies

Back to Table of Contents

Annex D – International Context

Back to Table of Contents

Annex E – Examples Of Major Projects

DGMPD (Air & Land) manages the acquisition of large-scale air and army equipment. 

DGMPD (Air & Land) Projects:

• FWSAR: Fixed Wing Search and Rescue Project
• FFCP: Future Fighter Capability Project
• MSVS: Medium Support Vehicles System Project
• LVM: Logistics Vehicle Modernization 
• STTC: Strategic Tanker Transport Capability 
• ACSV: Armoured Combat Support Vehicle
• RPAS: Remotely Piloted Aircraft System 

DGMPD (Sea) is responsible for the acquisition of major new sea equipment. 

DGMPD (Sea) Projects:

• AOPS: Arctic Offshore Patrol Ships 
• JSS: Joint Support Ship Project
• CSC: Canadian Surface Combatant

Back to Table of Contents

Annex F – Process Paths

Back to Table of Contents