Independent Review Panel for Defence Acquisition – Progress Report 2021-2022

Foreword From the Chair

The Independent Review Panel for Defence Acquisition (IRPDA) presents its fifth progress report, covering the 2021 and 2022 calendar years.

Over the reporting period the Panel witnessed growing pressure on the Defence programme. More urgent capability priorities emerged in response to the changing global security landscape, while personnel shortages and increasing costs put significant strain on the Department’s capacity and resources to deliver. In this context, the Panel adapted quickly to the post-pandemic working environment and responded to a steadily increasing workload through 2021 and 2022.

This report includes a summary of the Panel’s activities during this period, strategic observations based on the Panel’s work, and a statement of the Panel’s priorities and commitments going forward. It includes a complete list of the projects that the Panel has reviewed (Annex B), feedback received from IRPDA stakeholders (Annex C and Annex D), and an illustration of the Panel’s standard review methodology (Annex E). In preparation for this report, for the first time, the Panel hired an external party to undertake interviews with senior public service stakeholders to gain feedback on the Panel’s work and to gain a deeper sense of the strategic challenges facing the defence community (Annex D).

The Panel bid farewell to two Panel members in 2022 – Dr. Philippe Lagassé and Ms. Margaret Purdy. These individuals played a critical role in the work of the Panel over the years, and the Panel expresses its deepest gratitude for their invaluable contributions in support of its mandate. The Panel also welcomed two new members, Dr. Elinor Sloan and Ms. Linda Lizotte-MacPherson, in November 2022, who each bring highly relevant expertise and experience to the Panel’s work.

The Panel has now reviewed over 80 major defence procurement projects from across the Department of National Defence and also the Canadian Coast Guard. Recent feedback suggests that the Panel’s work remains as relevant now as it was when the Panel first began in 2015, if not more so. Going forward, the Panel will remain mindful of the significant challenges facing the defence procurement community – particularly in the areas of speed, personnel capacity, and agility – and will work to maximize the effectiveness of its review mandate while concurrently supporting project teams as they work through this challenging space.

On behalf of the Panel, I would like to extend my sincere thanks to the Department for its ongoing support to enable the Panel to successfully achieve its mandate. The IRPDA is privileged to support the Minister and the Defence Team in the defence procurement process and ultimately in delivering the equipment and infrastructure that is needed by the Canadian Armed Forces.

Mr. Larry Murray
Chair of the Independent Review Panel for Defence Acquisition

Back to top

Introduction

The IRPDA was created in 2015 as one of the core elements of the Defence Procurement Strategy. At that time, military procurement faced several challenges, including a perception that operational requirements were insufficiently justified and crafted in such a complex manner that they complicated the approval process. As a result, the Panel was established as an independent, third-party review body, reporting directly to the Minister of National Defence, to validate the requirements of major military procurement projects and ensure that they are appropriately stated prior to the expenditure of significant funds.

In accordance with the Panel’s Terms of Reference, every defence equipment and infrastructure project valued at over $100M is carefully reviewed by the Panel, in consultation with relevant Departmental officials and in line with the Panel’s well-established and disciplined review methodology (see Annex E). From time to time, the Panel may also review additional projects that have been referred to it by the Minister or Deputy Minister of National Defence, or when Treasury Board approval is required.

The Panel reviews projects at two points early in the defence procurement process, at the beginning and at the end of the Options Analysis phase, before the Department of National Defence/Canadian Armed Forces (DND/CAF) seeks approval from the Minister or Treasury Board for expenditure authority (Figure 1):

IRP1 engagement

The first Panel engagement (IRP1) focuses primarily on the strategic context and policy alignment of the project, the capability gap the project intends to address, and the High Level Mandatory Requirements (HLMRs). In this initial engagement, the Panel reviews the Strategic Context Document after it has been endorsed by senior DND/CAF officials at the Defence Capabilities Board (DCB). This first engagement helps the Panel identify any potential issues with project requirements before the detailed work in Options Analysis begins. Following the first engagement, detailed feedback is provided to the Department, both to the Vice Chief of the Defence Staff (VCDS) and other senior-level stakeholders, as well as directly to the project team.

IRP2 engagement

The second Panel engagement (IRP2) focuses primarily on the results of Options Analysis, and that the more detailed and technical project requirements are well-aligned with the HLMRs. As part of these engagements, the Panel reviews the Business Case Analysis and the Preliminary Statement of Operational Requirements, again following endorsement by senior DND/CAF officials. On an as-needed basis, additional feedback may be provided to the Department after IRP2. This additional feedback is distinct from, yet complementary to, the Panel’s final advice to the Minister.

Supplementary engagements

When necessary, the Panel will pursue supplementary engagements to ensure that any issues with the projects are addressed as early as possible, and that the Panel has sufficient information to produce credible, evidence-based advice for the Minister. For complex projects, the Panel may also request a ‘scene setter’ briefing at the outset of the review, to ensure that members have sufficient background information to revieew projects at IRP1.

Figure 1: Panel Review Process

Text version
  • Identification
    • Sponsor establishes High Level Mandatory Requirements and project scope
    • Defence Capabilities Board endorses Strategic Context Document
  • First Panel Engagement (IRP1)
    • Panel reviews High Level Mandatory Requirements and Strategic Context Document
    • Post-IRP1: Panel sends feedback to VCDS and sponsor
  • Option Analysis
    • Sponsor performs Options Analysis and develops Business Case Analysis
    • Defence Capabilities Board endorses Business Case Analysis
    • DND/CAF endorses Preliminary Statement of Operational Requirements
  • Second Panel Engagement (IRP2)
    • Panel reviews Business Case Analysis and Preliminary Statement of Operational Requirements
    • Post-IRP2: Panel submits advice to Minister
  • Definition
  • Implementation
  • Close-out

Upon completion of the review process, the Panel submits advice to the Minister, through the Deputy Minister, with a copy to the Chief of the Defence Staff and other relevant senior DND/CAF officials. The advice is submitted before the Department seeks approval from the Minister or Treasury Board to enter into Definition and to expend significant capital investment funds. The specific timeline for each project is unique. The Panel’s advice to the Minister is treated as Cabinet confidence.

In accordance with the Panel’s Terms of Reference, the Panel’s advice is focused on providing an independent evaluation of whether the CAF is facing a credible capability need, whether the project requirements are clearly and appropriately stated, and whether there is a strong rationale for the recommended option.

In addition, the Panel may highlight any further considerations that might be useful to the Minister or Deputy Minister to help them better understand the stated requirements for the project and to inform critical project decisions going forward. For example, the Panel may highlight potential cost or schedule risks, unique integration challenges, or potential benefits for Canadian industry. Panel advice can be shared with central agencies with the approval of the Minister.

Back to top

Year in Review: Activities in 2021 and 2022

“The Panel reviewed over $20B worth of major equipment and infrastructure projects in 2021 and 2022Footnote 1.”

The Panel reviewed over $20B worth of major equipment and infrastructure projects in 2021 and 2022Footnote 1, and remained responsive to the DND/CAF project readiness schedule throughout the reporting period. The project review rhythm was initially slower at the outset of 2021 than in previous years, likely due to the impact of working restrictions associated with the COVID-19 pandemic, project funding constraints, and shifting capability priorities, among other complex dynamics. However, the Panel has responded to a steadily increasing workload since 2021, and current trends suggest that the next reporting period will show even higher project review throughput, potentially reaching record levels.

During this period, the Panel continued to focus on projects associated with the ongoing implementation of Strong, Secure, Engaged, while also responding to urgent CAF capability requirements, including those related to ongoing operations.The Panel continued to undertake special reviews for the Canadian Coast Guard (CCG) in support of their fleet renewal efforts. Building on the Panel’s previous review work for the CCG, the Panel examined and provided advice on the Multi-Purpose Vessel Project and the Program Icebreaker Project.

Overall, in 2021 and 2022, the Panel reviewed 28 major capital procurement projectsFootnote 2 and conducted 35 distinct engagements related to these projects. As well, the Panel submitted 13 letters of advice to the Minister of National Defence. Advice on the remainder of the projects reviewed in 2021 and 2022 will be completed following their respective IRP2 engagements (as well as supplemental engagements, if required). The Panel also submitted nine feedback letters to the Vice Chief of the Defence Staff (VCDS) after IRP1 and IRP2 engagements.

“The Panel has responded to a steadily increasing workload since 2021, and current trends suggest that the next reporting period will show even higher project review throughput, potentially reaching record levels.”

Figure 2 presents the Panel’s 2021 and 2022 review activities compared with previous years, including those associated with the CCG. The Panel’s review agenda continues to be scheduled in close consultation with DND/CAF, to ensure projects are reviewed in a timely manner.

Figure 2 – Panel Activities and Output Associated with DND/CAF Project Reviews since 2015

Text version
- 2015 2016 2017 2018 2019 2020 2021 2022
Project Engagements 11 16 18 30 32 20 18 19
Scene Setter Briefings 6 3 1 7 - 3 4 5
Advice to the MND 2 6 8 8 11 12 4 9
Feedback to the Department - - - - 2 7 3 7

While not captured in the formal engagement numbers in Figure 2, the Panel also received on average one informational briefing or senior stakeholder engagement per month, on topics that ranged from greening procurement to NORAD modernization. These sessions were vital in ensuring the Panel maintained overall strategic awareness, and that the Panel’s focus and approach remained aligned with evolving Departmental priorities.

The Panel also renewed its outreach efforts with DND/CAF and other stakeholders, to which the Panel committed in the last progress report. In total, the Panel and its supporting office conducted over 20 outreach activities, including with senior DND/CAF leadership, industry stakeholders, and the broader defence and security community

Figure 3 – Projects Reviewed by Cost Estimate for 2021 and 2022

Text version
Range Number
Under $20M 1
$50M - $99M 1
$100M - $249M 8
$250M - $499M 7
$500M - $1B 5
$1B - $4.99B 7

Acknowledging that project cost estimates and funding levels may fluctuate over time, Figure 3 illustrates that most projects reviewed by the Panel in 2021 and 2022 were estimated to be in the $100-$249M range, based on values identified in the Defence Capabilities Blueprint.

Of note, thirteen of the projects reviewed by the Panel in 2021 and 2022 did not yet have any approved funding or were anticipating funding shortfalls, in many cases significant shortfalls.

In 2021 and 2022, the highest number of projects reviewed by the Panel were sponsored by the Canadian Army, including projects aimed at delivering new night vision systems and intelligence, surveillance and reconnaissance equipment. The Panel also continued to review major procurement projects from across the Department – from aircraft upgrades for the Royal Canadian Air Force to a new enterprise resource planning system for ADM (Data, Innovation and Analytics). The variety of project sponsors is reflected below in Figure 4, and illustrates the breadth of the Panel’s project review experience in 2021 and 2022.

Figure 4 – Projects Reviewed by Sponsor in 2021 and 2022Footnote 3

Text version
Range Number
CA 11
RCAF 6
RCN 3
ADM(IM) 2
CMP 2
ADM(DIA) 1
ADM(IE) 1

For the first time, the Panel undertook an in-depth study to better understand the development, use and impact of HLMRs, acknowledging the central role of HLMRs in articulating the capability requirements that a project is meant to deliver. The study leveraged the Panel’s extensive project review experience since 2015 and culminated in Panel recommendations to the Department in late 2022 on how to strengthen the approach to HLMRs.

The Panel’s supporting office also undertook a study with Departmental data analytics experts to explore the use of text analytics on Panel documentation. Given the Panel’s growing archive of analysis and advice since 2015, the Panel continues to explore ways to leverage its historical records to identify cross-cutting observations and to assess project evolution and Panel impact over time.

Financial information on the Panel and its supporting office (IRPDAO) is provided at Annex F. Since its inception, the Panel has not pursued significant growth in staff or budget, and has continued to manage fluctuations in Panel workload within existing resources.

Back to top

Panel Performance and Impact

The Panel is an independent third-party review body that provides advice directly to the Minister of National Defence. Nevertheless, Panel members recognize the importance of having productive relationships with DND/CAF stakeholders in carrying out their mandate. With that in mind, the Panel conducts periodic surveys to obtain valuable feedback from DND/CAF stakeholders.

In early 2023, the Panel conducted an online survey for DND/CAF project stakeholders, at the Director General level and below. While the response rate was lower than previous years, the 2023 online survey received a very positive response, with a notable increase in positivity as compared to the 2020 survey. A large majority of respondents indicated that the Panel positively impacted project evolution, and that Panel feedback was clear, useful, and incorporated into subsequent documentation. Details on the 2023 survey results can be found at Annex C.

For the first time, the Panel also engaged an external party to undertake a series of interviews with senior leaders at the Assistant Deputy Minister level and above, in DND/CAF, Public Services and Procurement Canada, and the Canadian Coast Guard. The interview feedback indicated that the Panel is widely perceived to be responsive and credible, and to provide high value to the Department and beyond. Details on the 2023 interview results can be found at Annex D.

Some stakeholders did query whether the Panel’s review threshold of $100M requires updating or refinement. Based on the current volume of projects, the Panel confirms that it still has the capacity to conduct a detailed review of all projects above this value. The Panel can also review projects below this threshold, if directed by the Minister or Deputy Minister, to address smaller projects of unique risk or complexity.

The Panel continues to use stakeholder input to periodically identify areas for refinement in its review process and methodology. For example, following the 2020 survey and since the last progress report, the Panel instituted a new IRP2 feedback process, that is separate and distinct from Ministerial advice, with the aim of offering additional supporting observations to officials as they prepare project submissions. The Panel also invited additional Departmental stakeholders to project review engagements on a more systematic basis, to foster a more comprehensive view of project interdependencies.

“In the interviews of senior leaders, the Panel was widely perceived to be responsive and credible, and to provide high value to the Department and beyond.”

Looking ahead, the Panel is committed to showcasing more positive examples of project development (including in the area of HLMRs), wherever possible. The Panel will also continue to refine its line of questioning in project reviews, including regarding the PRICIEGFootnote 4 analysis, to further understand the enterprise-wide considerations and/or dependencies associated with project requirements. The Panel will also explore ways to support a more efficient project progression between IRP1 and IRP2.

Back to top

Strategic Observations

Drawing on the experiences of this reporting period, the Panel has identified a series of strategic observations for DND/CAF’s consideration. These observations build on those identified in previous progress reports, while also reflecting the specific projects reviewed by the Panel in 2021 and 2022. The observations in this report also leverage insights that were provided by recent senior leader interviews. Through those interviews, senior leaders indicated that projects are persistently challenged by the lack of speed, human capacity, and agility in defence procurement.

Overall, the Panel continues to observe the significant value of strong leadership in the early phases of project development, which can set clear project direction and enable an efficient and targeted procurement effort.

Below, the Panel offers more specific observations in each of the Panel’s standard areas of focus.

Strategic Context

In 2021 and 2022, the Panel observed a heightened imperative and urgency to deliver CAF capability, at the same time that Departmental capacity and resources to execute have been – and remain – significantly constrained. In this context, most projects reviewed by the Panel continued to have a valid policy foundation. However, the changing security environment and the influx of new priorities and urgent equirements demand a policy update to provide a renewed and unified vision of the defence programme. The Panel is encouraged that the Department is updating its defence policy, Strong, Secure, Engaged.

Concurrent with the growing pressure on the Defence programme, the Panel has also observed that projects are increasingly complex and interdependent. Over the reporting period, the Panel reviewed several projects with enterprise-wide implications. In the Panel’s experience, these complex projects, such as those building the CAF digital architecture, will demand enhanced Departmental coordination, governance and leadership.

Capability Gap

In the Panel’s opinion, most projects continue to present well-articulated and compelling capability gaps. However, in the context of resource and capacity constraints, the Panel would encourage greater fidelity regarding the scale and scope of the capability need, to help inform project trade-offs. In particular, the Panel recommends that projects better articulate the quantitative aspects of the capability gap (i.e. the number of required units, or fleet size), to ensure clarity regarding the minimum requirement and inform capability ladder development.

The Panel also observed the significance of infrastructure, personnel, and information technology to project success. The Panel believes these components need to be more consistently considered in a project’s narrative and scope from the outset, thereby informing a more comprehensive assessment of resource requirements for all project stakeholders early in project development.

Requirements

Over the reporting period, HLMRs remained a regular focus for the Panel during project reviews. The Panel continued to recommend a more consistent application of the internal guidance on HLMRs, encouraging greater specificity, clarity and comprehensiveness, as well as a clear sense of sufficiency. The Panel also emphasized that well-defined HLMRs can serve as a foundational enabler to help expedite procurement for urgent operational requirements.

Of note, the Panel reviewed several projects that showed a positive trend in the effective development and application of HLMRs. The Panel also welcomed efforts by the Chief of Force Development organization to explore ways to strengthen the Department’s approach to HLMRs. The Panel looks forward to further action in that regard.

Options

The Panel found that options analysis was generally sound, and that projects present a reasonable and well-substantiated recommendation for decision-makers’ consideration. However, the Panel found that projects are often pressed to make significant scope decisions at the end of Options Analysis, often due to changing cost or funding figures. The Panel notes the importance of capability ladders to help guide these scope decisions and welcomes the fact that they have become a consistent part of project documentation.

Going forward, the Panel encourages the Department to issue guidance on, and standardize the use of, capability ladders that are solidly anchored in project HLMRs. In addition, the Panel recommends providing projects with earlier costing support to avoid project delays due to affordability issues and subsequent rescoping efforts.

Procurement

Over the past two years, defence procurement officials have faced increasingly challenging market dynamics characterized by limited industrial capacity, complex supply chains, rising inflation, and growing international demand, including from our allies. In its reviews, the Panel examined several projects where rapidly increasing market demand and inflationary dynamics created additional pressure to expedite procurement processes while also presenting significant cost risks.

The Panel believes that this further emphasizes the need for a more efficient and agile approach to defence procurement. Over the course of 2021 and 2022, the Panel continued to observe a growing appetite for alternate approaches to defence procurement. However, efforts in this regard have remained largely conceptual, and practical application is still required.

The Panel is encouraged by the Department’s Continuous Capability Sustainment Initiative and recommends the Department further define the concept and pilot it as soon as possible. The Panel also recommends rapidly mapping the current capability development and project process to determine roadblocks and delays, including in governance, to facilitate speeding up procurement where possible.

In 2021 and 2022, the Panel also reviewed several projects that only had one feasible solution due to regulatory, interoperability, or platform constraints. The Panel believes these projects could represent “quick wins” for a more expedited process and free up valuable departmental resources to address more complex projects.

Back to top

Looking Ahead: Priorities for 2023 and 2024

In line with its Mandate and Terms of Reference, the IRPDA will continue to uphold its fundamental and enduring priorities: maintaining Panel independence and credibility; supporting DND/CAF through the early phases of defence procurement; providing advice to senior decision-makers, most notably, the Minister of National Defence; and ultimately, enhancing trust, transparency and confidence in defence procurement – inside the Department and beyond.

Regarding the latter, the Panel will remain steadfast in its commitment to outreach and communication, and will continue to support the sharing of Panel advice with the Treasury Board, with the concurrence of the Minister of National Defence and when appropriate.

The Panel observes that speed, agility, and talent are increasingly important themes in defence procurement that will inform the context of its work for 2023 and 2024. Within the boundaries of its mandate, the Panel will remain mindful of its role in encouraging efficiency and effectiveness in the defence procurement process, and will continue to use its review role to foster understanding of increasingly complex projects. Looking ahead, the Panel will continue its important work of reviewing requirements, while staying attuned to these significant challenges, with the aim of having a positive, tangible, and lasting impact on defence procurement.

Back to top

Annex A

List of Acronyms
Name Acronym
Assistant Deputy Minister (Data, Innovation & Analytics) ADM DIA
Assistant Deputy Minister (Information Management) ADM IM
Assistant Deputy Minister (Infrastructure & Environment) ADM IE
Assistant Deputy Minister (Materiel) ADM Mat
Canadian Armed Force CAF
Canadian Army CA
Canadian Coast Guard CCG
Canadian Special Forces Command CANSOFCOM
Chief of Military Personnel CMP
Defence Capabilities Board DCB
Department of National Defence DND
High Level Mandatory Requirement HLMR
Independent Review Panel for Defence Acquisition IRPDA
Independent Review Panel for Defence Acquisition Office IRPDAO
Royal Canadian Air Force RCAF
Royal Canadian Navy RCN
Personnel, Research/Development, Infrastructure, Concepts, IM/IT, Equipment, and/or Gender-based Analysis Plus (GBA+) PRICIEG
Public Services and Procurement Canada PSPC
Vice Chief of the Defence Staff VCDS

Back to top

Annex B

The list below shows all projects that have initiated a review process with the Panel, as of the end of 2022. Projects that have completed the review process and upon which the Panel has provided advice to the Minister are marked with an asterisk (*).

Please note that the issuance of Panel advice does not necessarily indicate that a government decision on the project has been made.

List of Projects Reviewed by the IRPDA (2021 and 2022)
Project Name Sponsor
2018 Canadian Coast Guard Fleet Renewal Plan* CCG
Accommodate 4 Engineer Support Regiment* ADM(IE)
Accommodate 436 Squadron (Hangar 5) ADM(IE)
Advanced Improvised Explosive Device Detection and Defeat* CA
Advanced Short Range Missiles RCAF
AFEC Construct Main Facility (2 Wing Bagotville)* ADM(IE)
Airlift Capability Project - Multi-Role Flight Services* RCAF
Armoured Combat Support Vehicle* CA
Army Pistol Replacement* CA
Automatic Inventory Technology* ADM(Mat)
Bridge & Gap Crossing Modernization* CA
C6 General Purpose Machine Gun Modernization* CA
Camp Sustain CA
Canadian Joint Incident Response Unit Infrastructure* CANSOFCOM
Canadian Modular Assault Rifle CA
Canadian Multi-Mission Aircraft RCAF
Canadian Special Operations Training Centre* CANSOFCOM
Canadian Surface Combatant* RCN
CC-130J Block 8.1 Upgrade* RCAF
CH-148 Interoperability Upgrade RCAF
Combined Joint Intelligence Modernization CA
Common Heavy Equipment Replacement* CA
Cormorant Mid-Life Upgrade* RCAF
Cyber Defence - Decision Analysis and Response* ADM(IM)
Cyber Security Awareness ADM(IM)
Data-Centric Security Service ADM(IM)
Defence Enhanced Surveillance from Space RCAF
DefenceX (previoulsy Defence Resource Management Information System Modernization)* ADM(DIA)
Defensive Cyber Operations - Decision Support ADM(IM)
Domestic and Arctic Mobility Enhancement Project CA
DRDC Suffield Chemical Labs ADM(IE)
Electronic Health Record Platform* CMP
Enhanced Recovery Capability* CA
Enhanced Satellite Communications Project - Polar RCAF
Future Aircrew Training* RCAF
Future Combined Aerospace Operations Centre Capability RCAF
Future Fighter Capability* RCAF
Future Fighter Lead-In Training RCAF
Griffon Limited Life Extension* RCAF
Ground Based Air Defence* CA
Halifax Heating and Municipal Services Upgrade Project* ADM(IE)
Hornet Extension Project - Phase 1* RCAF
Hornet Extension Project - Phase 1 - Scope Change* RCAF
Hornet Extension Project - Phase 2* RCAF
Information Technology Infrastructure in Support of Command and Control* ADM(IM)
Integrated Torpedo Defence RCN
Joint Deployable Headquarters and Signal Regiment Modernization Project CA
Joint Fires Modernization* CA
Junior Non-Commission Member Training Accommodation Facility, CFB Esquimalt ADM(IE)
Land Command Support System Intelligence, Surveillance and Reconnaissance Modernization* CA
Land Command Support System Tactical Command and Control Information System Modernization CA
Land Command Support System Tactical Communications Modernization CA
Land Vehicle Crew Training System* CA
Light Armoured Vehicle - Specialist Variant Enhancement* CA
Light Utility Vehicle* CA
Lightweight Torpedo Upgrade* RCN
Logistics Vehicle Modernization* CA
Manned Airborne Intelligence, Surveillance and Reconnaissance* CANSOFCOM
Medium Range Air-to-Air Missiles RCAF
Multi Fleet Air Traffic Management Avionics* RCAF
Multi-Purpose Vessel* CCG
National Defence Operational Headquarters ADM(IE)
Naval Communication Modernization* ADM(IM)
Naval Inshore Support Vessel RCN
Naval Large Tug* RCN
Next Generation Military Pay System CMP
Night Vision System Modernization* CA
Polar Icebreaker* CCG
Program Ice Breaker* CCG
RCN Intelligence, Surveillance, Target Acquisition, and Reconnaissance* RCN
Remotely Piloted Aircraft System (formerly JUSTAS)* RCAF
Secure Radio Modernization* ADM(IM)
SOFCOM Capabilities and Recapitalization Project - Next Generation Fighting Vehicle* CANSOFCOM
Strategic Tanker Transport Capability* RCAF
STRONGBOW* RCN
Surveillance of Space 2* RCAF
System of Training and Operational Readiness Modernization RCN
Tactical Integrated Command Control and Communications* RCAF
Tactical Narrowband Satellite Communications - Geosynchronous Coverage* RCAF
Tactical Power System CA
UOR - Air Defence CA
UOR - Counter Uncrewed Aircraft System CA
UOR - Portable Anti-X Missile* CA
Utility Transport Aircraft RCAF
Victoria Class Modernization - Periscope and Flank Array* RCN
Weapon Effects Simulation Mid Life Upgrade CA

Back to top

Annex C: Survey Results

The Independent Review Panel for Defence Acquisition (IRPDA) survey ran from 8 February to 13 March 2023.

The survey’s intended audience included Directors General and working-level staff who had participated in a Panel engagement as either a project sponsor or a project implementer. The survey was circulated in both official languages to the relevant Directors General in the force development community, who in turn were asked to circulate the survey amongst their teams.

The IRPDAO received 23 completed surveys. The survey consisted of 28 multiple choice questions and four open-ended questions organized in four broad sections: mandate, engagement process, feedback and advice, and impact. The questions for the 2023 survey were the same as the 2020 survey, with only a few minor updates. This was done to ensure a consistent approach for the survey and facilitate an evaluation of results over time.

Overall, the 2023 survey received a very positive response, with a notable increase in positivity as compared to the 2020 survey. Below is a brief summary of the survey feedback organized by topic, followed by the detailed multiple choice survey results.

Mandate

Participants indicated that the Panel’s mandate is clear and well-understood, and that the Panel operates in an independent manner. These answers were generally consistent with similar responses from the 2020 survey. In particular, participants nearly unanimously agreed that the Panel continued to operate within its mandate, which is a notable increase in positivity from the 2020 survey (by more than 30%).

Engagement Process

Participants indicated that the Panel engagements are largely seen to be useful exercises that occur at the right moment in the project approval process. Additionally, project teams were generally confident that they knew what to expect from Panel engagements. The answers regarding preparatory work with the office were overall consistent with similar responses from the 2020 survey, but there was a notable increase in positive responses regarding the IRPDA engagements themselves (by 20% to 30%, depending on the question).

Feedback and Advice

A large majority of participants indicated that both the IRP1 feedback email and the IRP1 VCDS letter were clear, useful, and consistent with project management guidance. These answers were broadly consistent with responses from the 2020 survey, and in some cases, reflected a significant increase in the positive response rate.

Responses indicated a majority of project staff assessed that they incorporate over 50% of the Panel’s IRP1 and IRP2 feedback into subsequent project documentation. This is a notable increase from the 2020 survey, where the majority of project staff were uncertain whether the Panel’s IRP2 feedback was incorporated into subsequent decision documents.

Most participants (65%) continued to indicate that it would not be helpful to receive additional feedback from the Panel, but less so than in the previous survey (by 15%). Within the minority of respondents that indicated that more Panel feedback would be useful, they indicated a general desire for additional detail, background information on the Panel’s reasoning, and clear direction on what should be included in project documentation.

Impact

Participants indicated a more mixed understanding of the impact of the Panel on the project, although responses remained generally positive. For the most part, participants agreed that the Panel helped improve the project documentation; helped to clarify the policy cover, the capability gap, the HLMRs, and the project options; and had an overall positive impact on project evolution. These responses indicated a significant increase in the positive response rate compared to the 2020 survey.

Concluding Questions

The survey ended with a series of open-ended questions inviting feedback from participants on what the Panel does well, what could be improved, and the effectiveness of the Panel’s virtual working methods.

Regarding what the Panel does well, participants generally noted that the Panel helps improve a project’s narrative. Responses regarding potential improvements to the IRPDA were more varied, with suggestions ranging from procedural changes (e.g. earlier preparatory engagements with the IRPDAO, more direct and up-front feedback), to mandate changes (e.g. increasing the financial threshold for a project review, requesting the Panel review and advise the MND and other ministers on the procurement process), to sharing the Panel’s advice with the Canadian public.

Participants indicated that both virtual and in-person engagements have benefits. Although the Panel effectively adapted to a virtual environment, participants appreciated the ability to have direct and in-depth discussions with the Panel during in-person engagements. However, some respondents also noted that the virtual environment enabled more members of the project team to listen to the engagement, ensuring that the whole team had a better understanding of Panel concerns.

Multiple Choice Survey Results
- 0 1 to 2 3 to 4 5 to 6 More
Q1. How many Panel engagements have you participated in (including IRP1, IRP2 and supplementary engagements)? 4% 39% 22% 13% 22%
  Strongly Agree Agree Disagree Strongly Disagree N/A
Q2. I am familiar with the mandate of the Independent Review Panel for Defence Acquisition (IRPDA). 57% 43% 0% 0% 0%
Q3. The IRPDA mandate is clear. 52% 48% 0% 0% 0%
Q4. The IRPDA operates within its mandate. 39% 57% 4% 0% 0%
Q5. The IRPDA acts in an independent manner. 52% 48% 0% 0% 0%
  Yes No      
Q6. Was the project team invited to attend a preparatory meeting with the IRPDA office staff in advance of the Panel engagement(s)? 100% 0%      
  Strongly Agree Agree Disagree Strongly Disagree N/A
Q7. The preparatory meeting with the IRPDA was helpful in preparing for the Panel engagement(s). 74% 22% 0% 0% 4%
Q8. The IRPDA office staff gave me sufficient information to prepare for the Panel engagement. 61% 35% 0% 0% 4%
Q9. I knew what to expect from the Panel engagement(s). 43% 52% 4% 0% 0%
Q10. The Panel engagement(s) were useful. 57% 39% 5% 0% 0%
Q11. The Panel engagement(s) occurred at the right time in the project approval process. 35% 48% 13% 0% 4%
Q12. The IRP1 Panel feedback email was clear. 43% 57% 0% 0% 0%
Q13. The IPR1 feedback email was useful. 39% 61% 0% 0% 0%
Q14. The IPR1 feedback email was consistent with internal project management guidance. 35% 57% 4% 0% 4%
  Yes No N/A    
Q15. Was the Panel’s IRP1 VCDS Letter shared with you after the engagement(s)? 61% 35% 4%    
  Strongly Agree Agree Disagree Strongly Disagree N/A
Q16. The Panel’s IRP1 VCDS Letter was clear. 57% 43% 0% 0% 0%
Q17. The Panel’s IRP1 VCDS Letter was useful. 50% 50% 0% 0% 0%
Q18. The Panel’s IRP1 VCDS Letter was consistent with internal project management guidance. 43% 57% 0% 0% 0%
  Yes No      
Q19. Would you find it useful to receive additional feedback from the Panel? 35% 65%      
  Less than 25% 25-50% 51-75% 76-100% N/A
Q20. In general terms, what proportion of the Panel's IRP1 feedback was incorporated in subsequent project documentation, such as the Business Case Analysis and the Preliminary Statement of Operational Requirements? 4% 13% 26% 39% 17%
Q21. In general terms, what proportion of the Panel's IRP2 feedback was incorporated in subsequent project documentation and decision documents, such as Ministerial Submissions or Treasury Board submissions? 9% 4% 26% 35% 26%
  Strongly Agree Agree Disagree Strongly Disagree N/A
Q22. The IRPDA engagement(s) helped improve the project documentation. 52% 43% 4% 0% 0%
Q23. The IRPDA engagement(s) helped clarify the policy cover of the project. 36% 50% 9% 5% 0%
Q24. The IRPDA engagement(s) helped clarify the capability gap of the project. 39% 35% 17% 0% 9%
Q25. The IRPDA engagement(s) helped clarify the High-Level Mandatory Requirements (HLMRs) of the project. 39% 43% 13% 4% 0%
Q26. The IRPDA engagement(s) helped clarify the options of the project. 22% 61% 9% 0% 9%
Q27. The IRPDA engagement(s) positively impacted the evolution of the project. 39% 48% 4% 0% 9%
Q28. The IRPDA engaged in a timely and efficient way, such that the project was not unduly delayed. 48% 39% 13% 0% 0%

Back to top

Annex D: Interview Results

Summary

Interviews of eighteen senior leaders were conducted by a third party between 28 February and 31 March 2023. Feedback indicated that the Panel is widely perceived to be responsive and credible, and to provide significant value to the Department and beyond, with very few caveats identified.

Some recurring themes emerged in the interviews regarding the state of procurement more broadly – namely, the need for more speed, talent and agility in the defence procurement process. These themes may warrant further Panel reflection, as they may impact future acquisition practices and Panel process.

Consolidated Responses

1. To what extent do you think the IRPDA operates within its mandate?

Interviewees unanimously see the Panel operating within its mandate and being disciplined about this.

2. To what extent does the IRPDA increase your confidence in project requirements?

Interviewees agreed that the Panel’s questions are well-prepared and, as one explained, “will point to missing logic and call it when the business outcome does not make sense. This forces completion of the work and better products in the end.” Some interviewees responded that the Panel “increased their confidence in project requirements to a great extent” and gave examples of improved HLMRs. Others nuanced their response, explaining that their confidence in the requirements remained unchanged but that the Panel increased their confidence in the ability to articulate, describe, and capture all requirements.

Almost all mentioned how valuable the Panel was for helping make requirements succinct, and that the Panel’s validation was enormously beneficial.

3. Is IRPDA feedback and advice received at the right time in the project approval process? If not, what needs to change?

Interviewees unanimously agreed that the timing fits correctly within Departmental governance and approval processes. Interviewees indicated that feedback at IRP1 allows sponsors to address concerns through Options Analysis. Sponsors commended the Panel for agreeing to adjust the timing of some project reviews when urgent. Interviewees indicated that IRPDA is not a process bottleneck.

4. Do you find the IRPDA advice clear?

Almost all interviewees had received and read Panel advice (the exception being a newly appointed interviewee), and unanimously agreed that the advice was very clear. They appreciated how the Panel linked the threat environment and HLMRs, and found the consideration of capability ladders to be very useful in subsequent project phases.

5. Do you find the IRPDA advice credible (i.e. accurate, appropriate, thoughtful)?

Panel advice was seen as appropriate, thoughtful, concise yet comprehensive, and highly credible. Interviewees noted how Panel advice reflects the wealth of experience resident within the Panel.

6 & 7. How do you use the IRPDA feedback or advice? and Do you see others using the IRPDA feedback or advice? If yes, how?

Panel feedback and advice were deemed to be helpful on many fronts. Internally, Panel feedback helped project teams to sharpen the articulation of minimum requirements, and served as a valuable mentorship tool. More broadly, Panel feedback and advice supported improvements to the force development process, and enhanced communication on major procurement projects at the individual, institutional level, and Government of Canada level. Externally, Panel advice was viewed as a “vote of confidence” by central agencies.

8. In what ways do you think the IRPDA is effective?

Interviewees confirmed that the Panel is effective, responsive and provides useful advice. There was consensus on the vital importance of an independent review of project requirements, and acknowledgement that the Panel’s most important contribution is to provide truthful and “fearless advice” on CAF requirements, independent of cost considerations or internal expectations.

Some interviewees recognized that it is too early to tell whether the Panel’s external review of requirements yields better project outcomes, or whether more effort should be applied elsewhere in the procurement process.

9. In what ways do you think the IRPDA could be improved?

Additional Comments

During the interviews, it became clear that DND’s acquisition enterprise was facing some challenges beyond the Panel’s mandate. Three themes that came up repeatedly were the need for more speed, talent, and agility in defence procurement.

Back to top

Annex E: IRPDA Review Methodology

The grid below reflects the Panel’s general approach to project reviews, in accordance with the areas of focus that were set out in the Panel’s Terms of Reference and with a clear distinction between IRP1 and IRP2 engagements. This is an evergreen document, and is updated periodically to reflect evolving project considerations and priorities. The grid below is an accurate reflection of the review methodology during the reporting period.

IRPDA Questions
- Strategic Content Capability Gap Requirements Options Procurement
IRP1

What are the relevant strategic drivers (or constraints) for this project?

  • The international and/or domestic security environment? Adversary capabilities?
  • Technology?
  • Regulations?
  • Allied agreements and/or expectations?

What about Government policy?

  • Is the project aligned with SSE? Is it funded?
  • Is it aligned with other relevant government policies?
  • Is additional policy guidance required?

What are the main risks identified for this project (schedule, funding, etc.)?

What capability gap is being addressed?

  • Has this capability requirement been identified in capability-based planning?
  • What is the concept of operations for this capability? Is it well-illustrated (e.g. operational vignettes, etc.)?
  • How does it fit with other DND/CAF capabilities (i.e. linkages, dependencies)?
  • How is DND/CAF impeded without this capability (i.e. what is the risk of the status quo)?
  • What is the scope and the scale of the capability gap?
  • How does this capability fit with partner capabilities (i.e. other government departments, allies)?

What are the High Level Mandatory Requirements?

  • What is the evidence to support the HLMRs? Do they flow logically from the strategic context and capability gap, and align with the business outcomes?
  • Do the HLMRs capture the full intent of the project?
  • Do they reflect the scope and scale of the required capability?
  • Do the HLMRs fulfill the six criteria (clear, essential, measurable, sufficient, comprehensive, results oriented)?
  • Have requirements regarding training, infrastructure and growth potential been captured? If not, why?

What are the potential options?

  • Are the options capability-based?
  • Do they offer a range of potential capability levels (i.e. a capability ladder)?
  • Does the CAF need to own the capability? Are there other options? Partnership possibilities?
  • What are the criteria that will be used to assess the options? How were the criteria developed?

What are the procurement considerations?

  • How many potential suppliers are anticipated?
  • Is the project expected to leverage proven technology, or require developmental work?
IRP2
  • Has the strategic context, as noted above, changed since IRP1?
  • Has the capability gap evolved, become more/less acute, or been better defined since IRP1?
  • Have there been any recent changed to other DND/CAF and/or partner capabilities, thereby affecting the capability gap to be addressed in this project?
  • Have there been any changes to the HLMRs?
  • Has gender-based analysis informed the requirements?
  • Which HLMR(s) present the greatest challenge/risk?
  • What is the Preliminary Statement of Requirements?
  • Is the PSOR traceable to the HLMRs?
  • Are the mandatory requirements sufficient to meet the HLMRs?
  • Were there any changes to the design of the options, or the selection criteria?
  • Do certain requirements constrain the options available (e.g. interoperability)?
  • What is the recommended option?
  • What methodology was used to select the preferred option?
  • Is it affordable? If not, what is the prioritization methodology?
  • What are the key risks associated with this option? Cost? Schedule? Residual gap?
  • How have potential suppliers been consulted on the HLMRs?
  • What concerns have potential suppliers expressed?
  • To what extent will the HLMRs limit competition?
  • Is there a potential role for Canadian industry?
  • Is the proposed schedule feasible? Risk of delays?
  • Which requirements are expected to be the key cost drivers?

Back to top

Annex F

IRPDA and IRPDA Office Budget and Total Expenditures
- FY 2020-2021
Budget
FY 2020-2021
Expenditures
FY 2021-2022
Budget
FY 2021-2022
Expenditures
Operating and Maintenance $200,000.00 $29,816.00 $200,000.00 $27,292.00
Salary (Panel members and IRPDAO staff) $1,407,040.00 $1,281,231.00 $1,407,040.00 $1,182,194.00

Back to top

Annex G: Panel Members Biographies

Chair

Mr. Larry Murray CM, CMM, CD

Mr. Larry Murray has held a number of senior positions in the Canadian Armed Forces and Public Service. During his career with the Canadian Armed Forces, he served at sea in a variety of ships and held several senior positions, including as Deputy Chief of the Defence Staff, Commander of Maritime Command, Vice Chief of the Defence Staff and, finally, Acting Chief of the Defence Staff from October 1996 until September 1997. Mr. Murray retired from the Canadian Armed Forces in 1997 and joined the Public Service as Associate Deputy Minister of Fisheries and Oceans. He was appointed Deputy Minister of Veterans Affairs Canada in 1999 and Deputy Minister of Fisheries and Oceans in 2003. He retired from the Public Service of Canada in 2007.

Since retiring from the Public Service, Mr. Murray has served on Task Forces, Advisory and Audit Committees. He is also a former Chair of the Board of the Public Policy Forum, a former President of the Nova Scotia Mainland Division of the Navy League of Canada and honourary Grand President of the Royal Canadian Legion. Mr. Murray has received many prestigious awards in recognition of his leadership in the Canadian Armed Forces, Public Service of Canada, national voluntary commitments, as well as his support to Canadian Armed Forces personnel, Veterans and their families. He was appointed as a Member of the Order of Canada in 2013.

Members

Mr. Martin Gagné

Mr. Martin Gagné spent 17 years at CAE before retiring as Group President for Military Simulation and Training in 2012. During his career with CAE, he served in various roles such as: Vice-President of Visual Systems, Vice-President of Military Marketing and Sales, and Executive Vice-President of Civil Simulation and Training.

Prior to joining CAE, Mr. Gagné acquired extensive management and leadership experience during his 23 years as a member of the Canadian Armed Forces. As a senior aerospace engineering officer he was involved in the acquisition and maintenance activities of various aircraft fleets including the CF-18 and Maritime Helicopter Project. He holds degrees in electrical engineering and computer engineering as well as a certification in business administration from the McGill Executive Institute. Mr. Gagné served several terms on the Board of Directors for the Canadian Association of Defence and Security Industries (CADSI) and is on the Canadian Commercial Corporation’s (CCC) Board of Directors as the Chair of the Operations Committee.

Ms. Christine Tovee

Ms. Christine Tovee has over 15 years of engineering leadership in aerospace development programs spanning early concept and requirements through detailed design, test, and validation. She then returned to research and development culminating in being promoted to Vice President of Research and Technology and Chief Technology Officer for Airbus Group Inc., in the United States (formerly known as EADS North America) Recently, she has taken the position of CTO at a Canadian space startup aiming to deliver hyperspectral imagery to new vertical markets. At BAE Systems and EADS, Ms. Tovee held positions in Germany, France, and the United Kingdom, contributing to national and European space programs. She has led and collaborated on defence projects in all domains: land, air, sea and space, focusing on joint operations, C4ISTAR systems and military satellite communications.

Ms. Tovee was seconded to the UK Ministry of Defence to lead the technical aspects of the Joint Network Integration Body (JNIB). This programme combined the efforts of MoD and multiple defence contractors to identify and solve the integration challenges in providing a seamless information and communications system. She is currently a member of the Government of Canada’s Space Advisory Board. She has also been appointed as a Fellow at the Creative Destruction Lab at the Rotman School of Business in Toronto. Ms. Tovee holds a Bachelor of Applied Science in Aerospace Engineering from the University of Toronto, and a Master of Science in Aeronautics and Astronautics from Massachusetts Institute of Technology (MIT).

Ms. Linda Lizotte-MacPherson

Linda Lizotte-MacPherson is an experienced Director, Executive Chair and accomplished CEO and deputy minister with a unique blend of experience in government and the private sector. She is an expert in strategy development, IT, including cyber security, operations, risk management and public administration and has a proven track record with large scale complex projects and transformation and change initiatives.

Ms. Lizotte-MacPherson held several deputy minister positions in the federal government from 2005 to 2016. These included President of the Canada Border Services Agency, President of the Canada School of Public Service, Commissioner and Chief Executive Officer of the Canada Revenue Agency, Senior Associate Deputy Minister of Human Resources and Skills Development Canada and COO Service Canada and Associate Secretary, Treasury Board Secretariat.

In the private sector, Ms. Lizotte-MacPherson was President of two Canadian subsidiaries, Sapient Canada Inc. and AMS Management Systems Canada Inc., and she held a number of senior executive positions with Digital Equipment of Canada (DEC). She was also the first Chief Executive Officer of Canada Health Infoway, where she successfully launched the corporation.

Dr. Elinor Sloan

Elinor Sloan is Professor of International Relations in the Department of Political Science at Carleton University, Ottawa, where she specializes in Canadian and United States security and defence policy and military capabilities. A graduate of the Royal Military College of Canada, she holds a PhD from the Fletcher School of Law and Diplomacy at Tufts University, and an MA from the Norman Paterson School of International Affairs at Carleton University.

Prior to joining academia, Dr. Sloan was a strategic analyst in Canada’s National Defence Headquarters where she researched the policy implications of US-led advancements in military technologies, and prior to that a logistics officer in the Canadian Armed Forces regular force. Dr. Sloan teaches in the areas of International Security Studies, Transatlantic Security Issues, and North American Security and Defence Policy. She is author of seven books on security, defence and military strategy, including two second editions, as well as many journal articles.

Back to top

Annex H: IRPDAO Contact Details

Independent Review Panel for Defence Acquisition Office (IRPDAO)

Mailing address:

National Defence Headquarters – Pearkes BuildingFootnote *
101 Colonel By Drive
Ottawa, Ontario K1A 0K2
E-mail: IRPDA-CIEAD@forces.gc.ca
Website: https://www.canada.ca/en/independent-review-panel-defence-acquisition.html

Back to top

Back to top

Page details

Date modified: