Independent Review Panel for Defence Acquisition – Annual Report 2016-2017

Introduction

This is the Independent Review Panel for Defence Acquisition’s second annual report. It includes a summary of the Panel’s achievements for Fiscal Year 2016-2017, observations from its work over the past year, and a statement of the Panel’s priorities for the coming year. Appendix A provides information about the membership of the Panel, and Appendix B sets out a financial summary for both the Panel and the Office that serves as its day to day presence within the Department of National Defence (DND).

In accordance with its Terms of Reference and review methodology, the Panel engages projects at two points in the procurement project approval process before DND and the Canadian Armed Forces (CAF) approach the Minister of National Defence (the Minister) or Treasury Board for expenditure authority. Specifically, the Panel reviews projects after Identification, at the beginning of Options Analysis, and at the end of Options Analysis, prior to Definition (see Figure 1). During these engagements, the Panel mainly focuses on the alignment of a project with Government policy, the nature of the capability gap a project is intended to address and, ultimately, the clarity and appropriateness of the stated requirements. Advice, which is provided to the Minister through the Deputy Minister, with a copy to the Chief of Defence Staff, is provided after the Panel has reviewed a project at the end of the Options Analysis phase.

Starting next Fiscal Year, the context in which the Panel will perform its review will be enhanced by Canada’s new Defence Policy released in June 2017. Strong, Secure, Engaged provides comprehensive and current policy direction, and will serve as a fundamental benchmark against which the Panel carries out its challenge function. The Panel also believes that the release of the fully costed Strong, Secure, Engaged helps clarify the prioritization of procurement projects.

2016-2017 Accomplishments

During Fiscal Year 2016-2017, the Panel reviewed a total of fourteen (14) projects and submitted advice to the Minister of National Defence on seven (7) of these projects. Since it became operational in June 2015, the Panel has engaged on twenty-four (24) projects across the full spectrum of National Defence activities, including capabilities in support of the Royal Canadian Navy, the Royal Canadian Air Force, the Canadian Army, and the Special Operations Forces, infrastructure, and information technology.

To support the preparation of this report, the Panel assessed its performance using a Performance Measurement Framework with the following strategic objectives:

• Appropriate Requirements – Requirements validated by the IRPDA support the delivery of the right equipment to the CAF in a timely manner.
• Useful Advice – Advice provided to the Minister of National Defence, through the DM, enhances decision-making on major military procurement projects.
• Enhanced Confidence – The defence procurement process is improved by enhanced confidence and trust in decision-makers and improved credibility within the Canadian defence and security industry.

While there have been concerns in the past that the military’s requirements have exceeded what is necessary to address a stated capability gap or that they were directed towards a specific solution, the Panel has not seen evidence of excessive or biased requirements across the projects it has reviewed to date.

In assessing its performance, the IRPDA considered quantitative data in the form of numbers of projects reviewed and pieces of advice issued, as well as qualitative feedback gathered from a range of stakeholders through surveys and in-person meetings. The results of this assessment indicate that decision-makers and senior officials find the Panel’s advice and recommendations to be relevant and timely. In particular, DND/CAF officials have indicated that they find engagements with the Panel to be helpful in advancing their projects due to the comprehensive approach the Panel takes in reviewing and providing feedback on project documentation.  Specifically, officials have highlighted that the Panel’s observations and recommendations have enabled them to develop clearer requirements and helped them address shortfalls in project analyses, with the result that project staff are now better prepared for subsequent steps in the project approval process.

Given that the defence procurement process takes place over a number of years, it is still too early to assess the longer term strategic impact of the Panel’s work. Senior officials within the Department have noted that there has been a positive shift in perception in other Government departments, central agencies and in industry regarding the quality of military requirements as a result of the establishment of the Panel and enhanced governance. This change, however, is difficult to measure.

Panel Observations

DND/CAF Engagement with the Panel

Overall, the Panel has found that DND/CAF officials strive to meet the Panel’s expectations and that the quality of their engagements with DND/CAF continues to improve. 

The Panel appreciates the significant support it has received from the Chief of Force Development organization, both in terms of ensuring that projects come to the Panel for review at the appropriate time, but also for embedding the concept of an independent third-party review of project requirements in the project management guidance that is provided to all DND/CAF staff.

The Panel is also confident that it has had full access to all relevant project information and analysis required to carry out its challenge function effectively.

The Panel greatly appreciates the highly professional, dedicated and knowledgeable support provided by the IRPDA Office, as this support has been essential to its success since the Panel was established in 2015.

Streamlining the Project Approval Process

The Panel places a high priority on not creating unnecessary burdens for project teams or causing delays in the project approval process. In this regard, the Panel always endeavours to exercise its challenge function using existing DND/CAF documents and has avoided making requests for project teams to produce new analyses. The Panel and its Office also works closely with DND/CAF officials to ensure that its forward agenda corresponds with DND/CAF priorities, and that it engages projects in a timely and responsive manner.

The Panel notes the implementation, in the last Fiscal Year, of an expedited DND/CAF approval process for adjustments made to project documentation in response to Panel recommendations. This improvement has led to more timely supplementary engagements with the Panel and helped ensure that projects stay on track with projected timelines. The Panel encourages DND/CAF to continue seeking additional ways to streamline the project approval process.

Linking Strategic Context and Policy Direction to Project Requirements

The Panel notes that many of the projects it reviewed in Fiscal Year 2016-2017 demonstrated a better explanation of their policy foundation and a more rigorous and clear statement of the identified capability gap. That said, the Panel continues to find that some project documentation lacks detail on how a proposed project will address an identified capability gap from an operational perspective, and on the resulting impact should the proposed project not go forward. Furthermore, the Panel encourages project teams to place more emphasis on providing clear and comprehensive explanations of project interdependencies and on how proposed capabilities fit within the broader DND/CAF context. In this regard, the Panel is hopeful that the release of Strong, Secure, Engaged will assist project teams in better explaining linkages between individual projects and strategic level direction.

The Panel continues to place a high value on the CAF’s use of capability-based planning in identifying priorities based on rigorous analyses of CAF capabilities and gaps. In this regard, the Panel understands and is supportive of ongoing DND/CAF work to develop a detailed implementation plan for Strong, Secure, Engaged that is informed by capability-based planning analysis.

Improved Guidance

The Panel is very supportive of the considerable effort made by DND/CAF leadership in improving the analysis in support of the development of project requirements. In particular, the Panel notes the release of new departmental guidance to project teams on the development of business cases (including key considerations for the assessment of capability gaps and development of High Level Mandatory Requirements), and on the identification and analysis of options. This new departmental guidance also offers advice on Panel engagement based on lessons learned from the past two years. The Panel believes that the release of this guidance will help improve project management across DND/CAF over the long term.

High Level Mandatory Requirements

High Level Mandatory Requirements (HLMRs) are non-specialist expressions of defence requirements that articulate the expected outcomes or effects to be delivered by a project. The purpose of HLMRs is to support decision-making. To this end, the Panel assesses HLMRs for their ability to address a stated capability gap and their alignment with the Preliminary Statement of Operational Requirements for the project.

As noted in its 2015-2016 Annual Report, and as observed during this Fiscal Year, the Panel has seen no evidence of requirements significantly exceeding what is necessary to address a stated capability gap, or requirements being developed in such a way that they lead towards a specific solution.

The Panel observes, however, that the development of appropriate HLMRs continues to challenge project teams. For example, the Panel still occasionally sees very broad HLMRs that are not useful in screening potential options and/or HLMRs that are not measurable. The Panel notes that the feedback it has provided in these instances has generally been welcomed by the DND/CAF force development community, and is pleased that there is openness to ongoing dialogue about the quality of HLMRs, and the important role they play in acquiring appropriate defence capabilities.

Analysis of Options

During Fiscal Year 2016-2017, the Panel found that the majority of projects it reviewed required supplementary engagements, often because the rationale for the proposed options was underdeveloped or the supporting analysis was insufficient.  For example, the Panel continues to see many projects where the options presented are largely procurement strategies rather than options focused on capabilities or on the scale of the project. In addition, the Panel notes that project teams still struggle to provide sound rationales for the proposed quantity of equipment to be procured. Lastly, while the Panel acknowledges DND/CAF efforts to consult industry, allies and other subject matter experts during options analysis, it encourages DND/CAF to strengthen these consultations to achieve the most robust analyses possible.

Further, the Panel has indicated to DND/CAF that there would be benefit in simplifying the methodology used to assess the range of options being considered. This observation stems from the review of many projects where the analysis of options is not clearly explained, sometimes leading to unsubstantiated conclusions. In the Panel’s view, the consistent use of a simplified but rigorous methodology for options analysis would help decision-makers to better understand the selection of a preferred option and, potentially, the rationale for the level of investment being pursued.

Finally, the Panel continues to believe that teams engaged in the development and assessment of options in both the force development and materiel management communities require more resources and support.  

Priorities for Fiscal Year 2017-2018

The Panel will continue to review and provide advice to the Minister of National Defence on requirements for projects and capabilities identified in Strong, Secure, Engaged. The Panel will also maintain its focus on improving the clarity and comprehensiveness of HLMRs, and on the importance of identifying valid options and conducting rigorous options analyses.

The Panel must have appropriate resources and capacity to deliver on these priorities. To this end, it will support the Governor-in-Council appointment processes required to fill its current vacancy and address the fact that two current members’ terms are scheduled to end in May 2018.

As always, succession planning for key positions within the IRPDA Office will also be a priority given the essential contribution made by this small team of public servants who ensure the efficient and effective functioning of the Panel.

Conclusion

Over the course of Fiscal Year 2016-2017, the Panel has noted an improvement in the quality of analysis and explanation that is provided to support decision making, particularly as it relates to the explanation of linkages between government policy and project requirements. There is still room for improvement in the explanation of requirements, as well as in the analysis of project options. The Panel looks forward to continued improvement in the development of project documentation across the DND/CAF.

The Panel notes that its work during Fiscal Year 2016-2017 has been much enhanced by excellent support from both the DND/CAF force development community as well as the broader Government of Canada defence procurement community. The Panel would also like to commend the IRPDA Office for its excellent work in supporting the Panel.

Lastly, the Panel would like to commend and offer a special note of thanks to Ms. Mollie Royds, the former Executive Director of the IRPDAO, given her seminal role in standing up the Panel and effectively putting in place the processes required to support it in successfully delivering on its mandate. The Panel wishes to congratulate Mollie on her recent and well-deserved promotional appointment at the Treasury Board Secretariat.

Appendix A: Panel Membership

Note: The normal composition of the Panel consists of a Chair and four members. The vacancy will be filled in Fiscal Year 2017-2018.

Appendix B: IRPDA and IRPDA Office Budget and Total Expenditures for 2016-2017