Audit of Preliminary Requirement Development Process for Capital Equipment Projects

1.1.2 Key Stakeholders

Many participants from project teams to oversight bodies are involved in the PAP process. Their responsibilities related to preliminary requirement development are outlined and illustrated in Figure 2.

Vice Chief of the Defence Staff

As part of its mission to ensure that Departmental defence policy and strategic objectives are achieved, the Vice Chief of the Defence Staff (VCDS) is responsible for providing guidance and coordinating the PAP. Since 2013, PAP responsibilities have been divided between the Chief of Force Development (CFD) and the Chief of Programme (C Prog).

• CFD provides guidance and support for project development in the ID and OA phases.
• C Prog provides policy and guidance for projects in the Definition, Implementation and Close-out phases.

Project Sponsor

The project sponsor for capital equipment projects, usually one of the three environments (CA, RCAF, RCN), is responsible for identifying an operational capability deficiency and for communicating the value and importance of a project to the Department. As well, the project sponsor leads the project through the ID and OA phases. The project sponsor team is comprised of:

• Project Leader (PL) – management level, accountable for project performance in the ID and OA phases.
• Project Director (PD) – military staff, responsible for developing key project documents and navigating through the approval gates during the ID and OA phases. The PD receives support from a Project Manager (PM), who is a member of the project implementer team.

Project Implementer

The project implementer takes over the project leadership from the project sponsor at the end of the OA phase, when project funding is approved, and commits to delivering the capability on schedule and within budget. The project implementer for capital equipment projects is typically the Assistant Deputy Minister (Materiel) (ADM(Mat)), and has the following team members:

• New Project Leader (PL) (ADM(Mat)) – accountable for project performance in the Definition phase and beyond; and
• Project Manager (PM) – manages the project while working with the PD to ensure project milestones are achieved; responsible for tendering the contract and ensuring the capability is delivered.

1.1.3 Oversight bodies and Challenge Function

Senior Review Board

The Senior Review Board (SRB) is a project-specific body established at the beginning of the OA phase to support management oversight of the project. It is chaired by the PL representing the project sponsor or the project implementer, depending on the project phase. SRB meetings are to be convened, at a minimum, on an annual basis to assess formal progress and project performance against approved baseline performance measures of scope, cost and schedule.^{Footnote 2}  Key project documents, such as the Project Charter, the SCD and Preliminary SOR require SRB endorsement. Minimum membership includes representation from the VCDS, Assistant Deputy Minister (Finance), Commander Navy, Commander Army, Commander Air Force and others as required by the Project Charter.

Chief of Force Development Analysts

At the departmental level, CFD analysts serve as the first challenge function in the ID and OA phases. CFD analysts review key project documents, such as the SCD and BCA, before they are presented to the Defence Capability Board (DCB) to confirm capability needs are aligned with strategic outcomes, government policy and the Capability-Based Plan. Additionally, analysts coordinate project documentation for the DCB and the Independent Review Panel for Defence Acquisition (IRPDA). Analysts also support project sponsors in responding to oversight-body feedback, questions and recommendations.

Defence Capability Board

Chaired by the VCDS, the DCB serves as a challenge function to ensure the alignment of future capabilities against departmental needs, comprehensive options development and analysis, and transparency and detail of project costing figures.^{Footnote 3}  Capital equipment projects are required to seek DCB approval twice during the preliminary requirement development process. DCB 1 takes place in the ID phase to approve a project's capability requirements, including the HLMRs, and viable options to be assessed. DCB 2 reviews the detailed analysis of options in the BCA and approves the preferred option with which to move forward. DCB attendance is limited to the L1 or L1A level, which includes the project sponsor, the project implementer, CFD and C Prog.

Independent Review Panel for Defence Acquisition

As a core element of the Defence Procurement Strategy, the IRPDA provides independent advice to the Minister of National Defence (MND) through the Deputy Minister (DM) for major defence projects exceeding $100 million or exceeding the Minister's delegated approval authority, as well as those identified for approval. Specifically, the IRPDA assesses the validity of the capability gap and the HLMRs, whether requirements are stated in a clear and appropriate manner, and provides its advice prior to Ministerial or Treasury Board approval.

IRPDA typically reviews each capital equipment project twice. The first review, IRP 1, which takes place at the end of the ID phase, focuses on each project's capability gaps and HLMRs after their endorsement by DCB 1. The second review, IRP 2, which occurs at the end of the OA phase, examines the results of the options analysis and the linkages between the HLMRs and the Preliminary SOR, following endorsement by DCB 2. Supplementary engagements may occur as necessary to address issues identified during either IRP 1 or 2. The project sponsor, CFD and the project implementer are required to attend IRPDA engagements. The Chair of the IRPDA may invite other DND/CAF specialists to participate if necessary.

1.2 Rationale

Delivering the right equipment to the CAF in a timely manner is a top priority for the Department, especially with the increase in long-term investments in military capability and new initiatives as described in SSE. SSE notes that capability requirements have not always been clearly communicated to industry and Canadians, and the desire to reduce project development and approval times by 50 percent.^Footnote4  An effective and efficient project preliminary requirement development process will enable the Department to clearly define, communicate and deliver its capability requirements in a timely manner. Given the importance of this subject, the audit was included in the ADM(RS) Risk based Audit Plan for fiscal years 2016/17 to 2018/19.

1.3 Objective

The objective of this audit is to determine if DND/CAF has an adequate preliminary requirement development process for capital equipment projects to support the delivery of defence capabilities.

1.4 Scope

The scope of this audit included the following:

• Preliminary requirement development process for capital equipment projects from the initial development of the SCD in the ID phase to the approval of the BCA and SOR in the OA phase.
• Oversight bodies and challenge functions involved in the preliminary requirement development process including the SRB, CFD analysts, DCB and IRPDA.
• Lessons learned and best practices of the preliminary requirement development process shared amongst the project teams.

The audit did not examine capital equipment projects that were equal to or below $5 million as they are subject to a different approval process nor IM/IT projects and infrastructure projects nor the validity of specific requirements.

Audit work was performed during the period from May 2017 to April 2018, and included oversight activities and documentation related to initiation of selected projects from 2007 to April 2018.

1.5 Methodology

The audit results are based on evidence from the following sources:

• Interviews with key CFD, Naval, Land and Air Requirement staff, C Prog, ADM(Mat) and IRPDA office staff.
• Project documentation and examination of a sample of five active capital equipment projects, selected as a result of a risk analysis of equipment conducted by ADM(RS) in June 2015, and two capital equipment projects provided by ADM(Mat). More details of the sample can be found in Annex D.
• An examination of policies, processes, the Capability Investment Database (CID) and the SSE initiatives SharePoint database.

1.6 Audit Criteria

• Appropriate processes and procedures are in place to support the development of preliminary requirements for capital equipment projects.
• Capital equipment project preliminary requirements align with HLMRs and are developed in accordance with DND policy.
• Sufficient capacity exists to support the preliminary requirement development process (guidance, resources, skills, etc.)

The audit criteria can also be found at Annex B.

1.7 Statement of Conformance

The audit findings and conclusions contained in this report are based on sufficient and appropriate audit evidence gathered in accordance with procedures that meet the Institute of Internal Auditors' International Standards for the Professional Practice of Internal Auditing. The audit thus conforms to the Institute of Internal Auditors' International Standards for the Professional Practice of Internal Auditing as supported by the results of the quality assurance and improvement program. The opinions expressed in this report are based on conditions as they existed at the time of the audit and apply only to the entity examined.

Back to Table of Contents

2.0 Findings and Recommendations

2.1 Governance

We expected to find that:

• Clear policies and guidelines are in place;
• Key documents demonstrate preliminary requirements align with HLMRs; and
• Authorities, roles and responsibilities of key stakeholders are clearly defined and are carried out appropriately.

2.1.1 Policy and Guidance

The main policy governing the PAP is the Project Approval Directive (PAD). Last updated in 2015, the PAD is accompanied by a Project Guidance Memorandum (PGM) 1-17, which incorporates supplementary information and updated guidance for the ID and OA phases. CFD has also created an Aide Memoire which serves to guide analysts who review key project documents. Templates for key documents, such as the BCA, are also available. As well, the VCDS Directive on Strategic Governance Board and Committees provides guidance on briefing packages that are submitted to oversight bodies. Although considerable guidance exists, a review of applicable policy and guidance identified areas that require greater attention to ensure key requirement documents are developed in a consistent and efficient manner.

2.1.2 Key Project Documents

Multiple sources of templates and guides were found across several key project documents, which resulted in documentation discrepancies in the projects reviewed. For example, the SORs in the sample projects varied significantly in how the linkages between HLMRs and operational requirements were demonstrated. The available SOR guide was last updated in 2006 and, therefore, does not meet the current PGM 1‑17 requirement of linking the operational requirements to their corresponding HLMRs. This linkage of operational requirements to HLMRs is important to ensure clear expectations and capability needs are defined. The preparation of consistent and quality preliminary requirement documents that ensure alignment of HLMRs and Preliminary SOR would help expedite project timelines and approvals.

Despite the inconsistency in how the requirements were presented, the review of oversight documentation (e.g., CFD, IRPDA) found that preliminary requirements were aligned with the HLMRs for all projects within the sample.

The review of the key project documents showed that similar information is presented in multiple documents. Documents reviewed, including the BCA, Preliminary SOR and Project Charter, contain common elements such as background, milestones and project interdependencies. This information is presented in four key documents which are reviewed and/or endorsed multiple times, as illustrated in Table 1. As such, project documentation includes inconsistent information due to ongoing updates and revisions, which leads to inefficiencies in the documentation development and review.

The review also showed the Project Charter was rarely endorsed in a timely manner, or at all by the SRB, which may indicate the need to review its value added. As illustrated in Table 1, additional opportunities to streamline documentation and gain efficiencies may exist.

Table 1. Key Project Documents – Review and Approval Authorities and Document Content. This table illustrates the review/approval authorities for each document, as well as the information presented within the documents. (For the Review/Authority columns, the ✓ demonstrates review/approval required by; the * indicates review at the discretion of CFD; the blank space denotes not applicable. For the Information Elements columns, the ✓ illustrates the information contained in each document; the blank space denotes not applicable.)

2.1.3 CFD Analyst Review

Key documents submitted for DCB endorsement and IRPDA review are first provided to CFD analysts who perform a challenge function. The direction provided by the PAD, PGM and CFD's Aide Memoire on which documents should be reviewed by the CFD analyst is unclear. For instance, the PGM states the HLMRs must be reviewed, but the Aide Memoire does not. Additionally, neither document provides sufficient information regarding the roles and responsibilities of the analysts. These discrepancies in guidance cause confusion for CFD analysts as well as project staff regarding the requirement to engage CFD analysts and the expectations regarding project documents.

2.1.4 Roles and Responsibilities

The DCB and IRPDA each have Terms of Reference. The PAD states that the DCB is the approval body in the ID and OA phases and has the authority to direct projects to proceed to the next phase. The IRPDA is a third-party advisory body that provides independent advice to the MND before projects seek Ministerial or Treasury Board approval. In doing so, the IRPDA engages the projects after DCB approval to provide a challenge function. During the IRPDA review process, the Panel also provides recommendations and suggestions to help project sponsors better prepare for Treasury Board or Ministerial submissions.

As previously noted, IRPDA typically reviews each capital equipment project twice. Four projects reviewed were subject to IRPDA review, two of which received DCB approval to proceed to Definition phase with a pending IRPDA engagement^{Footnote 5}  and recommendations. As IRPDA is an advisory body which performs its review after DCB approval is obtained, it is unclear to some project sponsors whether projects need to address the recommended changes to key documents before proceeding to the Definition phase or whether obtaining DCB approval is sufficient to proceed. This lack of clarity may result in projects being submitted for approval with unresolved issues or outstanding IRPDA recommendations. Interviews with project sponsor teams revealed differing views on how projects should manage IRPDA engagements and feedback. Some teams treated IRPDA feedback as optional. For example, the audit found two projects where the IRPDA provided the same recommendations at a subsequent engagement, as the previously identified issues had not been addressed. Others viewed IRPDA as an approval gate, as it submits its advice to the MND through the DM, and as such addressed and incorporated, as applicable, all IRPDA recommendations before seeking approval.

2.1.5 Conclusion

Although policy and guidance to support the preliminary requirement development process is in place, certain areas require clarification and updating to support a more efficient process. Templates for key project documents are not aligned with the new process requirements. Roles and responsibilities of internal governance bodies and external challenge functions are unclear and insufficient to guide project staff when engaging these bodies. There is also a lack of consolidated reference materials, which resulted in project documents that are not aligned with policy requirements or expectations of oversight bodies and challenge functions.
The afore-mentioned inefficiencies may result in a duplication of effort and prolonged approval process. However, the inefficiencies could not be quantified since data to support such analysis was not available.

Updated, streamlined and centralized guidance materials are needed to provide project teams and review analysts with the support they require to prepare key requirements documents in a consistent and efficient manner. Additionally, clarifying the roles and responsibilities of the oversight bodies and challenge functions would ensure key documents are developed in accordance with current requirements and expectations. Furthermore, greater clarity on the process and expectations of DCB and IRPDA would allow projects to move forward with submissions to the MND or Treasury Board in an efficient manner.

2.2 Oversight

We expected to find that:

• Oversight and challenge functions are in place and conducted on a timely basis;
• Project requirement information is complete, up-to-date and accurate; and
• Project requirement information is timely to support decision making.

2.2.1 Senior Review Board

In accordance with policy, project SRBs are to convene, at a minimum, on an annual basis to oversee project development and management once the project enters the OA phase, as shown in Figure 3.^{Footnote 6}  With one exception, each project reviewed missed at least one annual SRB. Overall, for projects examined, almost a quarter of required SRBs did not occur as required.

SRBs serve as a valuable platform for project teams to provide key updates and seek assistance on known issues. For instance, one sampled project approached SRB for direction and support in writing the preliminary SOR. Missed SRBs can amplify project challenges or prolong the approval process. For example, another sampled project reported to SRB that its Preliminary SOR was 95 percent complete. However, documentation showed the subsequent SRB was missed, which resulted in the Preliminary SOR approval being delayed by a year. Interviews indicated that missed SRBs were mostly due to scheduling conflicts or PL availability. PL availability is discussed in section 2.3.1. Documenting these rationales would provide information and context for new and existing project team members, to support effective project management.

SRB engagements require a presentation briefing deck to provide key updates on the project. The briefing deck and meeting records of decision (ROD) should provide decision makers with a comprehensive understanding of the work completed thus far, the current status and challenges, and the way forward. It is important that RODs are approved in a timely manner to support oversight bodies' review of the preliminary requirements and HLMRs and thus, allow projects to move to the next step in the PAP.

2.2.2 Defence Capability Board

DCB serves as a project approval authority for project capabilities in the ID and OA phases. When engaging DCB, project teams submit briefing packages which include a presentation deck and the SCD for DCB1 and the BCA for DCB2. The VCDS Directive on Strategic Governance Board and Committees requires that project presentations include a recap of previous direction that summarizes feedback from other oversight bodies and previous DCB engagements as well as highlight any action items or whether recommendations had been addressed. Of the five presentations reviewed, three contained the required recap.
The DCB RODs reviewed for the projects were concise and provided little information. Interviews with project sponsor teams confirmed the majority of the challenge or oversight is conducted prior to DCB engagement. While project staff indicated significant discussions occur prior to the formal DCB meeting, including numerous pre-briefs to key stakeholders, there is no practice in place to document, follow-up or provide a recap on the feedback, issues or decisions that occurred in those pre-discussions. As such, the challenge function performed by DCB, including its review of preliminary requirements and HLMRs, is not recorded to support actions taken and decisions made, or support decision making at subsequent oversight engagements.

A review of IRPDA feedback provided to the sampled projects revealed instances where issues, which fall within the review scope of the DCB, were first documented by the IRPDA. One such example was the questioning of the range of options developed and the method and rationale used to select the preferred option. These questions may have been raised by the DCB prior to its formal IRPDA engagement and subsequently addressed by the project team, however, the DCB RODs do not demonstrate how or what level of oversight was applied to the project. As such, a lack of documentation to support the oversight challenge undertaken by DCB or other stakeholders may result in the IRPDA offering more extensive comments and recommendations.

2.2.3 Chief of Force Development

In coordinating the internal and external oversight bodies and providing oversight through the review of key projects documents, CFD uses two main project management tracking tools: a Director Capability Integration tracker and an alignment review table. These tools are both internal to CFD with one serving as a project-level tracker while the latter serves as a management tool to highlight upcoming DCB engagements and projects entering Definition phase to help better prioritize projects. While both tools are used for different purposes, the information contained within is not consistent. As well, there is no central tracking tool used to record and track governance body engagement dates, document approval dates and the status of projects to ensure project schedules are being met.

The Capability Investment Database (CID) is currently the main departmental information system to provide visibility on the investment portfolio. As found in previous internal audits, CID information is often outdated or misaligned with what is found in key project documents. The inconsistent data affects the reliability of the information provided by the CID and requires additional validation. DND/CAF is transitioning towards the new project information repository, Defence Resource Management Information System (DRMIS) to consolidate capital project information. There is a risk that current documentation practices and information quality issues could affect the future effectiveness of DRMIS or other modern information management tools, as the data in the CID is transferred.

There is also no mechanism in place to document and follow up on feedback and recommendations provided by oversight bodies and challenge functions, including CFD analysts. The audit found two projects where the IRPDA provided the same recommendations at a subsequent engagement, as the previously identified issues had not been addressed. This led to additional engagements with the IRPDA. Instances where other stakeholder feedback was not incorporated in the final draft of key project documents were also noted. Demonstrating how feedback is taken into consideration is important for improving the quality of information and for the continuous improvement of projects department wide.

2.2.4 Conclusion

Oversight is essential to keeping all process stakeholders informed of and accountable for preliminary requirement development and the attainment of project objectives. When oversight is insufficient or inconsistently applied and documented, important issues and risks may not be identified or addressed in a timely manner.

While oversight to ensure that project requirements are aligned with HLMRs is occurring, the frequency of the oversight, the documenting of oversight activities and monitoring of oversight feedback is inconsistent. Added mechanisms are needed to monitor key project milestones and oversight feedback to support the continuous improvement of the preliminary requirement development process and the attainment of project objectives.