Audit of Preliminary Requirement Development Process for Capital Equipment Projects

ADM(Mat)
Assistant Deputy Minister (Materiel)
ADM(RS)
Assistant Deputy Minister (Review Services)
BCA
Business Case Analysis
CAF
Canadian Armed Forces
CFD
Chief of Force Development
CID
Capability Investment Database
C Prog
Chief of Programme
DCB
Defence Capability Board
DM
Deputy Minister
DND
Department of National Defence
DRMIS  
 Defence Resource Management Information System
ECS
Environmental Chief of Staff
HLMR 
High Level Mandatory Requirement
ID   
Identification
IPV 
Integrated Program View
IRPDA 
Independent Review Panel for Defence Acquisition
MND
Minister of National Defence
OA
Options Analysis
OPI
Office of Primary Interest
PAD
Project Approval Directive
PAP
Project Approval Process
PCRA
Project Complexity Risk Assessment
PD
Project Director
PGM 1-17
Project Guidance Memorandum 1-17
PL
Project Leader
PM
Project Manager
RCAF
Royal Canadian Air Force
RCN
Royal Canadian Navy
ROD
Record of Decision
SCD
Strategic Context Document
SOR
Statement of Requirements
SRB
Senior Review Board
SSE
Canada's defence policy: Strong, Secured, Engaged (2017)
VCDS
Vice Chief of the Defence Staff

Overall Assessment

DND has a process to develop preliminary requirements for capital equipment projects that supports the delivery of defence capability. Key improvements in governance, oversight and capacity are required to increase the efficiency of this process.

Figure 1: Process flowchart of the Phases of Preliminary Requirement Development Process
Figure 1. Phases of Preliminary Requirement Development Process. This figure illustrates the key requirement documentsFootnote 1  in the associated project phase. Further information for the requirement development process is illustrated in Annex C—PAP Flowchart (*approximate timelines).

Figure 1 Details - Phases of Preliminary Requirement Development Process.

1.1.2 Key Stakeholders

Many participants from project teams to oversight bodies are involved in the PAP process. Their responsibilities related to preliminary requirement development are outlined and illustrated in Figure 2.

Vice Chief of the Defence Staff

As part of its mission to ensure that Departmental defence policy and strategic objectives are achieved, the Vice Chief of the Defence Staff (VCDS) is responsible for providing guidance and coordinating the PAP. Since 2013, PAP responsibilities have been divided between the Chief of Force Development (CFD) and the Chief of Programme (C Prog).

Project Sponsor

The project sponsor for capital equipment projects, usually one of the three environments (CA, RCAF, RCN), is responsible for identifying an operational capability deficiency and for communicating the value and importance of a project to the Department. As well, the project sponsor leads the project through the ID and OA phases. The project sponsor team is comprised of:

Project Implementer

The project implementer takes over the project leadership from the project sponsor at the end of the OA phase, when project funding is approved, and commits to delivering the capability on schedule and within budget. The project implementer for capital equipment projects is typically the Assistant Deputy Minister (Materiel) (ADM(Mat)), and has the following team members:

Figure 2: Chart of PAP Key Stakeholders
Figure 2. PAP Key Stakeholders. This figure illustrates the key stakeholders and their accountabilities throughout the life-cycle of the project.
Figure 2 Details - PAP Key Stakeholders.

1.1.3 Oversight bodies and Challenge Function

Senior Review Board

The Senior Review Board (SRB) is a project-specific body established at the beginning of the OA phase to support management oversight of the project. It is chaired by the PL representing the project sponsor or the project implementer, depending on the project phase. SRB meetings are to be convened, at a minimum, on an annual basis to assess formal progress and project performance against approved baseline performance measures of scope, cost and schedule.Footnote 2  Key project documents, such as the Project Charter, the SCD and Preliminary SOR require SRB endorsement. Minimum membership includes representation from the VCDS, Assistant Deputy Minister (Finance), Commander Navy, Commander Army, Commander Air Force and others as required by the Project Charter.

Chief of Force Development Analysts

At the departmental level, CFD analysts serve as the first challenge function in the ID and OA phases. CFD analysts review key project documents, such as the SCD and BCA, before they are presented to the Defence Capability Board (DCB) to confirm capability needs are aligned with strategic outcomes, government policy and the Capability-Based Plan. Additionally, analysts coordinate project documentation for the DCB and the Independent Review Panel for Defence Acquisition (IRPDA). Analysts also support project sponsors in responding to oversight-body feedback, questions and recommendations.

Defence Capability Board

Chaired by the VCDS, the DCB serves as a challenge function to ensure the alignment of future capabilities against departmental needs, comprehensive options development and analysis, and transparency and detail of project costing figures.Footnote 3  Capital equipment projects are required to seek DCB approval twice during the preliminary requirement development process. DCB 1 takes place in the ID phase to approve a project's capability requirements, including the HLMRs, and viable options to be assessed. DCB 2 reviews the detailed analysis of options in the BCA and approves the preferred option with which to move forward. DCB attendance is limited to the L1 or L1A level, which includes the project sponsor, the project implementer, CFD and C Prog.

Independent Review Panel for Defence Acquisition

As a core element of the Defence Procurement Strategy, the IRPDA provides independent advice to the Minister of National Defence (MND) through the Deputy Minister (DM) for major defence projects exceeding $100 million or exceeding the Minister's delegated approval authority, as well as those identified for approval. Specifically, the IRPDA assesses the validity of the capability gap and the HLMRs, whether requirements are stated in a clear and appropriate manner, and provides its advice prior to Ministerial or Treasury Board approval.

IRPDA typically reviews each capital equipment project twice. The first review, IRP 1, which takes place at the end of the ID phase, focuses on each project's capability gaps and HLMRs after their endorsement by DCB 1. The second review, IRP 2, which occurs at the end of the OA phase, examines the results of the options analysis and the linkages between the HLMRs and the Preliminary SOR, following endorsement by DCB 2. Supplementary engagements may occur as necessary to address issues identified during either IRP 1 or 2. The project sponsor, CFD and the project implementer are required to attend IRPDA engagements. The Chair of the IRPDA may invite other DND/CAF specialists to participate if necessary.

1.2 Rationale

Delivering the right equipment to the CAF in a timely manner is a top priority for the Department, especially with the increase in long-term investments in military capability and new initiatives as described in SSE. SSE notes that capability requirements have not always been clearly communicated to industry and Canadians, and the desire to reduce project development and approval times by 50 percent.Footnote4  An effective and efficient project preliminary requirement development process will enable the Department to clearly define, communicate and deliver its capability requirements in a timely manner. Given the importance of this subject, the audit was included in the ADM(RS) Risk based Audit Plan for fiscal years 2016/17 to 2018/19.

1.3 Objective

The objective of this audit is to determine if DND/CAF has an adequate preliminary requirement development process for capital equipment projects to support the delivery of defence capabilities.

1.4 Scope

The scope of this audit included the following:

The audit did not examine capital equipment projects that were equal to or below $5 million as they are subject to a different approval process nor IM/IT projects and infrastructure projects nor the validity of specific requirements.

Audit work was performed during the period from May 2017 to April 2018, and included oversight activities and documentation related to initiation of selected projects from 2007 to April 2018.

1.5 Methodology

The audit results are based on evidence from the following sources:

1.6 Audit Criteria

The audit criteria can also be found at Annex B.

1.7 Statement of Conformance

The audit findings and conclusions contained in this report are based on sufficient and appropriate audit evidence gathered in accordance with procedures that meet the Institute of Internal Auditors' International Standards for the Professional Practice of Internal Auditing. The audit thus conforms to the Institute of Internal Auditors' International Standards for the Professional Practice of Internal Auditing as supported by the results of the quality assurance and improvement program. The opinions expressed in this report are based on conditions as they existed at the time of the audit and apply only to the entity examined.


Back to Table of Contents

2.0 Findings and Recommendations

2.1 Governance

We expected to find that:

Key Project Documents

Strategic Context Document (SCD): description of the strategic environment, recent changes, linkages to key strategic documents and initial analysis of options. The SCD also includes the HLMRs, which are the capabilities required to meet the business outcomes; meant to provide a high level and non-specialist expression of project objectives.

Statement of Requirements (SOR): the operational requirements developed throughout OA and Definition phases and beyond, that will be delivered during Implementation phase.

Business Case Analysis (BCA): detailed options, recommended option, supporting analysis and extracts from initial project management plan.

2.1.1 Policy and Guidance

The main policy governing the PAP is the Project Approval Directive (PAD). Last updated in 2015, the PAD is accompanied by a Project Guidance Memorandum (PGM) 1-17, which incorporates supplementary information and updated guidance for the ID and OA phases. CFD has also created an Aide Memoire which serves to guide analysts who review key project documents. Templates for key documents, such as the BCA, are also available. As well, the VCDS Directive on Strategic Governance Board and Committees provides guidance on briefing packages that are submitted to oversight bodies. Although considerable guidance exists, a review of applicable policy and guidance identified areas that require greater attention to ensure key requirement documents are developed in a consistent and efficient manner.

2.1.2 Key Project Documents

Multiple sources of templates and guides were found across several key project documents, which resulted in documentation discrepancies in the projects reviewed. For example, the SORs in the sample projects varied significantly in how the linkages between HLMRs and operational requirements were demonstrated. The available SOR guide was last updated in 2006 and, therefore, does not meet the current PGM 1‑17 requirement of linking the operational requirements to their corresponding HLMRs. This linkage of operational requirements to HLMRs is important to ensure clear expectations and capability needs are defined. The preparation of consistent and quality preliminary requirement documents that ensure alignment of HLMRs and Preliminary SOR would help expedite project timelines and approvals.

Despite the inconsistency in how the requirements were presented, the review of oversight documentation (e.g., CFD, IRPDA) found that preliminary requirements were aligned with the HLMRs for all projects within the sample.

The review of the key project documents showed that similar information is presented in multiple documents. Documents reviewed, including the BCA, Preliminary SOR and Project Charter, contain common elements such as background, milestones and project interdependencies. This information is presented in four key documents which are reviewed and/or endorsed multiple times, as illustrated in Table 1. As such, project documentation includes inconsistent information due to ongoing updates and revisions, which leads to inefficiencies in the documentation development and review.

The review also showed the Project Charter was rarely endorsed in a timely manner, or at all by the SRB, which may indicate the need to review its value added. As illustrated in Table 1, additional opportunities to streamline documentation and gain efficiencies may exist.

Table 1. Key Project Documents – Review and Approval Authorities and Document Content.
Requirements Documents Review / Approval Authority Information Elements within each Document

-

Project Sponsor

SRB

CFD

DCB

IRPDA

Accountability, Responsibility, Authority

Project Background

HLMRs

Evaluation Criteria

Operational Requirements

Cost

Options

Milestones

Project Risks & Mitigation

Capability Deficiency

Project Inter- dependencies

SCD

-

-

-

-

-

-

-

-

BCA

-

-

Preliminary
SOR

*

-

-

-

-

-

Project Charter

-

-

-

-

-

-

-

-

-

Table 1. Key Project Documents – Review and Approval Authorities and Document Content. This table illustrates the review/approval authorities for each document, as well as the information presented within the documents. (For the Review/Authority columns, the ✓ demonstrates review/approval required by; the * indicates review at the discretion of CFD; the blank space denotes not applicable. For the Information Elements columns, the ✓ illustrates the information contained in each document; the blank space denotes not applicable.)

Table 1 Details - Summary of Key Project Documents – Review and Approval Authorities and Document Content.

2.1.3 CFD Analyst Review

Key documents submitted for DCB endorsement and IRPDA review are first provided to CFD analysts who perform a challenge function. The direction provided by the PAD, PGM and CFD's Aide Memoire on which documents should be reviewed by the CFD analyst is unclear. For instance, the PGM states the HLMRs must be reviewed, but the Aide Memoire does not. Additionally, neither document provides sufficient information regarding the roles and responsibilities of the analysts. These discrepancies in guidance cause confusion for CFD analysts as well as project staff regarding the requirement to engage CFD analysts and the expectations regarding project documents.

2.1.4 Roles and Responsibilities

The DCB and IRPDA each have Terms of Reference. The PAD states that the DCB is the approval body in the ID and OA phases and has the authority to direct projects to proceed to the next phase. The IRPDA is a third-party advisory body that provides independent advice to the MND before projects seek Ministerial or Treasury Board approval. In doing so, the IRPDA engages the projects after DCB approval to provide a challenge function. During the IRPDA review process, the Panel also provides recommendations and suggestions to help project sponsors better prepare for Treasury Board or Ministerial submissions.

As previously noted, IRPDA typically reviews each capital equipment project twice. Four projects reviewed were subject to IRPDA review, two of which received DCB approval to proceed to Definition phase with a pending IRPDA engagementFootnote 5  and recommendations. As IRPDA is an advisory body which performs its review after DCB approval is obtained, it is unclear to some project sponsors whether projects need to address the recommended changes to key documents before proceeding to the Definition phase or whether obtaining DCB approval is sufficient to proceed. This lack of clarity may result in projects being submitted for approval with unresolved issues or outstanding IRPDA recommendations. Interviews with project sponsor teams revealed differing views on how projects should manage IRPDA engagements and feedback. Some teams treated IRPDA feedback as optional. For example, the audit found two projects where the IRPDA provided the same recommendations at a subsequent engagement, as the previously identified issues had not been addressed. Others viewed IRPDA as an approval gate, as it submits its advice to the MND through the DM, and as such addressed and incorporated, as applicable, all IRPDA recommendations before seeking approval.

2.1.5 Conclusion

Although policy and guidance to support the preliminary requirement development process is in place, certain areas require clarification and updating to support a more efficient process. Templates for key project documents are not aligned with the new process requirements. Roles and responsibilities of internal governance bodies and external challenge functions are unclear and insufficient to guide project staff when engaging these bodies. There is also a lack of consolidated reference materials, which resulted in project documents that are not aligned with policy requirements or expectations of oversight bodies and challenge functions.
The afore-mentioned inefficiencies may result in a duplication of effort and prolonged approval process. However, the inefficiencies could not be quantified since data to support such analysis was not available.

Updated, streamlined and centralized guidance materials are needed to provide project teams and review analysts with the support they require to prepare key requirements documents in a consistent and efficient manner. Additionally, clarifying the roles and responsibilities of the oversight bodies and challenge functions would ensure key documents are developed in accordance with current requirements and expectations. Furthermore, greater clarity on the process and expectations of DCB and IRPDA would allow projects to move forward with submissions to the MND or Treasury Board in an efficient manner.

ADM(RS) Recommendation

2.2 Oversight

We expected to find that:

Figure 3: Process flowchart of Phases of Preliminary Requirement Development Process
Figure 3. Phases of Preliminary Requirement Development Process. This figure illustrates where the oversight/challenge functions occur during the preliminary requirement development process.
Figure 3 Details - Phases of Preliminary Requirement Development Process

2.2.1 Senior Review Board

In accordance with policy, project SRBs are to convene, at a minimum, on an annual basis to oversee project development and management once the project enters the OA phase, as shown in Figure 3.Footnote 6  With one exception, each project reviewed missed at least one annual SRB. Overall, for projects examined, almost a quarter of required SRBs did not occur as required.

SRBs serve as a valuable platform for project teams to provide key updates and seek assistance on known issues. For instance, one sampled project approached SRB for direction and support in writing the preliminary SOR. Missed SRBs can amplify project challenges or prolong the approval process. For example, another sampled project reported to SRB that its Preliminary SOR was 95 percent complete. However, documentation showed the subsequent SRB was missed, which resulted in the Preliminary SOR approval being delayed by a year. Interviews indicated that missed SRBs were mostly due to scheduling conflicts or PL availability. PL availability is discussed in section 2.3.1. Documenting these rationales would provide information and context for new and existing project team members, to support effective project management.

SRB engagements require a presentation briefing deck to provide key updates on the project. The briefing deck and meeting records of decision (ROD) should provide decision makers with a comprehensive understanding of the work completed thus far, the current status and challenges, and the way forward. It is important that RODs are approved in a timely manner to support oversight bodies' review of the preliminary requirements and HLMRs and thus, allow projects to move to the next step in the PAP.

2.2.2 Defence Capability Board

DCB serves as a project approval authority for project capabilities in the ID and OA phases. When engaging DCB, project teams submit briefing packages which include a presentation deck and the SCD for DCB1 and the BCA for DCB2. The VCDS Directive on Strategic Governance Board and Committees requires that project presentations include a recap of previous direction that summarizes feedback from other oversight bodies and previous DCB engagements as well as highlight any action items or whether recommendations had been addressed. Of the five presentations reviewed, three contained the required recap.
The DCB RODs reviewed for the projects were concise and provided little information. Interviews with project sponsor teams confirmed the majority of the challenge or oversight is conducted prior to DCB engagement. While project staff indicated significant discussions occur prior to the formal DCB meeting, including numerous pre-briefs to key stakeholders, there is no practice in place to document, follow-up or provide a recap on the feedback, issues or decisions that occurred in those pre-discussions. As such, the challenge function performed by DCB, including its review of preliminary requirements and HLMRs, is not recorded to support actions taken and decisions made, or support decision making at subsequent oversight engagements.

Noteworthy Practice

During the course of the audit, VCDS implemented the requirement to include a summary recap of key feedback received prior to a project's presentation at DCB. This measure should serve to support the tracking of key project information.

A review of IRPDA feedback provided to the sampled projects revealed instances where issues, which fall within the review scope of the DCB, were first documented by the IRPDA. One such example was the questioning of the range of options developed and the method and rationale used to select the preferred option. These questions may have been raised by the DCB prior to its formal IRPDA engagement and subsequently addressed by the project team, however, the DCB RODs do not demonstrate how or what level of oversight was applied to the project. As such, a lack of documentation to support the oversight challenge undertaken by DCB or other stakeholders may result in the IRPDA offering more extensive comments and recommendations.

2.2.3 Chief of Force Development

In coordinating the internal and external oversight bodies and providing oversight through the review of key projects documents, CFD uses two main project management tracking tools: a Director Capability Integration tracker and an alignment review table. These tools are both internal to CFD with one serving as a project-level tracker while the latter serves as a management tool to highlight upcoming DCB engagements and projects entering Definition phase to help better prioritize projects. While both tools are used for different purposes, the information contained within is not consistent. As well, there is no central tracking tool used to record and track governance body engagement dates, document approval dates and the status of projects to ensure project schedules are being met.

The Capability Investment Database (CID) is currently the main departmental information system to provide visibility on the investment portfolio. As found in previous internal audits, CID information is often outdated or misaligned with what is found in key project documents. The inconsistent data affects the reliability of the information provided by the CID and requires additional validation. DND/CAF is transitioning towards the new project information repository, Defence Resource Management Information System (DRMIS) to consolidate capital project information. There is a risk that current documentation practices and information quality issues could affect the future effectiveness of DRMIS or other modern information management tools, as the data in the CID is transferred.

Noteworthy Practice

The Project Director for one project created an Excel spreadsheet to record IRPDA comments and recommendations, as well as the project team's disposition of these items.

There is also no mechanism in place to document and follow up on feedback and recommendations provided by oversight bodies and challenge functions, including CFD analysts. The audit found two projects where the IRPDA provided the same recommendations at a subsequent engagement, as the previously identified issues had not been addressed. This led to additional engagements with the IRPDA. Instances where other stakeholder feedback was not incorporated in the final draft of key project documents were also noted. Demonstrating how feedback is taken into consideration is important for improving the quality of information and for the continuous improvement of projects department wide.

2.2.4 Conclusion

Oversight is essential to keeping all process stakeholders informed of and accountable for preliminary requirement development and the attainment of project objectives. When oversight is insufficient or inconsistently applied and documented, important issues and risks may not be identified or addressed in a timely manner.

While oversight to ensure that project requirements are aligned with HLMRs is occurring, the frequency of the oversight, the documenting of oversight activities and monitoring of oversight feedback is inconsistent. Added mechanisms are needed to monitor key project milestones and oversight feedback to support the continuous improvement of the preliminary requirement development process and the attainment of project objectives.


Back to Table of Contents

ADM(RS) Recommendation

2.3 Capacity

We expected to find that:

2.3.1 Project Team

Project teams are established at the beginning of the OA phase. As previously described in 1.1.2, each team consists of the project sponsor's PL and PD as well as the project implementer's PL and PM. While PLs are accountable for the overall project performance, the PD and the PM are responsible for requirements development. The PD ensures project objectives are established early, linked to a preliminary operational requirement and are maintained through to project completion. The PM manages the project while working with the PD to ensure project activities and milestones are achieved. Projects are also assigned a CFD review analyst for the ID and OA phases. CFD analysts provide a review and challenge function to projects prior to DCB and IRPDA engagements and also support PDs during the development of project requirements as necessary.

2.3.2 Turnover

As PDs are generally military members and subject to military postings, staff turnover is a known challenge. For the seven projects reviewed, continuity of PD staff was an important issue:

Though each environment faces similar effects of turnover, such as the continuous cycle of training first-time PDs, the environments have taken ad hoc succession planning approaches to increase the stability of the PD role. For example, one environment has hired consultants while another environment is exploring the addition of civilian positions to its organizational chart.
Similar to PDs, CFD analysts are subject to military postings. While data was not available to calculate the turnover rate within these positions, interviews indicated a similar turnover issue exists at the review analyst level. This impacts project continuity and consistency.

2.3.3 Project Leader Rank

In accordance with the PAD, PLs are supposed to be Level 1 Managers, but this role can be delegated depending on the Project Complexity and Risk Assessment (PCRA) level, as shown in Figure 4. Of the seven sample projects reviewed, one PL rank fell below the minimum level required and five PLs were ranked above the level required. There are risks associated with both of these situations: the former may not have sufficient experience or influence to provide the correct level of oversight as per the PCRA; the latter may cause project process delays due to competing demands and priorities, as reported in section 2.2.1.

Figure 4: Hierarchical triangle of Project Leader Rank
Figure 4. Project Leader Rank. This figure illustrates the required project leader rank delegation based on PCRA level.
Figure 4 Details - Project Leader Rank.

2.3.4 Project Management Experience and Support

To enable the development of project operational requirements, PDs are selected from the operational field based on their technical expertise. While PDs have the necessary technical expertise, few have project management experience such as managing schedules, performing cost estimates and determining project risks. While not reflected in current policy, one significant change to the PAP, resulting from a past Office of the Auditor General recommendation, is the practice of providing greater rigour in cost estimates in the OA phase rather than in the Definition phase. This change requires earlier engagement with industry to determine cost and availability, and also requires greater and earlier PM support to ensure preliminary requirements are developed effectively and efficiently.

PMs are assigned to project sponsor teams in the OA phase as per policy,Footnote 7  though they are typically assigned to multiple projects. A full-time PM is not assigned to a project until the end of the OA phase. This delayed PM support may affect the quality of deliverables, especially as many cost estimate tasks, which were once a part of the Definition phase, are now occurring in the OA phase. The allocation of PM resources earlier in the OA phase would contribute to the department's ability to deliver the right equipment in a timely manner.

2.3.5 Training and Professional Development

Noteworthy Practice

One environment's Strategic Plan includes the commitment to invest in the development and professionalization of force development personnel. This commitment is monitored by and reported to senior management.

PDs are provided with fundamental training, including DND's Project Approval Course and Best Practices for Project Management. While the audit did not examine the adequacy of the training offered, interviews indicated the available training is not sufficient and course material is often outdated.

While all three environments offer a one-year training program, it is not available to all staff nor is it standardized for PDs. As the average tenure for a PD is 17 months, there may be a limited return on this one-year investment at the PD level. Similarly, CFD analysts do not have a formal training plan related to their duties as a challenge function.

Without sufficient training, there is an increased risk that PDs and CFD analysts lack the skills and knowledge required to respectively develop key project documents to navigate through the PAP or to execute their oversight and challenge responsibilities related to the preliminary requirement development process.

2.3.6 Conclusion

Project team capacity is vital to ensuring adequate and timely development of preliminary requirements within the PAP. Staff turnover, the lack of timely project management support and insufficient training may contribute to project delays and a reduced quality of project deliverables. Investments in the staffing and resourcing of projects teams, including training needs, are required to ensure the department effectively resources its capital equipment projects and meets its capability needs.

ADM(RS) Recommendation


Back to Table of Contents

3.0 General Conclusion

Overall, the audit found processes and procedures are in place to support preliminary requirement development for capital equipment projects. Key improvements are required to ensure the process is efficient and streamlined.

Clarifying roles and responsibilities of key stakeholders, as well as centralizing, simplifying and streamlining guidance and tools will further help project staff develop key documents and prepare for oversight engagements, thereby improving the efficiency of the preliminary requirement development process.

Project requirements are aligned with HLMRs. Added mechanisms are needed to monitor key project milestones and oversight feedback to support the continuous improvement of the preliminary requirement development process, information for decision making and the attainment of project objectives.

The capacity of project teams responsible for the preliminary requirement development process should be enhanced. Training and resourcing strategies should be employed to support project teams and to professionalize force development personnel. Additionally, project management expertise provided to project sponsors earlier in the process would be beneficial.

Under SSE, there is a significant increase in capital investment over the next 20 years, to ensure DND/CAF meets its capability needs. An efficient preliminary requirement development process for capital equipment projects is critical to meeting these commitments. To make this happen, all players must understand their roles and responsibilities, and have the ability to provide needed information on project risks, status and options to decision makers.


Back to Table of Contents

Annex A—Management Action Plan

ADM(RS) uses recommendation significance criteria as follows:

Very High—Controls are not in place. Important issues have been identified and will have a significant negative impact on operations.
High—Controls are inadequate. Important issues are identified that could negatively impact the achievement of program/operational objectives.
Moderate—Controls are in place but are not being sufficiently complied with. Issues are identified that could negatively impact the efficiency and effectiveness of operations.
Low—Controls are in place but the level of compliance varies.
Very Low—Controls are in place with no level of variance.

Governance

ADM(RS) Recommendation (Moderate)

Management Action

Action 1 a. C Prog is leading an update of the PAD. Updating of the PAD will include some initiatives to streamline the process but is primarily focused on ensuring directions reflect existing best practices. Upon promulgation of the updated PAD, CFD will lead a review of project documentation, including relevant templates, to identify efficiencies that can be incorporated into project documentation and possibly identify and eliminate redundant document/process demands. This will be an ongoing process but the target is to complete the initial review and establish an implementation plan by December 2020.

b. C Prog is leading an update of the PAD that will be available on the DWAN. Once completed it will provide a consolidated, user-friendly portal that will employ contemporary web tools to ensure easy user access to project approval process governance, documentation and templates. The updated PAD will be a living document with C Prog responsible for upkeep. The target for implementing the updated PAD is December 2019.

c.  C Prog is leading an update of the PAD. Once completed the updated PAD will provide updated guidance and clarify the roles and responsibilities of all key stakeholders and oversight bodies. Target date for completion is December 2019.

OPI: VCDS
Target Date: December 2020

Oversight

ADM(RS) Recommendation (High)

Management Action

Action 2 a. In March 2018, VCDS began the implementation of a requirement to include in the DCB Records of Decision a summary of key challenge interactions that occurred prior to a project's engagement with DCB. 

b. CFD will review the various ad hoc solutions used to manage and track CFD, DCB and IRPDA recommendations and advice. CFD will implement a standardized approach to document the disposition of stakeholder and oversight advice and recommendations. Target date for completion is December 2019.

OPI: VCDS
Target Date: December 2019

ADM(RS) Recommendation (Moderate)

Management Action

On July 11, 2018, the VCDS, Senior Associate DM and CFO signed the Joint Master Implementation Plan to migrate the CID to manage the departmental capital programme using the Defence Resources Management Information System (DRMIS). In accordance with this directive, DND/CAF will transition the CID to DRMIS through a phased approach, and use DRMIS as the primary means of collecting data on the Capital program. The Department will transition, in phases, to a single common database (DRIMS), commencing with the migration of SSE Capital Projects.

Phase 1. 90 SSE Projects were created in DRMIS and a dedicated team within C Prog was created to oversee and coordinate the adoption of DRMIS as the primary source of data for the Department's Capital Program. As part of this phase, C Prog is undertaking an extensive clean up of CID data.

Phase 2. To be completed by March 2020. This phase will increase the Department's ability to forecast resource requirements and/or re-allocations, monitor the Capital Program (schedule/finance linkage) and create a sustainable program. This phase consists of capturing the entire DND Program in DRMIS.  This includes the 333 projects in the Departmental Investment Plan and 200 projects that do not yet have an identified source of funds across all phases from Identification to Close-Out. As part of this phase, a second clean up of CID data will be performed to ensure accurate data is migrated.

OPI: VCDS
Target Date: March 2020

Capacity

ADM(RS) Recommendation (High)

Management Action

Action 4 a. CFD will work to initiate a training needs analysis of the capability development process. 

  1. The analysis will focus on the knowledge and skills necessary to initiate projects, and progress them through the Identification and Options Analysis phases of the Project Approval Process. The analysis will focus on force development staff in each project sponsor and in CFD. The analysis will also focus on specific succession planning strategies to enable an increasingly professional force development staff at all rank levels and in civilian employees. The target is to have the training needs analysis initiated by December 2019 and analysis complete by December 2020.
  2. In the interim, CFD will work with project sponsors to offer improved orientation regarding the force development process. CFD will also seek opportunities to provide additional professional development to force development staff. Target Date is December 2019.

b & c. Canada's new defence policy, SSE, commits to improving defence procurement. Developing resourcing options will be a multi-year endeavour as follows:

  1. SSE initiative 98 commits to growing and professionalizing the defence procurement workforce. Strategies and implementation plans developed in response to initiative 98 will shape the CAF response to this recommendation. Workforce growth will be reflected in the Defence Team Establishment Plan starting in 2019.
  2. In the near term, CFD will work with MPC and trade advisors to identify preferred candidates for employment in capability development and aim to minimize the impact of military turnover. In addition, CFD has initiated staffing to create civilian force development positions in order to enhance the professionalization of the capability development staff and mitigate the impact of high military turnover. Target Date for completion is April 2019.

OPI: VCDS
Target Date: December 2020

ADM(RS) Recommendation (High)

Management Action

ADM(Mat) will continue to develop its Capital Projects' Integrated Program View (IPV) in consultation with the environments and applicable project sponsors, allowing better alignment of PM resources and identification of gaps. The IPV will also provide a forward looking schedule of all projects expected to transition to ADM(Mat) for implementation in the coming decade. This documented earlier insight will allow the Materiel Group to request additional resources for the OA phase from the project sponsor through the Business Planning process. Overall, this will ensure sufficient and consistent PM support is provided earlier in the OA phase pending approved funding from project sponsors.

Closure: This MAP will be considered closed when the IPV is approved by ADM(Mat) and shared within the group.

OPI: ADM(Mat)
Target Date: December 2019


Back to Table of Contents

Annex B—Audit Criteria

Criteria Assessment

The audit criteria were assessed using the following levels:

Assessment Level and Description

Level 1—Satisfactory
Level 2—Needs Minor Improvement
Level 3—Needs Moderate Improvement
Level 4—Needs Significant Improvement
Level 5—Unsatisfactory

Governance

  1. Appropriate processes and procedures are in place to support the preliminary requirement development process for capital equipment projects.

Assessment Level [Level 3] – Although processes and procedures are in place, there is no standardized, up-to-date guidance to sufficiently support the development of preliminary requirements. Some roles and responsibilities of stakeholders are not clearly defined.

Oversight

  1. Capital equipment preliminary requirements align with high-level mandatory requirements and are developed in accordance with DND policy.

Assessment Level [Level 4] – Preliminary requirements align with high-level mandatory requirements. Some oversight to support this process was not sufficiently documented or occurring as required. Centralizing and monitoring project milestones and documentation are required to provide information for decision making and ensure the efficient management of capital equipment projects.

Capacity

  1. Sufficient capability exists to support the preliminary requirement development process (resources, capacity, skills, etc.).

Assessment Level [Level 4] – Project teams require additional training, resources and guidance to efficiently support the preliminary requirement development process.

Sources of Criteria

Governance:

Oversight:

Capacity:


Back to Table of Contents

Annex C—Project Approval Process Flowchart

The project approval process contains five phases: ID, OA, Definition, Implementation and Close-Out. Only the ID and OA phases fell under this Audit's scope. Therefore, details for the other phases were not included in the following flowchart.

Figure C-1: Flowchart organized by project phase, major stakeholders and engagements
Figure C-1. Project Approval Flowchart. This figure is a flowchart organized by project phase, and major stakeholders and engagements. The chart flows with the development, review and approval of required documents by major stakeholders. The (*) only applies to projects subject to IRPDA review. The (**) only applies to projects where CFD has directed it to present the Preliminary SOR to DCB.
Figure C-1 Details -Project Approval Flowchart.


Back to Table of Contents

Annex D—Sample Projects

The sample included a total of seven projects from the three environments: Army, Navy and Air Force as shown in Table D-1. It includes five projects from the list of high-risk capital equipment projects from the Analysis of Capital Equipment and Information Management Projects in November 2015 and two projects provided by ADM(Mat). The sample selection also considered the materiality of the projects and the project phase at the time it was selected.

Table D-1. Sample Projects.

Environment Project ID Project Name Project Phase (at the beginning of audit)

Army

C.003035

CF Land Electronic Warfare Modernization (CFLEWM)

OA

C.002519**

Common Heavy Equipment Replacement (CHER)

OA

C.002514

High Risk Search Capability (HRSC)

Definition*

Navy

C.001351

Underwater Warfare Suite Upgrade (UWSU)

Definition*

C.001339**

Naval Large Tug Project (NLT)

OA*

Air Force

C.000032

Tactical Integrated Command, Control and Communications (TIC3)

OA

C.003146

Griffon Limited Life Extension (GLLE)

OA

Table D-1. Sample Projects. This table shows the seven projects included in the audit sample. The (*) indicates that the projects have since proceeded to next phase. (**) indicates the project sample provided by ADM(Mat).

Table D-1 Details - Sample Projects.


Back to Table of Contents

Page details

Date modified: