Audit of Defence Project Management

November 2022

1259-3-0068 ADM(RS)

Reviewed by ADM(RS) in accordance with the Access to Information Act.  Information UNCLASSIFIED.

Acronyms

Findings and Recommendations

Context

Project Management Within DND/CAF

As described in the PAD, project management is the systematic planning, organizing and monitoring of allocated resources to accomplish identified project objectives and outcomes. Project management is a critical component of maintaining operational capability and managing critical technology, infrastructure and equipment within DND/CAF.

The DSP is defined as the unified architecture of all DND-approved services, activities, projects, programmes and portfolios deemed to be essential to the delivery of affordable and effective Defence services to the Government and Canadians. DND/CAF is progressing in its efforts to fulfil the defence mandate, as articulated in the 2017 Defence Policy, SSE, while concurrently in the midst of the largest recapitalization effort in over forty years. This significant investment in the operational capability of the CAF brings both challenges and opportunities for departmental project management. Since SSE was launched, 130 projects have reached closeout and/or have been completed, including seven projects valued over $1B with a total project approval amount of $14.5B. The DSP is currently tracking 342 capital projects.

Major projects are defined as projects for the one-time acquisition of new equipment, materiel and/or services where the total project value inclusive of tax equals or exceeds $10M. Major projects, as shown in Figure 1, have five stages within the project lifecycle: Identification, Options Analysis, Definition, Implementation, and Closeout. Projects are led by the project sponsor, who have an identified operational need, for the first two phases, and by the project implementers: Assistant Deputy Minister (Infrastructure and Environment) (ADM(IE)); Assistant Deputy Minister (Information Management) (ADM(IM)); and Assistant Deputy Minister (Materiel) (ADM(Mat)), who deliver the project to completion, for the last three phases. For the distribution of projects across implementers, see Figure 2. This audit focused on the latter three phases of the project lifecycle.

Figure 2 Summary

Project Governance

Governance is the mechanism used by senior management to oversee the achievement of program outcomes in accordance with Treasury Board (TB) Policy and as prescribed in the Financial Administration Act. Key principles for effective governance include defining accountabilities, responsibilities and authorities (ARA), providing oversight and monitoring of risks, deliverables, challenge regarding capability ladder decisions, and insights/advice to mitigate risks. There are three levels of governance: government-wide; departmental; and project- level.

SRBs are critical to project governance. Their role is to enable corporate challenge and oversight in support of the Defence Capabilities Board (DCB) and PMB. SRBs also oversee project development and provide management advice to the Project Leader as well as project risk and project performance management oversight throughout the project life cycle. SRBs are one of many governance boards and committees which support projects and provide departmental oversight. For more details on defence governance boards and committees, see Annex D.

The PAD

The PAD contains departmental direction and guidance to ensure that the programme is delivered in a manner consistent with higher level policy and guidance. Part one lays out the expectations of leaders and managers in the delivery of capabilities articulated in the Defence Policy. Part two provides a comprehensive, step-by-step guide to project and programme management within DND, tailored to both the type and complexity of projects. The PAD conforms to recently published TB Policies: the Policy on the Planning and Management of Investments and the Directive on the Management of Projects and Programmes. The PAD establishes new streamlined project approval paths that align governance, documentation, process and delegations in accordance with the level of project complexity and risk.

Resourcing

Resourcing is a challenge for each L1, with competing demands for both financial and human resources across projects and DND/CAF. L1s and project teams are working to deliver key tasks with existing resources, and mitigate any resource shortages through prioritization of personnel, prioritizing tasks and business planning. Resourcing challenges are amplified given labour shortages, military postings, staff turnovers, and due to unique knowledge and security requirements for complex projects.

Figure 3 Summary

Key Stakeholders’ Roles and Responsibilities^{Footnote 1}

Key Themes

The Key Findings were aligned into two themes as follows:

1. Project Management Framework
2. Data Management

Project Management Framework

Why It Matters

Without a centralized approach to interdependency management, there is a risk that project changes to schedule or scope will not be tracked and communicated to all relevant stakeholders. This can impact the delivery of related projects and project elements, and can increase the risk to delivery of key capabilities. Interdependencies can result in a number of downstream impacts, with the most immediate being the project-level impact to cost and schedule. DND/CAF relies on internal governance at the L3, L2 and L1 levels to remedy and help aid projects in mitigating project risks. High-priority projects may benefit from a mechanism to allow current issues and risks to be briefed to higher level governance for advice, direction and information-sharing purposes.

What We Found

Project Interdependencies are factors required for the successful delivery of an individual project. Interdependencies can be either dependencies or constraints. Dependencies are tasks and events that occur outside the Project Team’s control that must happen before the project can progress. Dependencies can include deliverables from other projects or other stakeholders. For example, shared infrastructure between two projects that may delay a future project would be a dependency. Constraints are factors that must be accounted for when planning the project. For example, a project constraint may be a shared pool of funding for a project's equipment and infrastructure that will not be sufficient for all project requirements.

Project Interdependencies

Project interdependencies can have a substantial impact on the successful delivery of a project. For example, one sampled project has seen increasing costs and schedule impacts as a result of infrastructure interdependencies that were not accounted for early in the project’s life. As a result, this project, as well as a related project with shared infrastructure, are experiencing cost and schedule increases. Project interdependencies are primarily identified in the early phases of a project through the Business Case Analysis, reported in Defence Services Program Portal project dashboards and managed within the project. There does not appear to be holistic tracking of project interdependencies at a departmental level. The defence project management framework, through the PAD and L1 guidance, guides the management of numerous elements and projects that are integrated and interdependent, thus impacting both other projects and DND/CAF as a whole. While interdependencies are identified at the outset of a project, challenges exist in managing the interdependencies throughout the project lifecycle as dependent projects begin and additional constraints arise. Increased consultation, oversight and reporting could reduce the risk that project updates and their impacts are not accounted for or sufficiently communicated to all relevant stakeholders. Changes to a project’s cost, schedule, or a reduction or change to a capability represent an interdependency risk and should be managed accordingly. The realization of interdependency risks can have substantial downstream impacts across various domains and L1 organizations. Other reviews have highlighted that projects are not always adequately identifying indirect resource requirements such as key oversight or internal support functions.

Recent work has been done to address interdependency management in DND/CAF. For instance, infrastructure and information management interdependencies should benefit from the recent development whereby ADM(IM) and ADM(IE) are being invited to SRBs to ensure early collaboration. Secondly, C Prog and ADM(IE) are working on revisions to the PAD to incorporate infrastructure interdependency considerations. ARAs within the PAD and within project documentation are generally clear, well defined and relevant. Additionally, the SDO and their responsibilities within investment planning and management, including project management, procurement and materiel, information management, and real property, have been appointed by the Deputy Minister. Finally, further work is being done to address integration and interoperability of the military across the CAF and with Allies, through the newly formed Chief of Combat Systems Integration organization. Recognizing the progress made, other options and processes should be explored to further improve the holistic management of interdependencies.

Project Guidance

Project guidance, in the form of templates, guides, tools and direction, is abundant and varies across L1s, with a standard base of higher level guidance found in the PAD. The PAD provides guidance and templates to inform decision makers of defence project management while giving clear direction to project teams. It provides an overview of project processes, considerations, stakeholders, documentation and more. There is a lack of standardized scheduling and costing tools across project implementing L1s. ADM(Fin) costing guidance is detailed and being more proactively communicated with the introduction of a costing bootcamp. This should aid in the PAD’s recommended early engagement and communication between ADM(Fin) and projects during project costing exercises. Scheduling challenges, along with other key project management areas such as prioritization and resourcing, are being addressed as the result of numerous recent project management reviews. For more details, see Annex C. The recommendations from these reviews have helped improve departmental practices (e.g., the implementation of 3 point scheduling estimates to provide a range of pessimistic, likely and optimistic schedule estimates).

Risk Management

Risk Management occurs both formally and informally within projects and across L1s, where risks are reported and escalated through SRBs, management briefings and dashboard reporting. Projects engage with numerous governance bodies, which can be a standard part of the project approval process or unique to each L1, to progress projects and raise and report key risks. These governance bodies provide a forum for discussion and challenge of capability and project trade-offs as the project moves towards interim and full operational capability. The challenge function provides senior management with the opportunity to review project progress, risk areas and leverage their departmental knowledge in project decision making.

Although processes and oversight are in place to report and manage risk, projects continue to face unmitigated and unforeseen risks and issues. While interdependencies are identified and reported in some risk reporting frameworks, this is not uniform across DND/CAF or L1s. There is a defined process in the PAD for how projects report risk and are selected for higher level governance bodies. Not all projects are escalating risks in a timely manner. For example, one project experienced significant delays for which key risks were briefed to an L1 oversight committee, and subsequently to SRB, and corrective actions were taken. These risks/issues were not formally briefed to senior levels until after the project was back on track (approximately 20 months after their identification).

Consideration 1: Regarding the management of project interdependencies, potential options for consideration include mandating projects to report on interdependency risk to PMB and SRBs, requiring attestations (as part of the SRB checklist procedures) by the key implementers to confirm that adequate consultation is taking place throughout the lifecycle of the project, and maintaining a tracking tool or database to monitor the impacts of project changes across all interdependencies.

Data Management

Why It Matters

Data management and performance measurement are important as they enable the identification of issues and of unmitigated risks that may have been avoidable. It is difficult to track and measure the outcomes of the project and make strategic or evidence-based decisions if the data being collected is unreliable. A lack of consistent and effective data management limits the extent to which project data can be leveraged for continuous improvement purposes and reporting on Program outcomes.

What We Found

Data Quality

Projects use a combination of information systems and tools to manage their data and project information. DRMIS is the system of record for project, financial and other key data. Data quality in DRMIS requires improvement as the current data used by projects for tracking, monitoring and project reporting, is non-uniform, poorly labelled and incomplete. Data is infrequently updated or often taken from other reporting tools that, while more regularly updated, require manual retrieval and cleaning for analysis.

Project data is not always available or stored in DRMIS and is often held on desktops by staff or within the owner’s organization, limiting its availability for sharing of data amongst organizations. For example, L1s do not always share historical operations and sustainment costing data with ADM(Fin) to support costing of future equipment needs. As DRMIS is not user friendly, project teams do not always use DRMIS to its full capabilities. This is due to a number of factors, including a lack of training on DRMIS for project staff, lack of accessibility and the limited non- financial capabilities of DRMIS for project usage. These data quality and DRMIS issues limit the Defence Team from moving forward with developing its analytics capabilities, in line with industry standards. For example, project data updates are completed at the discretion of the project team, and dashboards are not frequently updated with supporting comments. These issues are exacerbated by limited controls to verify project data. Data quality issues further limit the ability for project stakeholders to conduct analysis and support projects.

Review and Validation of Data

Data review and/or quality control on project data occurs to various degrees across L1s. There is no consistent approach to reviewing and performing quality control on project data at a departmental level. Instead, project teams and their chain of command review the data and reporting of their own projects while other stakeholders such as ADM(Fin) or C Prog may review information that is presented on slide decks prior to SRBs. In addition, as an internal control for project information for SSE projects, Functional Authority Delivery Group representatives attest to the information on their L1 projects, once per year.

While project information is often challenged by the chain of command and in review boards, it is not validated to ensure quality and conformance with guidance. DND/CAF would benefit from projects having its data validated for reasonableness and accuracy. This would allow quick course correction for projects that may have unreliable or incomplete data entered in the system of record. L1 guidance on data input and quality requirements varied in depth and quality and lacked standardization. This is expected to improve with the revisions being made to the departmental and MGI-6-1, implementation of ADM(DIA)’s Data Strategy Implementation Plan, Data Governance and Quality Frameworks.

Consideration 2: VCDS, in collaboration with ADM(Mat), ADM(IM), ADM(IE), ADM(DIA), and ADM(Fin) should leverage Defence X to identify and implement controls to improve project management data quality and consistency.

Consideration 3: VCDS, as the SDO for the management of projects and programmes in Defence, will continue to develop and expand upon departmental guidance in support of project management activities for all of DND/CAF. This guidance should be further developed in collaboration with other implementing L1s, with considerations for improvement of data management in the areas of data standardization, guidance and DRMIS usage, and referred to in the PAD, if applicable.

Overall Conclusion

Project management requires strong leadership, detailed planning, identification and management of risks and interdependencies to deliver numerous key tasks during the project lifecycle. Effective project management facilitates the timely delivery of a broad suite of capabilities on time, on budget and in support of DND/CAF’s operational and organizational needs. DND/CAF has progressed significantly in recent years by developing and improving its departmental project management framework to support successful project delivery and expanding upon key guidance that is constantly evolving. Room for improvement exists in the areas of interdependency management and project data management.

A robust project management framework is in place to provide guidance, processes and oversight for projects. Without active and holistic interdependency management throughout the project lifecycle, there is a risk that project changes to schedule or scope will not be tracked and communicated to all relevant stakeholders. This can impact the delivery of related projects and project elements, and can increase the risk of failing to deliver key capabilities. Interdependencies at the project or departmental level can result in a number of downstream impacts, with the most immediate being the project level impact to cost and schedule. The need for improved interdependency management may be amplified with key departmental priorities such as NORAD Modernization.

A lack of data quality control impacts the availability of complete and reliable information to support effective project management and senior management decision making. While there is guidance at the departmental and L1 levels to support data and information management, additional guidance and processes are needed to ensure data quality and consistency to support analytics and reporting. While some review or quality assurance of data is occurring within the chain of command, the review and validation of data entry needs improvement.

DND/CAF has progressed significantly in recent years by developing and improving the departmental project management framework to support successful project delivery. By addressing shortfalls to interdependency management and project data management, as well as areas and recommendations identified in other reviews, defence project management will continue to improve. This will enhance the ability of DND/CAF to deliver projects on time, within scope, on budget and to meet the identified high-level mandatory requirements and other expected outcomes.

Annex A: Management Action Plan

Management Action

The VCDS, through the Policy Suite DG Steering Committee, will ensure the monitoring and tracking of interdependencies is improved by:

• Ensuring SSE Functional Authority Delivery Group Annual Attestation Process, which includes updating of initiative documentation and interdependencies assessments, is communicated across L1s.
  • Target – Already completed annually as an evergreen process.
• Ensure recent governance and oversight changes to the PAD are achieving desired effect, in particular with infrastructure interdependencies.
  • Target – September 2023 – This requires C Prog and ADM(IE) coordination to develop KPIs to measure and track the effectiveness of Defence Capability Infrastructure changes to the PAD implemented on June 27, 2022. An assessment using these KPIs will leverage data gathered over months in order to be valid.
• Improve current “static” reporting mechanisms for project interdependencies by leveraging emerging digital tracking methodologies, including the use of Power BI and Dashboards.
  • Target – December 2023 – This is an evergreen task with emerging digital tracking methodologies figuring prominently in our current and future tool sets.

This Management Action Plan requires collaboration across multiple stakeholders and is reliant upon Human Resources capacity and competency levels necessary to leverage new digital enabling tools.

Target Date – December 2023

OPI: VCDS
OCIs: ADM(Mat), ADM(IM), ADM(IE), CFINTCOM, RCN, RCAF, C Army, ADM(DRDC), CANSOFCOM, CJOC, CMP, SJS, ADM(DIA)

Annex B: About the Audit

The findings and recommendations of this report were derived from multiple sources of evidence collected throughout the planning and conduct phase of the project. These sources of evidence were verified with the OPIs to ensure their validity. The methodology used in this audit were as follows:

Audit Criteria

Criteria A: Major Capital projects implemented by ADM(Mat), ADM(IM) and ADM(IE) are supported by a robust project management framework.

• Governance is in place to support the Definition to Closeout phases, including the management of tradeoffs between cost, schedule and requirements.
• Policies and procedures are in place to support a robust project management framework.
• Risk management practices, consideration of stakeholder engagement and management of interdependencies are integrated into the project management process.
• Selected projects (case studies) are aligned with the project management framework in place.

Criteria B: The data collected by Major Capital Project implementers is reliable and usable.

• Major Capital Project implementers ADM(Mat), ADM(IM) and ADM(IE) collect and capture data in the system of record to report on project status.
• Quality (timely, accurate, complete and relevant) data is collected by Major Capital Project implementers (ADM(Mat), ADM(IM) and ADM(IE)).
• Conduct data analytics/trend analysis to support partners.

Annex C: Select Defence Project Management Reviews

Annex D: Defence Project Governance^{Footnote 3} 

Figure 6 Summary

Defence Capabilities Board (DCB)

DCB’s mandate is to provide VCDS, on behalf of the Deputy Minister/Chief of the Defence Staff, with the situational awareness and decision support with respect to the development and validation of future Defence capabilities. This board serves as the approval authority for all Strategic Context Documents (SCD) and Business Case Analysis (BCA) prior to acceptance of an initiative into the departmentally approved balanced portfolio

Infrastructure and Environment Board (IEB)

The IEB, chaired by ADM (IE), ensures that IE performance is meeting expectations and focused on enabling Canadian Armed Forces capabilities. It provides strategic advice and corporate guidance to ADM (IE) on infrastructure and environment matters. The IEB ensures the performance of the IE portfolio and the enabling real property management is regularly and systematically assessed for CAF operational suitability and relevance, utilization, efficiency, condition and financial performance. The primary goal is to ensure that IE performance is meeting expectations and focused on enabling CAF capabilities. 

Information Management Board (IMB)

The IMB is the senior Information Management (IM) governance body in DND/CAF. It is chaired by ADM(IM) and C Prog. The IMB provides strategic leadership and recommends priorities on all matters related to the delivery and support of IM/IT in DND/CAF.

Investment and Resource Management Committee (IRMC)

Investment and Resource Management Committee (IRMC)’s primary mandate is to promote the effective allocation and management of the Department of National Defence’s available financial resources. The Investment and Resource Management Committee (IRMC) provides advice to the DM on Budget priorities and requirements consistent with the strategic objectives. The Committee oversees the allocation, oversight and control of the Department’s financial resources, control of risks, reviews financial policies and practices and oversees the management and progress of major investments.

Programme Management Board (PMB)

The PMB provides Associate DM, VCDS and the Chief Financial Officer with decision support and advice with respect to the composition of the Investment Plan (IP) and the management of elements of the DSP.

Senior Review Board (SRB)

The SRB is a departmental committee that supports the Project Leader in the successful delivery of the capability for which an investment project has been established. There are two major components to the role of the SRB:

• To provide "corporate challenge" and stakeholder oversight of the project
• To advise the Project Leader on the development and management of the project, with a focus on providing cross-functional input to discussions on project risk and performance

SRB members: ADM(IM), ADM(IE), ADM(Mat), and ADM(Fin), C Prog and CFD analysts, as well as other key stakeholders.