Evaluation of Using Data as a Strategic Asset in the Department of National Defence/Canadian Armed Forces

November 2023

1258-3-059 ADM(RS)

Reviewed by ADM(RS) in accordance with the Access to Information Act. Information UNCLASSIFIED.

Acronyms

Back to Table of Contents

Executive Summary

Back to Table of Contents

Evaluation Issues and Scope

Back to Table of Contents

Description of the Activities Under Evaluation

The activities discussed in this evaluation are not defined as a program in DND/CAF’s Program Inventory. They are a series of activities undertaken across the organization with a view to applying digitalization and automated approaches to use data to support a wide range of decision making. This decision making is at all levels: strategic, tactical and operational. Annex D provides an overview of the systems that were described as part of this evaluation.

DND/CAF has been implementing decision support technologies for decades. However, in July 2017, through the Joint Directive to Develop and Operationalize a Defence Program Analytics Capability, DND/CAF set in place an initiative intended to drive digital transformation and accelerate the ability to support decision making by leveraging data and information. That initiative led to a series of actions across L1s, many of which complemented and/or enhanced decision support work already underway. Many of these actions and others are described in this report as part of the organization’s activities that encourage the use of data to support decision making. These actions involved the development of systems, tools and applications (all called “systems” in this report).

Funding: There is no funding envelope associated with these activities. The funding and resources for activities have come from within individual L1s and from departmental sources.

Stakeholders: Strategic, tactical and operational decision making affects every part of DND/CAF. Therefore, the stakeholder group for this activity area is the entire organization.

A changing environment that introduces complexity and uncertainty: Virtually all activities in this area are affected by fundamental changes occurring in DND/CAF’s IM and IT environments. This includes, for example, the modernization of major systems (e.g., Defence Resource Management Information System), the introduction of new systems (e.g., SAP HANA®), the introduction of analytics tools (e.g., Microsoft Power BI®), the development of enterprise-wide digital innovations (e.g., Value Cases), the creation of new organizational units (e.g., ADM [Data, Innovation, Analytics] [ADM(DIA)], Analytics Support Centres, etc.), major change initiatives (e.g., Defence Resource Business Modernization), advanced data initiatives (e.g., Artificial Intelligence), new technologies supporting operations and many others.

Major organizational change: During the evaluation, on December 6, 2022, the DM announced the creation of the Digital Transformation Office (DTO), which realigned responsibilities and accountabilities related to digitalization in DND/CAF. This is likely to affect all activities included in the scope of this evaluation.

Back to Table of Contents

Findings and Recommendations

Achievements to Date

Important new foundational elements were established across DND/CAF.

Since 2017, important components have been put in place, including (not exhaustive):

It is important to recognize these accomplishments:

• This was a completely new sphere of activity for DND/CAF.
• Many of these components required that new management, staff, organizational infrastructure, relationships, etc., be put in place.

However, a patchwork of L1 systems has been or is being developed.

When asked about new systems that support decision making, L1s described a wide range of different areas of development:

Figure 1 Summary

Table 1. State of Development.

State of Development – Systems in the Study	#
Functioning, with core items in place	9
Under development	5

Table 1 Summary

Conclusion: Important foundational steps have been taken, and innovation has taken place. However, progress has been slow in putting in place practical systems that enable DND/CAF as an enterprise to use data to support decision making.

Return to footnote*Value Caseswere defined at the outset of the Data Analytics Initiative as high-value propositions in priority areas such as the Departmental Results Framework, Business Planning and Supply Chain Management.

Back to Table of Contents

Enterprise-level Priority Setting

L1s developed systems to meet their individual needs with no DND/CAF priorities as reference.

• Each L1 knew their business or operational requirement, knew the data and had access to the necessary skills.
  • This enabled them to address an L1-specific need and, often, to do so quickly.
• Systems to develop were most often chosen at the L3-level.
  • There was little guidance on priorities from the L1 or L0 levels.
• There were virtually no interactions between L1s.
  • Progress in any one L1 was not replicated to others.
• Five enterprise-level Value Cases were defined, and they progressed slowly or were put aside.
  • There was little progress in areas identified as DND/CAF-wide projects.
• Seventy-five Use Case^{Footnote  *} proposals were submitted by L1s, covering a wide diversity of topics and capabilities.
  • The Use Cases contributed to innovation but were not aligned to a defined set of priorities.

Conclusion: There were no DND/CAF-wide priorities to guide L1s in their development efforts, resulting in a patchwork of new systems.

Caution: It is also essential to allow and enable L1s to develop “low-hanging fruit” solutions that meet their own needs. This enables innovation leading to increased effectiveness and efficiency.

Return to footnote * Use Cases: a request was issued to L1s in 2019 for methodologies for how a user could use a system to accomplish a goal. ADM(DIA) enabled solutions for various Use Cases.

Back to Table of Contents

Priority Setting by the Canadian Armed Forces

Operational L1s developed systems for addressing a wide diversity of needs, without a focus on what would enable the CAF most.

Operational L1s developed new systems in several areas:

Because there were no CAF-wide priorities set out:

• No coordination or leverage across the CAF
  • L1s did not work together on areas of common interest that would benefit the CAF overall.
  • L1s did not share what they developed, to see if others could also use it or further build onto it.
• Non-operational L1s prioritized without clear guidance
  • Other L1s that support the CAF were not able to prioritize their activities in a way that best met the CAF’s needs.
  • There was a perception of “little connection” between systems that are operational and non-operational L1s.
• Lack of guidance for enterprise-level priorities
  • Without clear CAF-specific priorities, it is not possible to develop sound priorities at the broader departmental/organizational level.

Conclusion: Without clear CAF-specific priorities, the broader department could not optimize the investments made in new systems such that they would respond to the CAF’s needs.

Back to Table of Contents

Success Factors for Innovation

L1s were successful at developing decision support systems when there was a clear operational or business need, the data and information were understood, and technical expertise was available. When new analytics tools became available across DND/CAF, L1s began to build systems to organize and visualize data to support decisions, to automate processes and to use for other purposes. They understood and had access to the necessary data and information.

Data “problems” were identified and fixed.

• Every one of the 14 systems described in the evaluation found data problems during development.
• In each case, the exercise provided the opportunity to highlight the problems and address them.

Why this is important:

• Actual data problems did not prevent the development of a useful analytics solution. The system’s development led to solving the data problems.
• There was no shortcoming in terms of “data literacy.” Because there was a recognized need for the system, the importance of the data was also recognized.

Back to Table of Contents

Governance Over Department of National Defence/Canadian Armed Forces Digital Transformation

The necessary enterprise-level governance is not fully defined. The recent creation of the DTO will produce a realignment of responsibilities and accountabilities. It is not yet clear which organization(s) will be responsible for communicating priorities, monitoring progress against those priorities and reporting on progress.

“Governance” is now incoherent and spread across several organizations.

• DTO responsibilities include digital transformation and change management, departmental project modernization, IM, and data and analytics enablement in certain areas.
• The CDS’s CAF Digital Campaign Plan (DCP) includes objectives for aligning governing structures across the CAF.
• As indicated earlier, L1s are deciding to develop decision support systems based on their own operational needs and opportunities.
• The relevant L1-, L2- and L3-level “governance” committees are becoming less active and involved.

The DND/CAF Policy Framework does not reflect the TB Policy Suite:

• The TB Policy on Service and Digital includes broad principles, emphasizing an integrated approach to governance, planning and management.
• DND/CAF’s policy suite is not integrated and framed in a holistic manner to support the enterprise: For example, “Integrated management of service, information, data, IT and cyber security within their department” (Policy s. 4.1.3.1.) is not in place in DND/CAF.

Back to Table of Contents

Data and Information Management

Most of the new systems were relying on data in SoR. However, data in other systems and newly digitized data were not formally managed.

Conclusion: Data and information are not yet codified, managed, accessible or updated as frequently as needed in DND/CAF.

Back to Table of Contents

Synthesis of Conclusions

The evaluation was intended to determine DND/CAF’s progress in leveraging data to support decision making at the strategic, operational and tactical levels. The evaluation research indicates that, at an enterprise level, DND/CAF has not made significant progress in terms of developing practical systems that use data to support decision making. Important progress has been made in implementing foundational elements for the organization, and capabilities and tools are available to L1s. However, a patchwork of L1- specific systems has been developed, which falls short of the objective of implementing an enterprise-wide culture and capability.

The evaluation determined that DND/CAF has not set out priorities that would guide the investments in systems development across the organization. Furthermore, the priorities for the CAF have not been established—priorities that would inform and help shape those DND/CAF investments.

Responsibilities and accountabilities for communicating these priorities, monitoring progress against the priorities and reporting on progress need to be clarified. The recent creation of the DTO provides an opportunity to define and implement those structures.

The evaluation findings indicated that weaknesses in data quality did not prevent the development of useful decision support systems. Furthermore, the findings suggested that perceived weaknesses in data literacy also did not prevent the development of systems. Data and information are not yet managed as an enterprise asset, even though there was a general recognition of the importance of achieving this objective. Additionally, the Department’s policy framework does not reflect the integrative principles set out in the TB Policy on Service and Digital.

Overall, DND/CAF is not optimizing the investments in decision support systems currently being made across the organization. Setting out enterprise-wide priorities to guide these activities and implementing a structure to monitor and report on progress will help the organization’s progress in leveraging data to support decision making at the strategic, operational and tactical levels.

Back to Table of Contents

Annex A: Key Findings and Recommendations

Back to Table of Contents

Annex B: Management Action Plan

Management Action

During FY 2022/23, key activities took place within DND/CAF to support data and digital transformation:

• Issued the 2019 CDS/DM Joint Directive on Data Management;
• Developed and promulgated a multi-stakeholder implementation plan for the DND/CAF Data Strategy approved by ADM(DIA);
• Published overarching Defence Administrative Orders and Directives on Data Management and Analytics, and on Data Access (Defence Administrative Orders and Directives 6500-0 and 6500-1);
• Published DND/CAF Data Governance Framework;
• Made meaningful progress in establishing a Data Stewardship Model within DND/CAF; and
• Set up and convened regularly a committee dedicated to data governance.

ADM(DTO), in collaboration with key DND/CAF stakeholders (ADM [Chief Information Officer] [ADM(CIO)], Vice Chief of the Defence Staff [VCDS]/Chief of Combat Systems Integration [CCSI], DND/CAF L1s), is clarifying authorities, responsibilities and accountabilities, which will result in changes to existing governance that will support the digital transformation of DND/CAF by providing strategic leadership and oversight on all matters related to the digital ecosystem (the merging of people, processes, technology and data). This governance will align with CCSI’s role in brokering CAF requirements for pan-DND/CAF enablement on behalf of the VCDS role as the CAF Digital Champion.

Deliverables:

1. Finalize and publish ADM(DTO) authorities, responsibilities and accountabilities.
2. Finalize and publish Defence Digital Service Board Terms of Reference.
3. Establish joint governance between the DTO and CIO that will support the digital transformation and will align with the VCDS role in brokering CAF requirements for pan-DND/CAF enablement.
4. Develop prioritization criteria for new Digital Decision Support Systems.
5. Complete modernization of Application Portfolio Management Program.

Target dates:

1. October 2023; 2. January 2024; 3. April 2025; 4. January 2024; 5. April 2025

This Management Action Plan will be considered closed once all five interim action items have been completed.

OPI: DTO
OCIs: CIO, VCDS, Corporate Secretariat

Management Action

During the conduct of the ADM(RS) Evaluation in FY 2022/23, numerous key activities took place within the CAF to support data and digital transformation.

CCSI was established in June 2022 and the CAF DCP was subsequently published to communicate the CDS’s plan to digitally transform the CAF by 2030. VCDS/CCSI is currently, in collaboration with key DND/CAF stakeholders (ADM[DTO], ADM[CIO], CAF L1s), developing a DCP Outcome-based Plan (DCP- OP) and the VCDS Implementation Directive, which will provide CAF L1s with key tasks that will enable CAF-specific prioritization of decision support systems. Following publication, the Directive and DCP-OP will designate CCSI as the lead for Operational Data Domains and establish a CAF digital governance mechanism, both of which will address this recommendation.

Deliverable: Formalize, promulgate and publish the DCP-OP and the VCDS Implementation Directive, after which the CAF Digital Hub, CAF Digital Governance and Chief Operational Data Office will be established.

Target date: December 2023

OPI: CCSI
OCIs: CIO, DTO and DND L1s

Back to Table of Contents

Annex C: Methodology and Limitations

Evaluation Methodology

The evaluation findings and recommendations were informed by multiple lines of evidence and qualitative research methods collected throughout the conduct phase to strengthen rigour and ensure the reliability of information and data-supporting findings. These lines of evidence were triangulated, and draft findings were shared with the Evaluation Advisory Committee. The research methodology used in the scoping and conduct of the evaluation are as follows:

Document Review

A preliminary review of the foundational documents was conducted during the planning phase, which supported developing a comprehensive understanding of the activities under evaluation and informed the development of the scope and the evaluation plan. Documentation included TB and departmental directives, policies, strategies, procedures, guidance and plans that related to the management and use of data in support of DND/CAF’s mandate. In addition, Records of Discussion from four governance committees from 2018 to present were reviewed. Selected documentation was reviewed for systems described in the study.

Case Studies

In order to assess the state of DND/CAF’s progress in developing tools, applications and systems to support decision making, the evaluation contacted nine L1s: the Canadian Army (CA), the RCN, the Royal Canadian Air Force, Canadian Joint Operations Command (CJOC), Canadian Special Operations Forces Command, VCDS, ADM (Materiel) (ADM [Mat]), ADM (Infrastructure and Environment) and ADM (Finance). These L1s were asked to identify the systems they had recently developed to help support strategic, operational or tactical decision making. Some L1s identified two systems, and some identified one system. The individuals who knew most about the development of the respective system were contacted, and they answered structured questions concerning the purpose of the system, the decision making underlying its development, status of data management, the development process, barriers to success, project status and other related matters.

In all, 14 systems were described as part of the evaluation. A summary was produced for each system, and the summary was provided to the interviewees for validation. An example is provided in Annex E.

Key Informant Interviews

The evaluation team identified key contacts in L1s across DND/CAF to identify newly developed systems to support strategic, operational or tactical decision making. Those interviewees were contacted to identify the owners of these systems. Detailed interviews were then held with owners of the systems using a structured interview guideline. In addition, interviews were held with key informants involved in the DND/CAF Data Analytics Initiative. Interviews were held with representatives from the TB Secretariat and three other federal organizations to discuss their challenges and experiences related to developing and improving systems to support decision making.

In all, the evaluation included 49 interviewees. Some interviews were one-on-one, and some were in small groups of two, three or four. Some individuals were interviewed multiple times to obtain additional information on the topics under discussion.

Validity of Methodology

The methodology is sufficient to ensure the validity and rigour of the findings and conclusions.

Evaluation Limitations

The limitations encountered by the evaluation and mitigation strategies employed in the evaluation process are set out in the following table:

Back to Table of Contents

Annex D: Systems Described as Part of the Evaluation

Back to Table of Contents

Annex E: Example

The RCN App

Purpose of the System

The RCN App is a portal for RCN sailors to access information over their phones. The app will allow sailors to access information that would normally only be available on a computer with a Defence Wide Area Network connection and help them manage their daily activities. It provides a mechanism for sailors to stay informed and connected, and get their daily work done.

Impetus for the System

• The RCN App concept was requested by the Commander of the RCN as a means to digitalize some of the tools and services only available on the Defence Wide Area Network.

Development Evolution

• The RCN had never developed an app before. In 2018, the development team started at the drawing board from the very beginning. They needed to learn how to create the app: what was the process to follow, what were the security requirements, etc.
• In 2019 and 2020, workshops/basecamps were held with Regular and Reserve Forces across the country. Sailors provided their input on what the app should look like and what initial functionality should be included. The four first components were identified and chosen through that exercise. It took about a year and a half to complete the basecamps and define how the app should work.
• This involved considerable relationship management. The development team needed to learn how to “get contractors to do what needed to be done,” and they often did not know how to proceed or how to ask for what they needed. The team had to learn how to work with ADM(IM) and ADM(Mat), and they had never dealt with them before. Those organizations were “upfront and supportive.” The development team learned as the project moved forward.

New Data and Processes

• There were some new data capture processes developed: for example, the in/out routine. That process had been a PDF sent to sailors’ civilian email. The developers worked to develop a digital in/out routine to be included in the app.

Success Factors

• One of the success factors was that the Commander of the RCN asked for this system to be developed. Additionally, the team had access to good people who “knew how to get things done,” and they had the flexibility and freedom to focus and move the project forward. Command was briefed along the way every few months. Command’s direction was “not to rush it and to get the core right.”