Evaluation of the Cyber Forces

Overall Conclusions

Table 1 Summary

This table has two columns. The left-hand column lists the twelve key findings and the right-hand column lists the associated recommendations. Key findings and their corresponding recommendations are located across each row.

Cyber Operations

Ready Cyber Forces

Cyber Force Development

Program Expenditures

Figure 1

Figure 1. Cyber Forces Expenditures FY 17/18- FY 19/20 ($ millions).Footnote 1

Figure 1 Summary

This chart represents fiscal year expenditures of the Cyber Forces between FY 17/18 and 19/20 with numbers in millions of dollars. It is a stacked bar chart for each Program with figures for 17/18, 18/19, 19/20 and the totals of all three programs per fiscal year.

FY 17/18

  • 1.5 Cyber Operations: 92.7
  • 2.6 Ready Cyber Forces: 78.6
  • 4.6 Cyber Force Development: 56.6
  • Total: 227.9

FY 18/19

  • 1.5 Cyber Operations: 73.9
  • 2.6 Ready Cyber Forces: 88.4
  • 4.6 Cyber Force Development: 18.8
  • Total: 181.1

FY 19/20

  • 1.5 Cyber Operations: 58.5
  • 2.6 Ready Cyber Forces: 70.5
  • 4.6 Cyber Force Development: 15.8
  • Total: 144.8


Program expenditures for each Program only go back to FY 2017/18, as illustrated in the chart. To note, FY 2019/20 figures represent planned expenditures. 

In  FY 2019/20, the Cyber Forces were supported by 1309 full-time equivalents in total, with 671 for Program 1.5 (blue), 602 for Program 2.6 (orange), and 36 for Program 4.6 (green). 

Program Stakeholders

Figure 2
Figure 2. The Cyber Forces in DND/CAF.
Figure 2 Summary

This figure represents the evolution of the Cyber Forces since its inception in 2010. It is a horizontal timeline from 2010 to 2017. The left-hand side of the figure notes that in 2010, the Cyber Task Force was established under ADM(IM). In 2011, the Cyber Task Force transferred to CFD and renamed DG Cyber. The right-hand side of the figure displays that in 2017, the Creation of the Cyber Forces occurred. At this time, DG Cyber transferred to ADM(IM), 6 SSE initiatives were linked to cyber (63, 65, 87, 88,89, 90), the Chief of Defence Staff (CDS) Cyber Initiating Directive was released, and the cyber operator trade was created.

The Cyber Mission Assurance Program

The evaluation assessed the development of the Cyber Mission Assurance Program (CMAP), which falls within the authority of the Cyber Forces. The CMAP forms a large portion of the development of the Cyber Forces and was created in response to SSE initiative 87. CMA incorporates the concepts of “cyberspace” and “mission assurance,” which is the ability of an organization, service, infrastructure, platform, weapons system or equipment to operate in contested cyberspace and accomplish its mission.Footnote 2 

The CMAP is a DND/CAF-wide endeavour led by DG Cyber to create a cyber resilient defence team. It is specifically designed to address the cyberspace resilience of people, process and technology from cyber-associated threats with five lines of activities:

  1. Policy
  2. Governance
  3. Stakeholder Engagement & Collaboration
  4. Education & Training
  5. Reporting

CMAP is working to establish a structure of centralized oversight with decentralized execution for cyber mission assurance to enable and empower Level 1s (L1) with the appropriate Accountabilities, Responsibilities and Authorities (ARA) to do their part to ensure cyber mission assurance and cyber resilience at every level of DND/CAF. In short, the concept of CMA is everyone’s responsibility.

CMAP ObjectivesFootnote 2 

“Protect critical military networks and equipment from cyber-attack by establishing a new Cyber Mission Assurance Program that will incorporate cyber security requirements in the procurement process”

SSE Initiative 87

Back to Table of Contents

Evaluation Context

This report presents the results of the evaluation of the Cyber Forces, conducted during FY 2019/20 by ADM(RS) in compliance with the 2016 Treasury Board Policy on Results. The evaluation examines the performance of the Cyber Forces over a three-year period, FY 2017/18 to 2019/20 and was conducted in accordance with the DND/CAF Five-Year Departmental Evaluation Plan. The findings and recommendations in this evaluation may be used to inform management decisions related to program design, delivery and resource allocation, as well as serve as a baseline for future evaluations. 

The Cyber Force was newly created upon the establishment of the Departmental Results Framework (DRF) in 2017.

*Link last accessed on Oct 23, 2020


Back to Table of Contents

Program Implementation & Management

The Cyber Forces lack direction without clear ARAs and a sufficient voice to inform decision making.

Finding 1: Unclear ARAs, the current organizational construct of ADM(IM), and the lack of a Cyber Champion ██████████████████████████ of the Cyber Forces for the CAF and the ability to inform senior management decision making.

Unclear ARAs

 

Figure 3
Figure 3. Survey results indicate that senior cyber stakeholders do not believe there are clear ARAs.
Figure 3 Summary

This figure is a horizontal bar graph indicating a range of strongly disagree, disagree, agree strongly agree. Results reflect the percentage of respondents who align with the two statements in the chart.

There are clear ARAs that empower leaders to achieve strategic outcomes.

  • Strongly disagree: 21%
  • Disagree: 44%
  • Agree: 31%
  • Strongly Agree: Negligible 

There is clear policy that direct program actions to achieve strategic outcomes

  • Strongly disagree: 12%
  • Disagree: 41%
  • Agree: 39%
  • Strongly Agree: 8% 

The Organizational Context

The Lack of a Cyber Champion

Recommendation 1


Back to Table of Contents

Cyber is currently integrated into existing governance bodies, but has none of its own.

Finding 2: There is no cyber-specific internal governance body. The cyber domain utilizes the existing ADM(IM) governance bodies of the Department, supported by Working Groups which tend to be informational. Stakeholders are responsible for developing their own cyber strategies.

Recommendation 2

Review the current governance framework to determine whether the cyber domain requires a distinct structure.


Back to Table of Contents

There is evidence of strategic planning and insight into program development.

Finding 3: Although there are a number of strategic documents for the planning of the Cyber Forces, awareness and use are not evident across DND/CAF.

A number of foundational documents concerning cyber, Cyber Forces development and Cyber Mission Assurance were reviewed. Some of these included the following:

Additionally, program managers have identified considerable work that is underway in the development of other concepts and doctrines, such as the renewal of the Joint Doctrine Note, as well as the drafting of new DAODs.

DG Cyber has produced and necessarily continues to produce multiple sources of foundational doctrines and concepts for the cyber domain...

…However, in spite of doctrines and concepts, there is still a lack of clarity.

Figure 4 Summary

This figure is a pie chart. Results reflect the percentage of respondents who agree with the pie chart statement.

  • 71% agree
  • 29% disagree

Improvement Strategies

During interviews, senior program managers agreed that there is a lack of clarity with regard to cyber. DG Cyber has plans to more thoroughly include cyber awareness and understanding into Officer and Non-Commissioned Members professional development. However, they acknowledge that developing training could take a long time.

  • Additionally, increased dissemination measures could be explored to ensure that a wider audience in DND/CAF has received and is aware of these strategic documents.


See Recommdendation 1

Back to Table of Contents

Implementation of the Cyber Mission Assurance Program has been delayed.

Finding 4: Implementation of SSE 87 will █████████████████████ as support for Cyber Mission Assurance is not universally prioritized across DND/CAF.

As mentioned previously, the CMAP is a key component of the cyber domain for all of DND/CAF.  The importance of CMA was noted and acknowledged by all interviewees; however they also █████████████████████████████████████████

Program managers agreed with these concerns, referring to a number of challenges in the implementation of CMAP initiatives. In particular, ██████████████████████████████████████████████████

 

Figure 5

 
Figure 5 Summary

Personnel

Signs of Progress

  • The recent establishment of a permanent team lead for the CMAP has led to improved progress and stability.
  • Work is underway for the CMAP Charter to enable more reliable funding levels as the program is formalized. 
  • ADM(Mat) has played a very significant role in undertaking development and implementation activities for CMA and materiel procurement processes for supply chain resilience.

“CMA requires not only the IM Group, but also the VCDS and other affected L1s to take it on board”

Figure 6 Summary

This figure is a pie graph. Results reflect the percentage of expenditures dedicated to CMAP.

  • 8% expenditures dedicated
  • 92% expenditures elsewhere 


 See Recommdendation 1

Back to Table of Contents

The Cyber Forces have not identified required levels of preparedness.

Finding 5: The responsibility for defining cyber readiness ███████████████████████████████████████████ with the exception of the Navy which has made some progress.

Suggestion for follow-up:

Examine the formalization, direction and compliance of cyber readiness


Back to Table of Contents

Lack of cyber-ready workspace is impeding both operations and training.

Finding 6: █████████████████████████████████████████████████████████████████████████████████████████████████████████████████

Figure 7
Figure 7. National Capital Region, distances between DND locations: NDHQ, Shirley’s Bay, Carling Campus, CFS Leitrim, and Star Top.
Figure 7 Summary

This figure illustrates the distances between various DND locations.

  • NDHQ and Shirley’s Bay: 27 km
  • Shirley’s Bay and Carling Campus: 8 km
  • Carling Campus and CFS Leitrim: 35 km
  • CFS Leitrim and Star Top: 11 km
  • Star Top and NDHQ: 10 km

Recommendation 3

Create an institutionalized, centralized, serviced cyber range with classification adaptability and remote access.


Back to Table of Contents

Research & Development

DND/CAF Cyber projects are not sufficiently nimble to cope with changes in the threat environment.

Finding 7: ████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████

Figure 8
Figure 8. Survey respondents strongly disagreed and disagreed that cyber projects are efficient.
Figure 8 Summary

This figure is a horizontal bar graph indicating a range of strongly disagree, disagree, agree strongly agree. Results reflect the percentage of respondents who align with the three statements in the chart.

Cyber projects are coordinated to ensure the optimal use of resources.

  • Strongly disagree: 25%
  • Disagree: 28%
  • Agree: 47%
  •  Strongly Agree: 0%

There are no instances of duplication in the development of cyber projects

  • Strongly disagree: 34%
  • Disagree: 50%
  • Agree: 14%
  • Strongly Agree: negligible

The cyber project development process has mechanisms to ensure efficient project delivery

  • Strongly disagree: 20%
  • Disagree: 41%
  • Agree: 37%
  • Strongly Agree: negligible

Challenges

Possible Mitigation Strategies

A properly integrated CMAP could mitigate the impact of changes in technology and the inclusion of cyber considerations.


Back to Table of Contents

The Cyber Forces are extensively engaged with numerous stakeholders.

Finding 8: The program design has incorporated relevant knowledge from DND/CAF, OGDs and allied stakeholders; however, engagement with private industry and academia has been a challenge.

DND/CAF

There is significant evidence of internal engagement and involvement from other L1 organizations and Environmental Commands:

Figure 9
Figure 9. Survey results indicate that a majority of DND/CAF cyber stakeholders agreed and strongly agreed that they experienced effective management practices.
Figure 9 Summary

This figure is a horizontal bar chart indicating a range of strongly disagree, disagree, agree strongly agree. Results reflect the percentage of respondents who align with the four statements in the chart.

My organization has sufficient engagement with program managers.

  • Strongly disagree: Negligible
  • Disagree: 29%
  • Agree: 55%
  • Strongly Agree: 13%

My organization has sufficient voice on cyber committees

  • Strongly disagree: 10%
  • Disagree: 23%
  • Agree: 47%
  • Strongly Agree: 20%

My organization has regular communication with DGIMO

  • Strongly disagree: 10%
  • Disagree: 21%
  • Agree: 34%
  • Strongly Agree: 35%

My organization has regular communication with DG Cyber

  • Strongly disagree: negligible
  • Disagree: 16%
  • Agree: 52%
  • Strongly Agree: 29%

OGDs

Overall, interviews with program managers highlighted the various relationships with OGDs. DND/CAF is present at a number of interdepartmental committees, such as the DG Cyber Strategic Committee.

Allies

DGIMO and DG Cyber have numerous relationships with international partners and allies, which have been a source of best practices to incorporate into the Cyber Forces. In some instances, allied relationships have been more active than OGD relationships:

Private Industry

There are 275 cyber firms in Canada, of which 250 are related to cyber IT security and 25 are specific to cyber defence.Footnote 6  Industry has a lot to offer, such as innovation and agility, which may be used to further enable the Cyber Forces. The DND MINDS program brings DND and industry to the table and may be the quickest way to incorporate cyber into DND/CAF operations and activities. However, there are a number of barriers which limit increased partnerships:

Academia

There has been very little engagement with academia except with the Royal Military College (RMC), where engagement is robust, providing science and technology, and research and development perspectives in the cyber domain. Interviews with senior program managers identified some challenges with engaging academia:

“RMC is filling the gaps but it is not a scalable solution”


Back to Table of Contents

Personnel Generation

The Cyber Forces have a sufficient number of positions; however, filling the positions has proved to be difficult.

Finding 9: Cyber positions are being allocated across DND/CAF ████████████████████████

Civilian Cyber Force

Figure 10 Summary

This figure is a pie chart. Results reflect the percentage of those who align with the pie chart statement.

  • 84% ███████████████
  • 16% █████████████

 

Figure 11
Figure 11. DND/CAF is unable to be financially competitive with respect to analysts’ salaries in the cyber domain.
Figure 11 Summary

This figure is a horizontal lollipop chart, which is similar to a horizontal bar chart, but with a bulb at the end. Each bar illustrates the different salaries made by those working in the cyber domain between, CAF, DND, CSE and the private sector.

  • CAF – Corporal: $67,392
  • DND – CS-02: $ 75,027
  • CSE – UNI-7: 78,001
  • Average Canadian Cyber Analyst SalaryFootnote 7 : $97,500

A lack of cyber personnel is a risk to successful program implementation.

63% of survey respondents indicated that there is no effective career management for the cyber operator occupation.

In order to keep pace with both Allies and adversaries, DND/CAF need to ensure cyber personnel remain current and employed within the cyber domain.


Back to Table of Contents

Opinions remain mixed on the need for a Cyber Officer occupation.

  Case Study: Cyber Officer

Why:

The evaluation conducted a case study of a quasi-Cyber Officer, after the topic arose during scoping interviews, to determine whether the expectations and technical requirements of a Cyber Operator warranted the possibility for a Cyber Officer position. As it is too early to make an informed opinion to this regard, the evaluation analyzed present opinions and context around a potential Cyber Officer Position.

How:

The evaluation interviewed the case subject, queried survey and questionnaire respondents and discussed with interviewees.

Analysis Highlights


84% of survey respondents believe that DND/CAF would benefit from the creation of a Cyber Officer occupation. The majority of these respondents were at the tactical and operational levels.

However, the majority of interviews with senior management disagreed.

Suggestion for Follow-up:

Examine the results of the Cyber Leadership study conducted by D Cyber Ops FD, slated to begin in March 2020, which will examine this issue ██████████████████████


Back to Table of Contents

Delays in security clearances negatively affect the Cyber Forces.

Finding 11: Security clearances processes and timelines ███████████████████████████████████████████████████████

82% of senior manager survey respondents indicated that security clearance approvals ███████████████████████████████████████

72% of survey respondents noted that security clearance approvals ████████████████████████████████████

██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████

████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████


Back to Table of Contents

CAF training is unable to keep pace with cyber needs; however, third-party cyber training lacks standardization and validation.

Finding 12: As cyber training continues to be developed and formalized, █████████████████████████████████████████████████████████████

The draft CMA Program Charter (2020) states “It is anticipated that some specialist training will be required to develop the necessary skill sets to enable this program -  the nature of which will be determined during options analysis.” 

In FY 2020/21, the CMAP is expected to complete a training needs analysis, formalize training and awareness framework, and issue common training guidance, all of which are reported to be on track.

Without knowing whether cyber operator training is achieving an expected standard, cyber effects may face unintended consequences.

Recommendation 4

Assess the feasibility of standardizing third-party training with training validations.


Back to Table of Contents

Conclusions

As a new group, the Cyber Forces are actively engaged in numerous activities to establish Canada’s position in the cyber domain of warfare. DG Cyber and DGIMO, as the leads for cyber in DND/CAF, have produced and are currently working on various initiatives to set in place the foundational components needed for an effective Cyber Force in the future. However, unless appropriate attention is given to the cyber domain, the rate of implementation will continue to be constrained.

The Cyber Forces’ program design theory is robust. Extensive strategic planning and insight into the development of the Cyber Forces is apparent through the production of cyber concepts and doctrines, as well as the formalization of terminologies. Further, DG Cyber and DGIMO have maintained extensive engagement to ensure that relevant knowledge and stakeholder engagement were included in the Cyber Forces’ design. Continued regular engagement with stakeholders as well as the incorporation of best practices and lessons learned will enable an effective Cyber Force.

The Cyber Forces need DND/CAF-wide support and investment to ensure a holistic and effective implementation of CMA and other cyber initiatives. DND/CAF must undergo a cultural shift to recognize the importance of CMA, as it is critical to all DND/CAF activities and operations. Pan-DND/CAF cyber initiatives cannot be implemented without the support of L1s and their respective cyber teams, and processes may need to evolve in order to effectively enable the Cyber Forces. Presently, cyber implementation activities are at risk of delays and constraints without the appropriate resources to facilitate their efforts across the Department.

Cyber stakeholders require greater strategic guidance. In spite of a robust program theory, cyber stakeholders have indicated that increased strategic direction is needed to guide their respective implementation of cyber initiatives. Instead, cyber stakeholders are acting without guidance (e.g., cyber readiness). During the evaluation period assessed, there was no overarching DND/CAF Cyber Vision.

Early program developments indicate signs of progress for training of the Cyber Forces. The creation of the Cyber Operator trade required rapid development of courseware and professional development to support the new trade. Additionally, initiatives to increase awareness and knowledge of the cyber domain among military and civilian members have been rolled out across the country.


Back to Table of Contents

Annex A — Management Action Plan

ADM(RS) Recommendation 1

To make Cyber Forces management more effective, ADM(IM) should review, update and promulgate cyber relevant ARAs across DND/CAF to ensure their awareness.

Management Action 1

  • ADM(IM) recognizes the Armed Forces Council decision of February 2018 that endorsed the ARAs for the creation of CFC, Chief of the Cyberspace Staff (C Cyber) and the Joint Force Cyber Component Commander (JFCCC). The C Cyber position was established by DM/CDS in NDHQ Organization letter in February 2018.
  • Three part action plan as follows:
    • 1.1: Determine the status of the Cyber Force leadership positions such as CFC, C Cyber, JFCCC, and take necessary steps to ensure that the positions are properly established;
    • 1.2: Upon further strategic direction, work with L1 stakeholders to determine the proper ARAs for each role (Force Management, Force Development, Force Generation, Force Employment) within the Cyber Force at the senior leadership level; and
    • 1.3: Promulgate ARAs using appropriate organisational instruments including DAODs, doctrine and DND policies.

OPI: C Cyber/DGICFD
Target Date: March 2021

ADM(RS) Recommendation 2

Review the current governance framework to determine whether the cyber domain requires a distinct structure.

Management Action 2

  • C Cyber and Directorate General Information Capability Force Development (DGICFD), in consultation with the VCDS and other L1 Authorities will develop, evaluate and make recommendations on whether or not the cyber domain requires a distinct governance structure. The recommendations will consider:
    • 2.1: Matching governance mechanisms for the cyber domain with land, maritime, air and space domains in line with the CAF approach to Pan Domain Force Employment Concept;
    • 2.2: Ensuring that the governance of the CMAP reflects the pan-CAF/DND nature of cyber resilience in people, processes and technology leading to CAF mission success in any cyber contested domain; and
    • 2.3: The organizational design of governance systems to ensure they are practical and sustainable.

OPI: C Cyber/DGICFD
Target Date: June 2021

ADM(RS) Recommendation 3

Create an institutionalized, centralized, serviced cyber range with classification versatility and remote access.

Management Action 3

  • The C Cyber and DGICFD acknowledge the concerns with the limited access to cyber ranges used for simulations and training, as well as the physical infrastructure to access them.
  • Three part action plan as follows:
    • 3.1: Formally accept the Collaborative Security Test Environment/Interim Cyber Training Environment as the interim CAF cyber-immersive training environment;
    • 3.2: Determine the CAF’s cyber-immersive training environment requirements in coordination with all L1s; and
    • 3.3: Determine if a capital project is required as the cyber range permanent solution.

OPI: C Cyber/DGICFD
Target Date: March 2022

ADM(RS) Recommendation 4

Assess the feasibility of standardizing third-party training with training validations.

Management Action 4

  • As C Cyber and DGICFD develop the cyber training and where third-party training is considered, it is being validated in conjunction with the Cyber Training Authority (Chief of Military Personnel/Military Personnel Generation Training Group (CMP/MPGTG)). Some existing third party training is the industry standard (certification or qualification), which does not require validation.
  • Three part action plan as follows:
    • 4.1: In conjunction with the Cyber Training Authority, DGICFD will continue to validate third-party training. We will leverage third-party training, including allies as required where there is a training gap;
    • 4.2: Continue working the CAF-ACE program to recognize admissible Post-Secondary Institutions; and
    • 4.3: Continue working with the Government of Canada (GC) Cyber Skills Workforce Development Working Group that is examining holistic cyber training across the GC, utilize common areas of interest for cyber individual training and education, and, where possible, share validation and lessons learned.

OPI: C Cyber/DGICFD
OCI: CMP/MPGTG
Target Date: Validation process ongoing / GC Cyber Skills Working Group output, July 2021


Back to Table of Contents

Annex B — Gender-Based Analysis Plus (GBA+)

As per the TB Directive on Results (2016), Mandatory Procedures for Evaluation, this evaluation took into account the government-wide policy on GBA+ as it was deemed relevant.

The Evaluation Matrix included a Key Performance Indicator dedicated to GBA+: “GBA+ considerations are included in the hiring process”

The Cyber Forces Survey that was distributed included various GBA+ questions:

Table B-1. Cyber Forces Survey Questions
Cyber Forces Survey Questions % Agree
The planning and implementation of the Cyber Forces takes into consideration DND/CAF diversity and inclusion initiatives. 96%
Cyber Forces policies incorporate GBA+ considerations. 85%
GBA+ considerations are included during the hiring process for cyber personnel. 87%
Gender and diversity initiatives are taken into consideration in the development of training for cyber operators. 77%
Table B-1 Summary

This table represents a sample of questions from the Cyber Forces Survey that related to GBA+. It has two columns and five rows. In the first row are titles for the “Survey Question” column, and the “% Agree” column. 

  • The planning and implementation of the Cyber Forces takes into consideration DND/CAF diversity and inclusion initiatives. – 96% 
  • Cyber Forces policies incorporates GBA+ considerations. – 85% 
  • GBA+ considerations are included during the hiring process for cyber personnel. – 87%
  • Gender and diversity initiatives are taken into consideration in the development of training for cyber operators. – 77%


Status of Women in Canada defines GBA+ as:

an analytical process used to assess how diverse groups of women, men and non-binary people may experience policies, programs and initiatives. The “plus” in GBA+ acknowledges that GBA goes beyond biological (sex) and socio-cultural (gender) differences. We all have multiple identity factors that intersect to make us who we are; GBA+ also considers many other identity factors, like race, ethnicity, religion, age, and mental or physical disability. 

Only approximately 5 percent of respondents to the Cyber Forces Survey identified themselves as women, and the majority of those were military members. In reflection of the CDS’ initiative to increase the presence of women in the CAF, certain initiatives could be explored to align with that intent. In the Office of the CDS Canadian Armed Forces Diversity Strategy (2016), it states “it is imperative that the Canadian Armed Forces (CAF) reflect the society it serves if we are to connect with Canadians and retain our relevance as a national institution. … Moreover, our operational experiences have demonstrated that diversity is a force enabler that enhances our operational effectiveness.” According to SSE Initiative 10, DND/CAF will fully leverage Canada’s diversity by promoting diversity and inclusion as a core institutional value across the Defence team.

Of respondents to the Cyber Forces Survey, for primary language statistics, 81 percent indicated English, 18 percent listed French, with 1 percent noting Mandarin. Approximately 15 percent of the respondents indicated an ethnicity other than white, illustrating a certain level of diversity in the Cyber Forces.

Awareness of differing identities should be actively included in all facets of the Cyber Forces to ensure an inclusive and diverse force structure.


Back to Table of Contents

Annex C — Evaluation Methodology

Data Sources

The findings and recommendations of this report were informed by multiple lines of evidence collected throughout the conduct phase of the evaluation. These lines of evidence were triangulated with each other and verified with program managers to ensure their validity.  The research methodology used in the scoping and conduct of the evaluations are as follows:

Literature Review: 

As part of the planning phase of the evaluation, a preliminary document review was conducted to develop a foundational understanding of the Cyber Forces and Cyber Mission Assurance to determine the scope of the evaluation. This was expanded upon during the conduct phase of the evaluation, as other documents were examined to find data that would help in the assessment. Documents included: government websites; departmental administrative reports, program documents, both in draft and finalized; and external reports.

Short-Form Questionnaire: 

During the conduct of the evaluation, particular issue areas were identified for further clarification and information. As a result, a number of short form questionnaires (2 – 3 questions) were sent to key points of contact through email. Organizations contacted included: DGIMO, ADM(IE), VCDS, as well as the Communications Electronics Engineering and Signals Officer career managers.

Case Study: 

The evaluation team conducted a case study concerning the necessity of a cyber officer trade. This study drew upon military documentation, survey data, interview notes and administrative data. Further information concerning the case study can be found in the report. 

Interviews: 

The evaluation team conducted over 30 interviews with organizations internal and external to DND/CAF. These responses were aggregated to inform opinion and perspectives in support of the evaluation. Unless otherwise noted, reference to “senior program managers” only refers to those who are at the director level and above in DG Cyber and DGIMO. Organizations interviewed included:

  • ADM(IM)
    • DG Cyber
    • DGIMO
    • CFIOG
    • Director Project Delivery Command and Control
    • DIM Secur
  • ADM(S&T)
  • ADM(Mat)
  • SJS
  • VCDS
  • CA
  • RCN
  • RCAF
  • CANSOFCOM
  • CFINTCOM
  • CFSCE

External to DND/CAF

  • CSE
  • CADSI

Survey: 

The evaluation team conducted a survey over the month of October 2019. Certain questions were targeted for key individuals, such as senior managers, military members or cyber stakeholder organizations external to DG Cyber and DGIMO. Charts throughout the report reflect these population nuances in their titles. Unless otherwise stated, “survey respondents” refers to the entire survey population.

In order to engage a wide population for opinions, perspectives and GBA+ data, the evaluation developed a survey in English and French. Certain questions in the survey were developed for targeted members of the population, for example, senior managers, CAF members only, or responses from cyber stakeholders external to the Cyber Forces. These nuances are reflected in the charts illustrating the data throughout the report. 

The survey was administered to targeted organizations and individuals who work within the cyber domain or have connections to the Cyber Mission Assurance Program across DND/CAF. Survey distribution relied on the Points of Contact identified through research of the DND Directory. These individuals would then pass the survey to other relevant individuals or subordinates in their Chain of Command. 

The survey remained online for approximately one month, and had a total of 120 responses from ADM(IM), ADM(S&T), ADM(Mat), ADM(IE), JAG, SJS, VCDS, CFINTCOM, CJOC, CANSOFCOM and the Environmental Commands (CA, RCN, RCAF). To note, additional respondents from OUTCAN positions submitted surveys through Microsoft Word, as they did not have access to the DWAN. 

Site Visit: 

The evaluation team visited CFS Leitrim and conducted five interviews with individuals working at CFIOG, this included the Commander of CFIOG. Due to limited security clearances, the team was unable to get a tour of the site, but a number of presentations and briefings by CFIOG units were given to the evaluation team.  

Benchmarking: 

The evaluation team conducted a comparative analysis by benchmarking Canada’s Cyber Force with the military cyber groups of the United States, the United Kingdom and Australia. Using various indicators, data was collected from various online sources, such as government websites, to enable comparison. 

Focus Groups: 

The evaluation undertook a number of focus groups to capture data from targeted populations within the program areas of the population. In particular, a focus group was held with Director Information Management Engineering and Integration. 


Back to Table of Contents

Annex D — Evaluation Limitations

The limitations encountered by the evaluation, and mitigation strategies employed in the evaluation process.

Table D-1. Evaluation. Evaluation Limitations and Mitigation Strategies.
- Limitation Mitigation Strategy
Security Clearances The nature of the Cyber Forces puts much of its business in the Secret and Top Secret realm. The evaluation was kept at the unclassified level, scoping out areas that would not meet this criteria, specifically C4I content, as described in the Evaluation Scope.
Survey Access The survey that was distributed was not able to be accessed electronically by all cyber personnel, in particular those in OUTCAN postings. Upon receiving information about such challenges, the evaluation team sent Microsoft Word versions of the survey and manually inputted their response data to be included in analysis.
Nascent Programs Due to the fact that the Cyber Forces are new and thus a formative evaluation was conducted, many documents were amended and updated as the evaluation progressed. The evaluation team kept in regular contact with program stakeholders to obtain current drafts of pertinent documentation.
Survey Selection Bias Bias could arise based on the selection of the individuals or organizations chosen for the survey, which could skew survey results. All organizations that are connected to Cyber were contacted for the purposes of the survey. Respondents were selected from units from within the respective member organizations.
Interview Bias Bias could arise based on the subjective impressions and comments of interviewees, which could lead to biased views. Interview comments were corroborated with other sources to ensure validity. Interview notes were conducted by more than one individual to confirm understanding of discussions and decrease the likelihood of bias.
Global Pandemic Due to the outbreak of COVID-19 on a global scale and within Canada, the evaluation was unable to complete the final round of high-level stakeholder engagement. Evaluation team members conducted the final phase of the project remotely.
Table D-1 Summary

This table represents evaluation limitations and mitigation strategies. The table has two rows and two columns, including titles. The first column contains information regarding limitations and the second column contains information regarding mitigation strategies.


Back to Table of Contents

Report a problem or mistake on this page
Please select all that apply:

Thank you for your help!

You will not receive a reply. For enquiries, contact us.

Date modified: