Evaluation of the Cyber Forces

Cyber Operations

Ready Cyber Forces

Cyber Force Development

Program Expenditures

Figure 1. Cyber Forces Expenditures FY 17/18- FY 19/20 ($ millions).^{Footnote 1}

Program expenditures for each Program only go back to FY 2017/18, as illustrated in the chart. To note, FY 2019/20 figures represent planned expenditures. 

In  FY 2019/20, the Cyber Forces were supported by 1309 full-time equivalents in total, with 671 for Program 1.5 (blue), 602 for Program 2.6 (orange), and 36 for Program 4.6 (green). 

Program Stakeholders

• DND/CAF: Assistant Deputy Minister (Human Resources - Civilian), Assistant Deputy Minister (Infrastructure and Environment) (ADM(IE)), Assistant Deputy Minister (Materiel) (ADM(Mat)), Assistant Deputy Minister (Policy), Assistant Deputy Minister (Public Affairs), ADM(RS), Assistant Deputy Minister (Science & Technology) (ADM(S&T)), Chief of Force Development (CFD), Chief of Military Personnel, Judge Advocate General, Strategic Joint Staff (SJS), Vice Chief of the Defence Staff (VCDS), Canadian Army (CA), Royal Canadian Air Force (RCAF), Royal Canadian Navy (RCN), Canadian Special Operations Force Command (CANSOFCOM), Canadian Forces Intelligence Command (CFINTCOM), Canadian Joint Operations Command (CJOC).
• Other Government Departments (OGD): Canadian Security and Intelligence Service, Communication Security Establishment (CSE), Defence Construction Canada, Department of Finance, Innovation, Science and Economic Development, Privy Council Office, Public Safety, Public Services and Procurement Canada, Public-Private Partnerships Canada, Royal Canadian Mounted Police, Shared Services Canada (SSC), Treasury Board Secretariat (TBS).
• Allied & International Partnerships: Include: CYBERCOM – United States of America (USA), National Security Agency – USA (NSA), North American Aerospace Defence Command (NORAD), North Atlantic Treaty Organization (NATO), Five Eyes Partners (USA, United Kingdom, Australia and New Zealand) .
• Private Industry & Academia: There are approximately 275 firms that have been identified as related to cyber defence and cyber security in Canada, and 91 academic institutions with computer science degrees. 

The Cyber Mission Assurance Program

The evaluation assessed the development of the Cyber Mission Assurance Program (CMAP), which falls within the authority of the Cyber Forces. The CMAP forms a large portion of the development of the Cyber Forces and was created in response to SSE initiative 87. CMA incorporates the concepts of “cyberspace” and “mission assurance,” which is the ability of an organization, service, infrastructure, platform, weapons system or equipment to operate in contested cyberspace and accomplish its mission.^{Footnote 2} 

The CMAP is a DND/CAF-wide endeavour led by DG Cyber to create a cyber resilient defence team. It is specifically designed to address the cyberspace resilience of people, process and technology from cyber-associated threats with five lines of activities:

1. Policy
2. Governance
3. Stakeholder Engagement & Collaboration
4. Education & Training
5. Reporting

CMAP is working to establish a structure of centralized oversight with decentralized execution for cyber mission assurance to enable and empower Level 1s (L1) with the appropriate Accountabilities, Responsibilities and Authorities (ARA) to do their part to ensure cyber mission assurance and cyber resilience at every level of DND/CAF. In short, the concept of CMA is everyone’s responsibility.

CMAP Objectives^{Footnote 2} 

• Enable informed, timely and effective risk-management decisions and action at both the pan-DND/CAF level and within the supporting Programs
• Establish standing information feeds from a diverse range of sources to inform risk-management activities
• Enhance collective action with allies and OGDs and agencies
• Improve cyberspace resilience of CAF force elements
• Improve cyberspace protection of DND/CAF critical infrastructure and services
• Enhance materiel acquisition and support (including supply chain and operational sustainment) assurance
• Close risk gaps within and between existing programmatic boundaries
• Establish standing surveillance for the development of vulnerabilities and indications that adversaries are seeking to access vulnerabilities

Back to Table of Contents

Evaluation Context

This report presents the results of the evaluation of the Cyber Forces, conducted during FY 2019/20 by ADM(RS) in compliance with the 2016 Treasury Board Policy on Results. The evaluation examines the performance of the Cyber Forces over a three-year period, FY 2017/18 to 2019/20 and was conducted in accordance with the DND/CAF Five-Year Departmental Evaluation Plan. The findings and recommendations in this evaluation may be used to inform management decisions related to program design, delivery and resource allocation, as well as serve as a baseline for future evaluations. 

The Cyber Force was newly created upon the establishment of the Departmental Results Framework (DRF) in 2017.

• The Cyber Forces have not been evaluated previously; however, DND/CAF was part of the Horizontal Evaluation of Canada’s Cyber Security Strategy * (2017) completed by Public Safety. “The Strategy is built on three pillars: securing Government of Canada systems; partnering to secure vital cyber systems outside the Government of Canada; and helping Canadians to be secure online.”^{Footnote 3}  The Horizontal Evaluation examined:
  • The extent to which the horizontal governance structure was effective in overseeing the Strategy’s implementation; 
  • The extent to which participating departments and agencies implemented the Strategy’s funded activities; and
  • The extent to which planned activities contributed to achieving the Strategy’s main objectives.^{Footnote 3}

*Link last accessed on Oct 23, 2020

Back to Table of Contents

Program Implementation & Management

The Cyber Forces lack direction without clear ARAs and a sufficient voice to inform decision making.

Unclear ARAs

• The majority of program stakeholders believed that ARAs associated with cyber are still unclear in terms of direction and responsibilities.
• The concepts of Information Technology (IT), Operational Technology and Platform Technology (PT) have been used to delineate cyber Functional Authority (FA) to ADM(IM), ADM(IE) and ADM(Mat). However, in practice, stakeholders acknowledged that not all technology can easily fit into these concepts and further clarification is needed. 
• DGIMO and DG Cyber have been working to further define ARAs, for example, determining responsibilities between cyber defence (Canadian Forces Network Operations Center (CFNOC)) and cyber security (Director IM Security (DIM Secur).
• Senior program management acknowledges that many ARAs concerning cyber are still unclear; however, they anticipate that new Defence Administrative Orders and Directives (DAOD) will help clarify them as the Cyber Forces mature. 

The Organizational Context

• Having cyber resident within ADM(IM) has resulted in challenges for the cyber forces:
  • ADM(IM) exists outside the operational Chain of Command, and as such,  does not have command authority. 
  • ADM(IM), with its corporate, department-wide, proactive and long-term focus ███████████████████████████████████████████████████████████████████
  • Although oversight of the CMAP is conducted within ADM(IM), the Functional Authority is held by VCDS. This has resulted in delayed approvals, according to program managers.
• Interviews with senior program managers and survey responses highlighted the important synergy between cyber and networking activities, indicating that separating them would have negative repercussions. 
• Senior managers have suggested establishing a military commander within ADM(IM) with the appropriate ARAs. However, to prevent negatively impacting cyber operations, this would require additional staffing resources similar to the staffing levels of other operational commands.

The Lack of a Cyber Champion

• Interviews with program managers have raised concerns that there is an insufficient voice representing cyber at the senior-most levels.
  • Review of meeting minutes from the last year (2019) of the Information Management Board, the IM/IT Capability Development Board, the Programme Management Board and the Defence Capability Board revealed little to no evidence of cyber discussion at these high-level committees.  
• The CFC ARAs have not been formalized, nor have they been exercised. 
  • The CFC role is one of five roles of COS(IM) that is without staff support. As a result, cyber initiatives have not been able to be prioritized or raised at senior decision-making meetings.
  • Although the CFC is supposed to have a direct link to the CDS, this role is not commonly exercised, which causes delays in actioning of military cyber initiatives.
  • Interviewees have suggested that the role of the CFC is that of an advisor and not a commander, which is why they do not have the same ARAs as a commander.
  • Interviewees believe the ARAs of a full commander are required to effectively operationalize the Cyber Forces. This could be compared to the establishment of the Intelligence Commander in CFINTCOM or the Space Commander in the RCAF.

Back to Table of Contents

Cyber is currently integrated into existing governance bodies, but has none of its own.

Back to Table of Contents

There is evidence of strategic planning and insight into program development.

A number of foundational documents concerning cyber, Cyber Forces development and Cyber Mission Assurance were reviewed. Some of these included the following:

• CDS Initiating Directive (2017); 
• Cyber Joint Doctrine Note (2017); 
• CMAP Mandate (2018); 
• Defensive Cyber Operations Concept Note (2019); 
• [Draft] CMAP Charter (2019);  and
• Defence Terminology Database updates.

Additionally, program managers have identified considerable work that is underway in the development of other concepts and doctrines, such as the renewal of the Joint Doctrine Note, as well as the drafting of new DAODs.

See Recommdendation 1

Back to Table of Contents

Implementation of the Cyber Mission Assurance Program has been delayed.

As mentioned previously, the CMAP is a key component of the cyber domain for all of DND/CAF.  The importance of CMA was noted and acknowledged by all interviewees; however they also █████████████████████████████████████████

Program managers agreed with these concerns, referring to a number of challenges in the implementation of CMAP initiatives. In particular, ██████████████████████████████████████████████████

Personnel

• Cyber responsibilities are often assigned to staff who have a number of other pre-established roles. As a result, CMA cannot always be a priority task to be undertaken.
• Cyber groups across DND/CAF are usually ███████████████████████████████████████████████████████████████████████████████████
• █████████████████████████████████████████████████████████████████████████████████████████

 See Recommdendation 1

Back to Table of Contents

The Cyber Forces have not identified required levels of preparedness.

• Defined cyber readiness levels are becoming increasingly important as the demand for cyber activities continues to grow, indicating future readiness challenges.
• Readiness will not work the same for cyber as for other operational domains. The terrain is constantly changing and a Cyber War concept is challenging to formally identify. Cyber preparedness may be a more descriptive terminology.
• There is disagreement among cyber managers as to whether cyber readiness standards need to be defined before personnel qualifications are established or vice versa. Currently this is very ad hoc, identified individually or as a team, and reported upwards instead of receiving top-down direction. This is discussed further in Finding 12.
• While it is important to ensure cyber components are integrated in Force Posture & Readiness (FP&R), it is too early to do this since the Cyber Forces are still in development. ██████████████████████████ as discussed in Finding 9, it was reported that the Cyber Forces █████████████████████████████████████████████████████████████████████████
• With respect to the responsibility for the setting of readiness standards, the strongest arguments are for the Cyber Forces to set them to ensure technical standards are established and technical connections to external organizations remain current. This is as a challenge for the IM Group to do as they were perceived by senior managers during interviews to be more focused on corporate demands versus operational.
• In the ████████████████████████████████ the Navy is in the process of developing its own cyber readiness levels within three combat readiness requirements.

Back to Table of Contents

Lack of cyber-ready workspace is impeding both operations and training.

• The draft CMA Program Charter (2020) states, “It is anticipated that suitable accommodation spaces for required staff can be found within existing DND facilities.” However, no further details were given, ██████████████████████
• Certain personnel have to travel between five separate locations across the National Capital Region in order to access the appropriately cleared networks, as the main buildings housing cyber personnel outside of Canadian Forces Information Operations Group (CFIOG) ██████████████████████████ This can result in up to $200 in taxi fares a week for one person.
• Interviewees indicated that ████████████████████████████████████████████████████████████████████████████████████████████████████████ Survey responses echoed this concern over physical infrastructure limitations.
• ████████████████████████████████████████████████████████████████████████████████████████████████████████████████
• While the fit-for-purpose National Defence Secure Campus may eventually be a solution, in the meantime, operations, personnel and daily activities are experiencing inordinate inefficiencies.
• At the January 2020 Defence Capability Board, the CFD indicated that “the National Defence Secure Campus is a critically important capability and it must be expedited wherever possible.” The topic was discussed at the Project Management Board in March 2020.
• ███████████████████████████████████████████████████████████████████████████████████████████████████
• Survey respondents were █████████████████████████████████████████████████████████████████████████████████████████████████████████
• The Cyber Collaborative Imperative by Canadian Association of Defence and Security Industries (CADSI) noted that the leading collaboration functions, policies and practices from Canada’s Allies include Cyber Experimental Ranges and Capability Testing Environments. This “offers an environment where emerging solutions can be tested against known challenges…”
   

Back to Table of Contents

Research & Development

DND/CAF Cyber projects are not sufficiently nimble to cope with changes in the threat environment.

Challenges

• Force Generators and Force Employers^{Footnote 5}  indicated that ██████████████████████████████████ due to the current procurement process.
• All interviewees noted the ████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████
• Cyber is now part of every defence capability – there is a need to have governance bodies across DND/CAF engaged and energised. Platform CMA is tied to the capital procurement process; if this process is unclear with respect to cyber, ███████████████████████████████
• Many projects currently underway began before cyber requirements were considered, necessitating timely and costly amendments to include the cyber requirements.

Possible Mitigation Strategies

• Projects that remain under $5 million can move through the procurement process more quickly and lower the threats and risks associated with changing technologies, as well as elevate the opportunity to capitalize on opportunities that technology provides.
• ADM(IM) is discussing options with TBS in order to review capital project process with the goal of finding efficiencies, according to interviewees.
• ADM(S&T) can inform what can be done throughout prototype development to show stakeholders the type of functionalities they should be looking for when buying off the shelf.
• Implementing a funded program in order to allow the procurement of necessary tools quickly rather than project by project. The crypto program in ADM(IM) is one example of this, and CANSOFCOM is implementing capability-based procurement to address this same issue.

Back to Table of Contents

The Cyber Forces are extensively engaged with numerous stakeholders.

DND/CAF

There is significant evidence of internal engagement and involvement from other L1 organizations and Environmental Commands:

• All of the 12 internal DND/CAF cyber stakeholder organizations who were interviewed indicated that they were engaged in the development of the Cyber Forces in their respective areas and continue to be engaged with DG Cyber through Working Groups. In particular, the Cyber Steering Committee facilitates knowledge transfer; however, it is not a governance body.
• ADM(Mat) is sufficiently engaged in talks concerning PT and leverages ADM(S&T) to integrate cyber support systems.
• ADM(S&T) identifies, tests and prioritizes cyber requirements for capabilities that cannot be bought off the shelves. It also demonstrates linkages between IT and PT for the Environmental Commands.
• ADM(IE) is engaged in talks concerning Operational Technology as they work to develop and implement cyber strategies for defence infrastructure.
  

OGDs

Overall, interviews with program managers highlighted the various relationships with OGDs. DND/CAF is present at a number of interdepartmental committees, such as the DG Cyber Strategic Committee.

• CSE was identified as a significant OGD partner with DND/CAF as the key deliverer of the Canadian Centre of Cyber Security.
  • There are promising initiatives for increased collaboration between the organizations, such as training. However, the differences in corporate cultures between CSE and DND sometimes lead to miscommunication that may hinder collaborative opportunities.
• DND/CAF is regularly engaged with Public Safety, which leads a number of Government of Canada forums on cyber.
• ADM(S&T) sits on a number of cyber research bodies, such as the one run by ISED concerning workforce in training.
• A few interviewees noted, however, that there is a need for increased clarification of ARAs with SSC.

Allies

DGIMO and DG Cyber have numerous relationships with international partners and allies, which have been a source of best practices to incorporate into the Cyber Forces. In some instances, allied relationships have been more active than OGD relationships:

• OUTCAN liaison positions act as conduits with allies for information-sharing; however, ████████████████████████████████████
• Particular to the US, there are regular Cyber Coordination Committee briefings between the CAF Cyber Forces and US CYBERCOM to enable effective information-sharing and the identification of best practices.
  • The US Cyber Safe Program was identified as a model for the protection and security of information, which could be emulated.
• DND/CAF participates in US Cyber Flag group training exercises to learn and exchange strategies in the development of Cyber Force training with Five Eyes partners.
• DND/CAF is also a participant in various NATO cyber initiatives and other multinational conferences.

Private Industry

There are 275 cyber firms in Canada, of which 250 are related to cyber IT security and 25 are specific to cyber defence.^{Footnote 6}  Industry has a lot to offer, such as innovation and agility, which may be used to further enable the Cyber Forces. The DND MINDS program brings DND and industry to the table and may be the quickest way to incorporate cyber into DND/CAF operations and activities. However, there are a number of barriers which limit increased partnerships:

• A lack of research funding, according to industry;
• DND’s delayed engagement with industry due to current procurement processes;
• The lack of capacity for a permanent liaison position between DND and industry;
• Differing scopes of vision between DND and industry;
• Rules dictating departmental involvement in regard to contracts for capability development and capability procurement; and
• Overall security concerns due to:
  • Ownership and protection of Intellectual Property of DND cyber defence capabilities; and
  • Risks of firms being sold or failing.

Academia

There has been very little engagement with academia except with the Royal Military College (RMC), where engagement is robust, providing science and technology, and research and development perspectives in the cyber domain. Interviews with senior program managers identified some challenges with engaging academia:

• Inherent security risks in partnering with academia; and
• A lack of available Vote 10 grant funds to use for research.

Back to Table of Contents

Personnel Generation

The Cyber Forces have a sufficient number of positions; however, filling the positions has proved to be difficult.

• Attracting and retaining cyber personnel for both military and civilian personnel was █████████████████████████████
• ████████████████████████████████████████████████████
• █████████████████████████████████████████████████████████████████████████████
• Additionally, ██████████████████████████████████████████████ by ADM(IM) corporate resource requirements.
• In a cyclical fashion, ███████████████████████████████████████████████████████████████████ as indicated in the survey.
• Progress has recently been made in developing stronger relationships with educational institutions, by attending attraction events and using the Subsidized Training Education Plan as an opportunity to entice applicants.
• CFNOC reported that they were at ████████████ at the time of the evaluation.

Civilian Cyber Force

• Civilian personnel are particularly challenging to attract and retain due to the competitive nature of cyber employment opportunities.
• 93 percent of survey respondents and the majority of interviewees agreed that DND/CAF would benefit from the hiring of (more) civilian cyber personnel, stabilizing the institutional memory.
• Hiring civilian cyber personnel will remain a challenge due to current financial compensation models, tied to classification, which has constrained the growth potential for technical personnel. This has also led to retention issues of military personnel once they become cyber-trained. The Computer Science (CS) classification is not thought of as adequate for cyber-related activities.
• “…[I]f you promote them to increase compensation, then by the job definition, they will not be doing the actual work you hired them to do.” Interviewees also indicated that many technical personnel do not want to be in management roles.

63% of survey respondents indicated that there is no effective career management for the cyber operator occupation.

Back to Table of Contents

Opinions remain mixed on the need for a Cyber Officer occupation.

  Case Study: Cyber Officer

Why:

The evaluation conducted a case study of a quasi-Cyber Officer, after the topic arose during scoping interviews, to determine whether the expectations and technical requirements of a Cyber Operator warranted the possibility for a Cyber Officer position. As it is too early to make an informed opinion to this regard, the evaluation analyzed present opinions and context around a potential Cyber Officer Position.

How:

The evaluation interviewed the case subject, queried survey and questionnaire respondents and discussed with interviewees.

Analysis Highlights

• The role of an officer has additional responsibilities that revolve around management as well as the strategic planning and implementation of initiatives.
• There is an expected level of cyber technical knowledge and understanding which takes years to develop, to be functional in the cyber domain.
• The case subject reported that current responsibilities of Cyber Operators, specifically on Active Cyber Operations, require higher levels of initiative and resourcefulness than those usually associated with roles of typical operators but more similar to roles typically associated with officers.
• If cyber is a domain of its own, to be fully developed as all other domains, such as land, sea and air, it was argued that a Cyber Officer should exist to enable further development of the Cyber Forces.
• Even with projected growth over the next few years, there may not be enough personnel to warrant a specific Cyber Officer role to oversee the Cyber Operator cadre, where career progression would be limited by available positions.
• A more generalized officer role has been argued by some interviewees, to ensure that they are not too focused on “cyber” and losing the big picture of Command, Control, Communications, Computers, Intelligence, Surveillance and Reconnaissance, and network operations. This pool would likely draw from the overarching Communications Electronics Engineering and Signals Officer occupations.

84% of survey respondents believe that DND/CAF would benefit from the creation of a Cyber Officer occupation. The majority of these respondents were at the tactical and operational levels.

However, the majority of interviews with senior management disagreed.

• 66 percent of interviewees thought there was no need for a Cyber Officer position.
• Regardless of the opinion on the need for a Cyber Officer opinion, 53 percent of interviewees referenced a need for better career management within the cyber domain, discussed further in Finding 10.

Back to Table of Contents

Delays in security clearances negatively affect the Cyber Forces.

• Overwhelming evidence indicates that the security clearance process and timelines for both civilian and military members ███████████████████████████████████████████████████████████████████████
• ███████████████████████████████████████████████████████████████████████████████████
• Interviewees reported that they are unable to attend necessary meetings due to security clearance requirements.
• Senior managers report being unable to recruit through certain channels such as Federal Student Work Experience Program students and co-op students, as they cannot show them the actual work they might be interested in doing due to the limitations of not having the proper security clearance requirements in place, ██████████████████████████   However, interviewees reported that OGDs are able to bring in several hundred students each summer, and have them screened within a month to the Top Secret level with polygraph. This could be further investigated by the Cyber Forces. 
• The issue is two-pronged – security clearances are slow, and the work needs to be accurately classified. Interviews indicated that a lot of activity is purportedly done at the Top Secret level that does not need to be, creating a self-imposed problem. Secret and unclassified work could be maximized to reduce the burden on limited security-cleared infrastructure, discussed in Finding 6.

82% of senior manager survey respondents indicated that security clearance approvals ███████████████████████████████████████

72% of survey respondents noted that security clearance approvals ████████████████████████████████████

██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████

Back to Table of Contents

CAF training is unable to keep pace with cyber needs; however, third-party cyber training lacks standardization and validation.

• Interviewees agreed that due to the complexity and constant evolution of the cyber domain, CAF training is not agile enough to keep pace. Third-party training is preferred by 91 percent of survey respondents but requires standardization and validation. 
• One interviewee noted that “[t]he Canadian Forces School of Communication and Electronics (CFSCE) alone cannot sustain the training of cyber.” This is supported by the Cyber Operator Occupation Briefing (2017) which noted that the Cyber Forces currently have a “Just enough just in time” training model.
• Third party training is thought to be more cost effective and may better enable the Cyber Forces to react to the rapid advancement seen in this realm. 
• Comments from survey respondents further highlighted this gap in training, as respondents were concerned that the lack of standard expected skillset paired with the absence of third-party course validation poses a risk that operators may not be meeting the level of skill required. 
• As discussed in Finding 9, ███████████████████ This presents the CAF with the alternative to choose between conducting cyber operations or providing training to the cyber force to build up necessary skill sets. Evidence shows that operations were prioritized.

The draft CMA Program Charter (2020) states “It is anticipated that some specialist training will be required to develop the necessary skill sets to enable this program -  the nature of which will be determined during options analysis.” 

In FY 2020/21, the CMAP is expected to complete a training needs analysis, formalize training and awareness framework, and issue common training guidance, all of which are reported to be on track.

Back to Table of Contents

Conclusions

As a new group, the Cyber Forces are actively engaged in numerous activities to establish Canada’s position in the cyber domain of warfare. DG Cyber and DGIMO, as the leads for cyber in DND/CAF, have produced and are currently working on various initiatives to set in place the foundational components needed for an effective Cyber Force in the future. However, unless appropriate attention is given to the cyber domain, the rate of implementation will continue to be constrained.

Back to Table of Contents

Annex A — Management Action Plan

Back to Table of Contents

Annex B — Gender-Based Analysis Plus (GBA+)

As per the TB Directive on Results (2016), Mandatory Procedures for Evaluation, this evaluation took into account the government-wide policy on GBA+ as it was deemed relevant.

The Evaluation Matrix included a Key Performance Indicator dedicated to GBA+: “GBA+ considerations are included in the hiring process”

The Cyber Forces Survey that was distributed included various GBA+ questions:

• Within the Self-Identification section, for the purpose of disaggregation of data, we asked whether respondents were military, civilian or ex-military; their age; primary and secondary language; sex; gender-identity; and ethnic origins.
• The questions below were included in the Training section:

• While the survey results were positive regarding GBA+ considerations, interview responses were varied. Many voiced that there were no unique GBA+ factors incorporated during planning or hiring, and no metrics are being tracked.
• One questionnaire response noted that DG Cyber is an equal opportunity employer and will hire anyone with various identity factors as long as they meet the educational and technical qualifications and can obtain the required security clearance. That same response indicated that they are currently assessing whether access to their services have any barriers to equality for those of differing abilities.  

Status of Women in Canada defines GBA+ as:
an analytical process used to assess how diverse groups of women, men and non-binary people may experience policies, programs and initiatives. The “plus” in GBA+ acknowledges that GBA goes beyond biological (sex) and socio-cultural (gender) differences. We all have multiple identity factors that intersect to make us who we are; GBA+ also considers many other identity factors, like race, ethnicity, religion, age, and mental or physical disability. 

Only approximately 5 percent of respondents to the Cyber Forces Survey identified themselves as women, and the majority of those were military members. In reflection of the CDS’ initiative to increase the presence of women in the CAF, certain initiatives could be explored to align with that intent. In the Office of the CDS Canadian Armed Forces Diversity Strategy (2016), it states “it is imperative that the Canadian Armed Forces (CAF) reflect the society it serves if we are to connect with Canadians and retain our relevance as a national institution. … Moreover, our operational experiences have demonstrated that diversity is a force enabler that enhances our operational effectiveness.” According to SSE Initiative 10, DND/CAF will fully leverage Canada’s diversity by promoting diversity and inclusion as a core institutional value across the Defence team.

Of respondents to the Cyber Forces Survey, for primary language statistics, 81 percent indicated English, 18 percent listed French, with 1 percent noting Mandarin. Approximately 15 percent of the respondents indicated an ethnicity other than white, illustrating a certain level of diversity in the Cyber Forces.

Back to Table of Contents

Annex C — Evaluation Methodology

Data Sources

The findings and recommendations of this report were informed by multiple lines of evidence collected throughout the conduct phase of the evaluation. These lines of evidence were triangulated with each other and verified with program managers to ensure their validity.  The research methodology used in the scoping and conduct of the evaluations are as follows:

Literature Review: 

As part of the planning phase of the evaluation, a preliminary document review was conducted to develop a foundational understanding of the Cyber Forces and Cyber Mission Assurance to determine the scope of the evaluation. This was expanded upon during the conduct phase of the evaluation, as other documents were examined to find data that would help in the assessment. Documents included: government websites; departmental administrative reports, program documents, both in draft and finalized; and external reports.

Short-Form Questionnaire: 

During the conduct of the evaluation, particular issue areas were identified for further clarification and information. As a result, a number of short form questionnaires (2 – 3 questions) were sent to key points of contact through email. Organizations contacted included: DGIMO, ADM(IE), VCDS, as well as the Communications Electronics Engineering and Signals Officer career managers.

Case Study: 

The evaluation team conducted a case study concerning the necessity of a cyber officer trade. This study drew upon military documentation, survey data, interview notes and administrative data. Further information concerning the case study can be found in the report. 

Interviews: 

The evaluation team conducted over 30 interviews with organizations internal and external to DND/CAF. These responses were aggregated to inform opinion and perspectives in support of the evaluation. Unless otherwise noted, reference to “senior program managers” only refers to those who are at the director level and above in DG Cyber and DGIMO. Organizations interviewed included:

Survey: 

The evaluation team conducted a survey over the month of October 2019. Certain questions were targeted for key individuals, such as senior managers, military members or cyber stakeholder organizations external to DG Cyber and DGIMO. Charts throughout the report reflect these population nuances in their titles. Unless otherwise stated, “survey respondents” refers to the entire survey population.

In order to engage a wide population for opinions, perspectives and GBA+ data, the evaluation developed a survey in English and French. Certain questions in the survey were developed for targeted members of the population, for example, senior managers, CAF members only, or responses from cyber stakeholders external to the Cyber Forces. These nuances are reflected in the charts illustrating the data throughout the report. 

The survey was administered to targeted organizations and individuals who work within the cyber domain or have connections to the Cyber Mission Assurance Program across DND/CAF. Survey distribution relied on the Points of Contact identified through research of the DND Directory. These individuals would then pass the survey to other relevant individuals or subordinates in their Chain of Command. 

The survey remained online for approximately one month, and had a total of 120 responses from ADM(IM), ADM(S&T), ADM(Mat), ADM(IE), JAG, SJS, VCDS, CFINTCOM, CJOC, CANSOFCOM and the Environmental Commands (CA, RCN, RCAF). To note, additional respondents from OUTCAN positions submitted surveys through Microsoft Word, as they did not have access to the DWAN. 

Site Visit: 

The evaluation team visited CFS Leitrim and conducted five interviews with individuals working at CFIOG, this included the Commander of CFIOG. Due to limited security clearances, the team was unable to get a tour of the site, but a number of presentations and briefings by CFIOG units were given to the evaluation team.  

Benchmarking: 

The evaluation team conducted a comparative analysis by benchmarking Canada’s Cyber Force with the military cyber groups of the United States, the United Kingdom and Australia. Using various indicators, data was collected from various online sources, such as government websites, to enable comparison. 

Focus Groups: 

The evaluation undertook a number of focus groups to capture data from targeted populations within the program areas of the population. In particular, a focus group was held with Director Information Management Engineering and Integration. 

Back to Table of Contents

Annex D — Evaluation Limitations

The limitations encountered by the evaluation, and mitigation strategies employed in the evaluation process.

Back to Table of Contents