Evaluation of the All Domain Situational Awareness Program – Defence Research and Development Canada

August 2021

1258-03-045 - ADM(RS)

Reviewed by ADM(RS) in accordance with the Access to Information Act.  Information UNCLASSIFIED.

Acronyms

Back to Table of Contents

Executive summary

This report presents the results of the evaluation of the All Domain Situational Awareness (ADSA) Science and Technology (S&T) program, conducted during Fiscal Year (FY) 2020/21 by Assistant Deputy Minster (Review Services) (ADM(RS) in compliance with the Treasury Board Policy on Results. The evaluation examines the relevance, effectiveness and efficiency of ADSA over its planned five-year period, FY 2015/16 to FY 2019/20.

ADSA Program Description

The chief objective of the ADSA research program is to generate knowledge to serve as a foundation for evidence-based advice regarding surveillance options of Canada’s northern approaches. Defence Research and Development Canada (DRDC) is the primary delivery agent, and provides advice to Level 1 (L1) organizations to inform capability options and decision making. The program was A-base funded starting in FY 2015/16, and was extended to FY 2020/21.

Evaluation Scope

This report aligns with the Future Force Design (Science, Technology and Innovation) Program Inventory element for National Defence. The evaluation examined the ADSA program as the largest and most representative component of DRDC defence S&T research programming.  Specifically, the assessment focused on the extent to which the ADSA program enables collaboration and generates knowledge, technologies and innovative options to address surveillance challenges, particularly in the Artic.

Summary of Findings

Relevance

There is an ongoing and demonstrable need for ADSA scientific and technological knowledge and advice.  The program is clearly aligned to governmental plans and priorities, and is directly informing policy development. ADSA research and development (R&D) provides relevant and innovative options for defence planning and future programs, and is informing the path forward in new large-scale research initiatives.

Effectiveness

The program made significant progress in generating options, knowledge and technology to inform defence planning and policy options. Innovative and promising research and solutions options have been achieved, particularly in system of systems.   Moreover, strong partnerships are in place with research partners for joint achievements, particularly with North American Aerospace Defence Command (NORAD) and the United States (US) Department of Defense.

Opportunities for improvement include the need to have a sufficient planning/formulation phase prior to program launch, and having stronger integration of capacity and security requirements with partners.  As well, there are gaps in the alignment of Department of National Defence (DND)/Canadian Armed Forces (CAF) operational priority setting and DRDC research planning.

Efficiency

The ADSA program spurred an innovative and successful integration of governance and shared outcomes. Its pan-domain approach led to better research, and is a significant achievement from the previous fragmented outcomes. 

However, gaps remain that impeded progress and optimal efficiency of the program, including in the areas of: procurement management in support of research operations; people and communications management; and expenditure planning.  In addition, there was a lack of strategic performance reporting to help inform corporate-level decisions in an orderly, consistent and performance-driven manner through the lifespan of the program.

The majority of gaps assessed by the evaluation were particularly present in the initial years of the program, and Assistant Deputy Minister (Defence Research and Development Canada) (ADM(DRDC)) has already made strides towards addressing them.

See Annex A for a complete list of findings and recommendations.

Back to Table of Contents

Evaluation scope

Coverage and Responsibilities

This evaluation was conducted in accordance with the TB Policy on Results and the DND/CAF Five-Year Departmental Evaluation Plan (FY 2017/18 to FY 2021/22), approved annually by the Performance Management and Evaluation Committee.

The evaluation focused on examining a component of DRDC programs that is the largest and most representative of defence S&T research.  As such, the assessment focused on the extent to which the ADSA program enables collaboration and generates knowledge, technologies and innovative options to address surveillance challenges in Canada, in particular those in the Arctic. The examination covers  the program’s planned lifespan of FYs 2015/16 to 2019/20, as well as the one-year extension of the program to FY 2020/21.  The ADSA program will sunset in FY 2021/22.  The outcomes from this evaluation, as well as a detailed lessons learned program report, will be used to inform the path forward for subsequent research programs – particularly the DNA initiative.

The objective of the evaluation is to assess the relevance, effectiveness and efficiency of the program.

The scope was developed in consultation with key stakeholders, including ADM(DRDC), Director General Research and Development Science and Engineering (DGRDSE), and Director General Research and Development Programme (DGRDP). Guidance was also received from ADM(DRDC) and ADM(RS).

Out of Scope

The following components were out of the scope for this evaluation:

• Research Centres (other than those related to ADSA)
• Mobilizing Innovation in Security and Defence
• Innovation for Defence Excellence and Security
• Canadian Safety and Security Program
• SCInergy 2020 and related strategic initiatives (although recent activities are identified towards building from ADSA outcomes)

The evaluation used multiple lines of evidence collected through qualitative and quantitative research methods (see Annex C for methodology and limitations).

Key Findings were developed and themed as per approved assessment indicators:

1. Relevance
2. Effectiveness
3. Efficiency

Back to Table of Contents

Program profile

Program description

The Defence and Security Science and Technology (DSST) program aims to provide DND/CAF with an advantage in knowledge, technologies and solutions for mission success while enabling a robust innovation base for Defence excellence and security.

The DSST program (PI 4.8) supports the core responsibility (CR) to “develop and design the future force through a deep understanding of the future operating environment and security risks to Canada and Canadian interests and to enhance Defence’s ability to identify, prevent, adapt and respond to a wide range of contingencies through collaborative innovation networks and advanced research” (Departmental Results Framework, CR 4 Future Force Design).

In support of Canada’s defence policy: Strong, Secure, Engaged (SSE) priorities, the Deputy Head approved A-base funding for the ADSA S&T program, which launched in FY 2015/16 with funding for five years ending in FY 2019/20. A one-year extension was put in place for FY 2020/21 to allow for the closure of certain projects and for the consolidation of results validation.

The main objective of the ADSA program is to generate knowledge to serve as a foundation for evidence-based advice regarding surveillance options of Canada’s northern approaches. Most R&D is achieved by contracting out to industry and academia. A number of projects have received funding via the ADSA S&T Program including:

• Over-the-Horizon Radar (OTHR);
• Canadian Arctic Underwater Sentinel Experimentation (CAUSE);
• Threat, Requirement and Gap (TRG) Analysis;
• Compression of the Tasking, Collection, Processing, Exploitation and Dissemination (TCPED) Cycle.

See Annex D and Annex E for details on the main program components and subcomponents..

DRDC is the primary delivery agent for S&T R&D investments and needs. In addition to serving the needs of DND/CAF, DRDC also serves most L1 organizations including Army, Navy and Air Force, Strategic Decision Support, Joint Force Development, Operations and Personnel.

Back to Table of Contents

Program funding profile

In July 2015, in support of SSE priorities, the Deputy Head approved a five-year funding allocation of $133 million, ending in FY 2019/20. ADM(DRDC) is the sponsor of the ADSA S&T program, on behalf of the Deputy Head and all L1s.  An additional $8.6 million was received through the ADM(DRDC) Business Plan for FY 2019/20, and a one-year extension was put in place for FY 2020/21 to allow for the closure of certain projects and for the consolidation of results validation (see Annex F for fiscal details).

Source: ADSA Financial Reports (2019/2020; 2020/2021).  *2020/21 actual expenditures are current as of January 28, 2021. ** FTE allocations only tracked from 2017/18 due to new database.

Back to Table of Contents

Program stakeholders

DND/CAF: ADM(DRDC), Royal Canadian Air Force (RCAF) (including Director General Space), NORAD, Royal Canadian Navy, Canadian Joint Operations Command (CJOC), Strategic Joint Staff (SJS), Chief of Force Development (CFD) and Assistant Deputy Minister (Policy) (ADM(Pol)).

Other Government Departments (OGD): National Research Council, Canadian Space Agency, Canada Border Services Agency, Canadian Coast Guard, Royal Canadian Mounted Police, Natural Resources Canada, Fisheries and Oceans, Transport Canada and Environment Canada.

Allied & International Partnerships: Pentagon Assistant Secretary of Defense for Research and Engineering, US NORAD and Northern Command, US Space Command, North Atlantic Treaty Organization (NATO), Five Eyes Partners (US, United Kingdom, Australia and New Zealand), US Department of Defense Research Labs.

Private Industry & Academia: More than 27 industry partners and 9 key academic institutions are engaged in research and analysis to support the development of options for enhanced domain awareness of air, maritime surface and subsurface approaches to Canada and the Arctic in particular. 

Back to Table of Contents

Relevance

Alignment with roles, priorities and results

Alignment with government and departmental roles and responsibilities

The ADSA S&T program is a key element of SSE, which commits to better situational awareness through a number of initiatives, such as:

• Prioritizing Arctic joint intelligence, surveillance and reconnaissance as a defence R&D priority to produce innovative solutions to surveillance challenges in the North. (Initiative # 69)
• Collaborating with the US on development of new technologies to improve Arctic surveillance and control, including renewal of the North Warning System. (Initiative # 109)

In addition, the National Defence Act (Sect. 4) establishes the Minister as responsible for research relating to the defence of Canada.

Alignment with government and departmental plans, priorities and expected results

• Defence and Security S&T Strategy
• Canada first Defence Strategy and Canada’s Defence Plan 2018-2023
• Speeches from the Throne and Mandate Letters (2015, 2019)
• Departmental Plans (Defence Innovation; Future Force Design)
• Departmental Results Reports (S&T)

Speeches from the Throne and Mandate Letters (2015, 2019) both indicate that the Government has mandated and legislated roles and responsibilities to focus on surveillance and control of Canadian territory and approaches, particularly the Arctic regions in order to defend Canadian sovereignty, protect North America and enhance international security.

ADSA is clearly aligned with the priorities of DND, and is denoted in the following Departmental Results:

• 4.1 – Defence capabilities are designed to meet future threats
• 4.2 – Defence and security challenges are addressed through innovative solutions.

Ongoing need

Prior to ADSA, stewardship for northern and remote surveillance was scattered and fragmented between CAF elements; ADSA is a comprehensive research program operating across the domains (e.g., air, maritime surface, maritime subsurface) and developing synergies.

ADM(Pol) considers the ADSA program to be a key foundational element of NORAD modernization and continental defence work, The results and strategic advice provided by the program, have been critical to informing policy development and advice.  For example, the ADSA program informs key departmental and governmental direction, including Budget 2021 and the current array continental defence policy.
 

Survey respondents agree and strongly agree that:

There is ongoing and demonstrable need for ADSA S&T knowledge and advice. 

Back to Table of Contents

Addressing strategic gaps

The Government of Canada has prioritized Security and Technology science planning and has also placed emphasis on Arctic files and on government-based science supporting high-level outcomes.  To respond to this additional responsibility,  DND made an investment of $133 million for the ADSA Program, from 2015 until 2020 to research and analyze, as well as identify, assess and validate technologies that can increase situational awareness in the air, on water and under water in the North.

ADSA has contributed to providing strategic advice to senior decision making within DND, the CAF, the US and NORAD.

Back to Table of Contents

Effectiveness

Business planning

Rapid and sudden funding

ADSA’s funding allocation was rapid and sudden, which resulted in the initiation of projects without sufficient plans/charters and an accelerated execution of the program.

Lack of predictable funding resulted in insufficient planning/ formulation phase

The Program was launched without a fulsome formulation phase, including  the setting of priorities and milestones, and without sufficient gap analysis and business planning. Project business cases and proposals were developed quickly, the funding was approved and issued while ADSA had not yet fully articulated expected outcomes.

Document analysis and key interviews indicate that the lack of a formal planning/definition phase is one of the critical issues that limited the ability to effectively establish goals and outcomes, slowed progress in later years and delayed research and operational progress, thereby affecting ADSA’s milestones, timeframes and schedules as well as clarity of goals and objectives. A one-year extension was put in place for FY 2020-21 to allow for the closure of certain projects and for the consolidation of results.

Some survey respondents commented that the lack of a clear formulation of research goals was an issue. Some interviewees noted while ADM(DRDC) officials helped inform departmental planning in northern surveillance planning, the ADSA-specific announcement seemed rushed.

Lack of predictable annual funding was also perceived as a barrier to a more effective program management cycle and long-term planning.

Challenges encountered in early years

Back to Table of Contents

Research and knowledge generation

Differing levels of progress across domains

• Advanced relatively more on the air side, in particular with NORAD.
• OTHR is the most promising, and it will be a key indicator for determining the degree to which ADSA has generated options that can be fully deployed.
• Logistics and security system differences between Canada and the US were sometimes an issue. Integration of research was somewhat more seamless with the US Air Force vs Navy.

Overall, survey respondents are satisfied that the knowledge and potential solutions generated by ADSA are on track to meet current and future CAF needs in terms of the quality and feasibility of ADSA-related projects, and are less satisfied with the timelines of projects.

Unique challenges of operating in northern and remote areas

Progress was made despite the fact that working in the Arctic has unique challenges.

Back to Table of Contents

Collaboration and leverage of partnerships

Successful partnerships are in place but with certain challenges

ADSA’s research and analysis was conducted in partnership with OGDs, Five Eyes allies, as well as engagements with academia and industry.  Collaboration and partnership were key to ADSA. The US is Canada’s primary ally as the two countries share common threats and there is a bi-national command in NORAD. The two countries share the Arctic and the responsibility of defending the North.

Collaboration existed also with NATO allies and northern nations that had some similar areas of focus (see Annex G for international context).  However, since the Program was specialized to northern approaches and involved highly classified projects and technologies, collaboration outside of Canada-US partnerships was limited.

Almost all interviewees and the majority of survey respondents agree that the ADSA program is fostering and effectively utilizing partnerships.

However, 68% of survey respondents indicated that the ADSA program faces internal and 37% external issues.

Challenges

Approaches to classified information

There is a need to establish more effective and consistent approaches to classified information, which adversely affect the timeliness of projects and their integration with partners.

• Majority of interviewees and survey respondents noted that ineffective security and classification mechanisms and classification tools negatively affected partnerships and led to projects being delayed (e.g., differing security clearance systems in the US vs Canada).
• Terms and conditions of partnerships with US/NORAD would benefit from review or update to better coordinate research activities.
• Not prepared to deal with the differing security classification systems ahead of time, which caused delays.
• Lack of access to vehicles/research vessels for classified contracts interrupted the progress of some projects.
• Lack of dedicated facilities in DND/CAF for classified work.
• Some concern about security challenges in collaborations with the industry/academic realm.

Back to Table of Contents

Strategic value-added

Gaps in the integration of DND/CAF operational planning and DRDC research planning

Canada’s defence and security rely on DRDC and its partners to identify advances in S&T that give the CAF the edge they need for building capacity to overcome evolving unconventional or novel threats and to carry out their missions effectively. The ability of the CAF to defend and operate in the Arctic as well as other extreme or hostile environments is a priority.

To help DRDC deliver the needed S&T solutions to DND/CAF, as well as provide the advice needed to maintain a technological and knowledge advantage to protect and defend Canada’s North, DND/CAF should have a proactive role to ensure clarity in their research and technology needs to DRDC.

DRDC

76% of survey respondents agreed that the ADSA projects are aligned with the wider priorities of the DSST program.

However, respondents and interviewees indicated that there are inconsistent messages about DRDC priorities and how they should structure their work.

• DRDC could engage more closely with the CAF.
• Workforce perceives a lack of clarity in how their work aligns with DRDC priorities and CAF operations.
• Work seems to be only on a project-by-project basis without prioritization.
• Awareness of DRDC priorities and strategic objectives not effectively in place.

A concern from research staff is that DRDC should rely less on serving the needs of individual projects or Generals across different elements, and instead use an integrated and prioritized approach. 

DND/CAF

The majority of survey respondents agree/strongly agree  that ADSA has helped to inform planning options in the field of defence S&T (e.g., Force Development planning,  CAF surveillance capacity in northern and remote areas).

However, there is a lack of proactive direction in terms of prioritization of solutions or technologies that should move forward:

• Priorities in the North are not clearly set/communicated.
• Resources and work remains on multiple research files that may not be a priority or reach fruition; lack of go/no go decisions.
• Lack of focussing resources on identified promising knowledge

Back to Table of Contents

Outcomes

Significant amount of research is still at an experimental stage

The potential capabilities generated by knowledge, prototype equipment and other technical R&D options can only be demonstrated once they move from an experimental to a feasibility test state.  Recognizing that ADSA is exploratory in nature, its investments are significant and have moved rapidly to get through the TRL and above. Some advice and solutions have been acted upon, and a significant amount of research is still at the cusp of demonstrating validity and field trials.

Virtually all interviewees expressed that ADSA’s contributions toward developing innovative solution options for surveillance cannot be validated until moved to validation/feasibility level.

Challenges

Some of the challenges faced to reach the level of feasibility/operationalization include:

• Lack of exploitation plans and lack of clear decision making;
• Some projects feasibility/deployment level delayed due to COVID-19 (e.g., OTHR);
• Scope of the surveillance problem is larger than the resources attached to ADSA;
• While recognized as an options-gathering initiative, the five-year window of ADSA is not sufficient to determine operational solutions and elevating the TRL for most projects.

It is essential:
Applicability against operational requirements should be ascertained going forward.

Back to Table of Contents

Efficiency

Contracting and procurement management

Gaps in support to research operations

Procurement management is essential to ensure contracting efficiency, fairness and transparency. Multiple factors contributed to perceived project delays and challenges, including lack of dedicated procurement expertise and a project management office to support research work and to provide guidance on intellectual property requirements. This was a particular challenge in the initial years of the program.

Processing time was often noted as an issue in both the document review and interviews.

• Of the estimated 492 contracts awarded to support ADSA, the evaluation found that the average processing time for contracts valued at $25,000+ is more than five months.
• Also of note is the lack of performance data due to DRMIS integrity issue between 2016-2018.
• Recurring ADSA status reports indicate that delays in contracting impeded the achievement of High-level Deliverables and added risk to the schedule.
• For almost 80 percent of interviewees, contracting rules are viewed as overly complicated and lacking in clarity.

Contracting processes were also impacted by factors unique to ADSA, including:

• Sensitive and classified nature of some projects;
• Physical and remote environment in the Arctic, where work is restricted to summer months, and purchase approvals needed well in advance or season is lost; and
• Challenges in engaging local workforce due to perceived reporting burden from Ottawa Headquarters.

90% of interviewees and 19% of survey comments indicated that ADSA faced procurement challenges, creating delays.

Back to Table of Contents

People management

Certain key perceived gaps to a healthy workforce and workplace were identified throughout the evaluation’s survey, interviews, working groups and site visits.  These include:

Unfunded support staff

The five-year $133 million ADSA allocation was not commensurate with an increase in the Salary and Wage Envelope (SWE) for additional scientific or support staff, or a dedicated project management office.  Moreover, a perceived lack of operational priorities resulted in existing staff undertaking ADSA research on top of their ongoing work. 

Back to Table of Contents

Financial management

Gaps in Business Planning

In the context of sudden funding, with annual allocation, flexibility and capacity to plan long term was limited.   Evidence indicates that:

• there is a lack of proactive budget planning and risk analyses, especially in initial years;
• multi-fiscal years’ nature of large complex projects not proactively considered;
• lack of flexibility to reallocate funding between years and long duration of large calls for proposals led to lapses. 

Detailed business planning occurred after program launch in year one, with contracts and implementation of research not starting in earnest until year two or later. This also created delays and slippages of expenditures into forward fiscal years. Some reallocations were absorbed out of the baseline S&T budget to compensate at the end.

Inefficiencies in expenditure planning within the five years affected project timelines, and was a factor in the program seeking a one-year extension to the planned close-out of projects.

Furthermore, there is a lack of data on the FTEs and SWE linked to ADSA, and the funding did not include a dedicated allocation for the SWE.
 

Back to Table of Contents

Governance and oversight

The ADSA program spurred an innovative and successful integration of governance and shared outcomes. Its pan-domain approach led to better research outcomes for the CAF across the elements.  This represents a significant achievement and progress from the previous fragmented outcomes.  ADSA was a key catalyst to move defence and security research away from a siloed stakeholder perspective to more holistic and  integrated results for surveillance knowledge generation across the domains (e.g., air, maritime surface and maritime subsurface).   

The majority of survey respondents and interviewees agree that the program has effective governance and oversight structures, with stakeholders/clients sufficiently represented.

Back to Table of Contents

Performance-based decision making

At the project level, progress is documented including data such as activities, budget, schedule, and scope and risk monitoring. Regular project Status Reports are used. This was enhanced with phased updates and Program Quarterly Reviews (PQR), or more frequently, as required. The results of the PQR from a financial, individual project status, critical timings, risk assessment and key milestones perspective were used to update the Program Status Report.

Progress was monitored on a project-by-project basis. Results were provided to senior management; the SRBs and the senior directors to monitor progress, understand the barriers, the constraints or the challenges. However, there was no external or neutral perspectives involved to monitor performance and provide informed scientific opinion to program management before proceeding to their decision making.

A holistic program-level approach in reporting could have been more effective in an integrated cross-portfolio program like ADSA S&T. The program lacks a holistic strategic reporting on its performance to help inform decisions in a consistent and performance-driven manner which affected providing real-time update on changes to the scope of the projects at the program level.

When the program was set up, it was not instrumented well, and it was not clear enough on what needed to be tracked in terms of performance metrics. So, performance measures and indicators which are fundamental to monitoring program progress and advancement to outcomes were not fully developed; work is needed to establish performance metrics in a consistent manner.

Additionally, the program did not have a robust strategic capability to analyze and integrate data from the various projects into meaningful and tactical tools and in a way that could be understood by all program stakeholders in order to inform decision making. These tools could include strategic dashboards, which would help in monitoring and analyzing the status of key performance indicators and help in setting the recommendations and direction.

The rapid launch of ADSA impacted the ability for an effective planning phase for the Program, including fully defining scope and setting milestones. This made validation of results against objectives difficult. Program stakeholders explained that since they find themselves at the back end of ADSA funding without knowing what comes next means that  the program might not  have had  clarity on the setting of goals and outcomes, and measuring progress/achievements against them. 

Back to Table of Contents

Conclusions

There is a demonstrable need for the ADSA program and its scientific and technological knowledge and advice. The program is clearly aligned to governmental plans and priorities, and is directly informing policy. ADSA R&D provides relevant and innovative options for defence planning and future programs, and is also informing the path forward in new large-scale research initiatives. To support these programs in achieving expected results in an effective and efficient manner, this evaluation has made recommendations in the following areas:

Back to Table of Contents

Annex A – Findings and recommendations

Back to Table of Contents

Annex B – Management Action Plan

ADM(DRDC) has proactively utilized lessons learned from the ADSA program to inform current and future programs, particularly the DNA program. Furthermore, ADM(DRDC) will engage DND Chief Financial Officer (CFO)  to establish more predictable funding levels for the DSST Program so as to better serve Partners’ requirements.

In addition, ADM(DRDC) launched SCInergy 2020 in fall 2018, a coordinated strategic effort to identify gaps and define recommendations to improve the overall management of the organization. All activities undertaken were linked to four pillars: Governance; People; Process; and Impact. Several SCInergy recommendations are in process or have been implemented that directly or indirectly respond to recommendations in the ADSA evaluation, including creating clear responsibilities and authorities for Program Management. ADM(DRDC) is committed to continuous improvement on all aspects of its program including finding ways to be more agile with procurement processes to increase the overall impact of science for the department.

In the years since ADSA was launched, ADM(DRDC) has revised its organizational structure to break down silos and create an internal synergy between the various program stakeholders. As such, ADM(DRDC) is now managing one integrated DSST Program. A new reinforced governance with the Science and Technology Investment Steering Committee, the DSST is aligned with CAF priorities.

Management Action

• ADM(DRDC) has established the directorate of DGRDP to centrally plan and coordinate all DSST program activities, including setting program objectives, defining program milestones and establishing delegated authorities.

OPI: Chief Of Staff (DRDC)    
Target Date: COMPLETE

• As unexpected influxes of ADSA funds caused significant challenges and prevented adequate planning processes, to prevent this from occurring again, the CFO will be engaged to re-evaluate ADM(DRDC) baseline funding requirements.

OPI: Chief of Staff (DRDC)  
Target Date: March 2023 (a one-page status update will be provided by March 2022)
OCI: Assistant Deputy Minister (Finance)

Management Action

• ADM(DRDC) will continue to strengthen strategic partnerships with the US, NORAD and other stakeholders through analysis of current agreements and tools; complete a decision tool for prioritizing strategic partnerships; provide key recommendations to ADM(DRDC) for future partnerships and associated processes.

OPI: DGRDSP  
Target Date: June 2023 (a one-page status update will be provided by March 2022)

• ADM(DRDC) will complete options analysis and subsequently present the proposal for the Science & Technology Enhanced Research Capability (a Secret and Top Secret Research network enabling the sharing of research and information with partners and allies) to DND’s Project Management Board for Project Approval (Definition) and subsequent transfer of project responsibility to Assistant Deputy Minister (Information Management).

OPI: DRDTIS  
Target Date: April 2022 

Management Action

• Building upon an independent study, action has been completed on increasing dedicated corporate communications resources and implementing improved internal communications processes, ADM(DRDC) will continue to enhance its internal communications capabilities.

OPI: DRDSRPM  
Target Date: COMPLETE

• ADM(DRDC) will develop a strategic HR plan to deal with challenges and opportunities related to talent management, succession management and recruitment.

OPI: DRDSRPM    
Target Date: March 2022

• ADM(DRDC) will continue to strengthen its application of the established R&D Capability Health Assessment process to assess its capacity to deliver current R&D capabilities and anticipate future capability needs. This broader assessment will enable DRDC to identify capability areas targeted for growth, sustainment and/or divestment. This process will address workload challenges, reallocate resources to priorities and fill the identified gaps.

OPI: DGRDSE     
Target Date: March 2023 (a one-page status update will be provided by March 2022)

Management Action

• The establishment of the Science and Technology Investment Steering Committee in December 2020, and the process by which L1 R&D requirements are assessed and prioritized against DND priorities has been completed.

OPI: DGRDP  
Target Date: COMPLETE

• The establishment of the collection, management and prioritization process of L1 partner requirements has been completed.

OPI: DGRDP  
Target Date: COMPLETE

Management Action

• ADM(DRDC) has implemented new DSST program governance structure in December 2020 to steer R&D investment, assess and prioritize R&D activities and deliver large-scale research programs.

OPI: ADM(DRDC)    
Target Date: COMPLETE

Management Action

• DRDC procurement processes have been improved and/or implemented to directly support major DSST programs or initiatives. The DRDC Procurement team has implemented a fully harmonized procurement process that is in accordance with all departmental and Treasury Board Secretariat policies and directives, and implemented an electronic procurement ticketing application to improve accuracy and efficiency of the procurement request process. DRDC has also established a strong Community of Practice amongst the regional procurement offices and the senior procurement leadership within DRDC.

OPI: DRDCS     
Target Date: COMPLETE

• In the designing of future major research initiatives, DRDC has established the practice of incorporating procurement and project management staff as part of the planning process.

OPI: DRDCS  
Target Date: COMPLETE

Management Action

• ADM(DRDC) will establish and enhance strategic level performance dashboards that cover all program activities to inform decision making at various DRDC committees, including the Science and Technology Program Advisory Committee.

OPI: DGRDP   
Target Date: December 2021

• DRDC will expand its performance management framework to incorporate high-level performance requirements for the DSST program.

OPI: DRDSRPM    
Target Date: March 2023 (a one-page status update will be provided by March 2022)

Back to Table of Contents

Annex C – Evaluation methodology and limitations

The evaluation findings and recommendations were informed by multiple lines of evidence and qualitative and quantitative research methods collected throughout the conduct phase to strengthen rigour and ensure the reliability of information and data supporting findings. These lines of evidence were triangulated and draft findings shared with program management as part of a collaborative process to ensure accuracy and impartiality.  The research methodology used in the scoping and conduct of the evaluations are as follows:

Document and literature review, administrative and financial data review

A preliminary review of the foundational documents was conducted during the planning phase which supported developing a comprehensive understanding of the program and informed the development of the scope and the evaluation matrix. The review was expanded extensively during the conduct phase of the evaluation, and the program provided a database of planning, performance measurement, financial and HR, and other documents that were requested for data gathering and analyses. The evaluation team reviewed over 200 documents, including; departmental administrative reports; program documents; program status reports; minutes of meetings; departmental plans; results reports; policies and mandates applicable to the program; and internal and external websites.

Key informant interviews

The evaluation team worked with a program liaison to identify interviewees. There was a total of 36 interviews conducted with a range of stakeholders including ADM, DG, DSTCE, DRDC DCs, TCPED, CAUSE, OTHR, DND/CAF partners: NORAD, RCAF. A number of program stakeholders had confidential discussions with the evaluation team. Interview data was thoroughly captured which allowed a robust thematic analysis to be performed. The data was cross-referenced against other lines of evidence. 

Survey

A bilingual web-based survey was developed and administered to a wide range of stakeholder respondents internally and externally. The survey focussed on assessing the effectiveness and efficiency of the program. The survey was developed using Snap Survey Software and conducted using the internal (Defence Wide Area Network) and external Snap WebHost platforms. The web survey link was distributed by email to personnel/ stakeholders with a variety of levels and of past or current involvement in the program: CAF members, finance/project management, scientists, procurement, etc. The online survey was live for two weeks (from December 4 – December 18, 2020). During this timeframe one reminder email was sent. Overall, there was a strong response rate of ~ 48 percent (41/86).

Case studies and site visits

In order to capture fulsome and operational information pertaining to programs, the evaluation teams typically conduct site visits.  The evaluation team employed a case study approach which included conducting a site visit to gain additional insight into some of the projects’ S&T activities and work in progress. The evaluation team undertook a tour of the antenna arrays models at Ottawa Research Centre near Shirley’s Bay. The tour covered the Transmit Area 2 and the Receive Area 6. Although originally it was intended to cover all four regions where DRDC research centres are located: Alberta, Ontario, Quebec and Atlantic, the pandemic and associated travel restrictions prevented the evaluation team from conducting the fulsome case study portion of the assessment methodology.

Focus group interviews

Due to the pandemic, the evaluation team conducted only one focus group session to capture direct in-person information from DRDC scientists from Ottawa Research Center. The focus group was also followed up with confidential discussions that were requested by either some of the focus group participants or other stakeholders who were referred to the evaluation management by their colleagues to share their perspectives on some aspects of the evaluation.

Evaluation Limitations

The limitations encountered by the evaluation and mitigation strategies employed in the evaluation process are outlined below.

Back to Table of Contents

Annex D – Program components

Overview

The ADSA initiative applies S&T research to identify, develop and demonstrate potential solutions for enhanced surveillance and detection capabilities across space, air and maritime domains for the CAF, particularly in the North. Projects in support of the ADSA program include:

Back to Table of Contents

Annex E – Program subcomponents

The following contracts were awarded for the ADSA projects:

Back to Table of Contents

Annex F – Detailed funding profile

ADSA – Overall situation

Source: ADSA Financial Reports (2019/2020; 2020/21)

Note 1. Additional $8.6 million was received through ADM(DRDC) L1 Business Plan for 2019/20.
Note 2. 2020/21 actual expenditure is currently spent as of January 28, 2021.
Note 3. A one-year extension (2020/21) was put in place to allow for the closure of certain projects and for the consolidation of results validation.

Back to Table of Contents

Annex G – International context

Research relevant to Domain Situational Awareness in major defence R&D agencies of the Five Eyes nations  

Back to Table of Contents