Evaluation of CAF Operations – Search and Rescue

Acronyms and Abbreviations

ADM(Fin)
Assistant Deputy Minister (Finance)
ADM(IM)
Assistant Deputy Minister (Information Management)
ADM(RS)
Assistant Deputy Minister (Review Services)
CA
Canadian Army
CAF
Canadian Armed Forces
CANSOFCOM
Canadian Special Operations Forces Command
CDS
Chief of the Defence Staff
CJOC 
Canadian Joint Operations Command
CMP 
Chief of Military Personnel
Comd
Commander
C Prog 
Chief of Programme
DART
Disaster Assistance Response Team
DG 
Director General
DGE  
Director General Evaluation
DND  
Department of National Defence
DRF
Departmental Results Framework
DRO  
Disaster Relief Operation
ELT
Emergency Locator Transmitter
FSAROGC
Federal SAR Operations Governance Committee
FY
Fiscal Year
GC 
Government of Canada
HSS
Health Services Support
JRCC
Joint Rescue Coordination Centre
JTF 
Joint Task Force
L1
A DND/CAF organization with direct report to the DND Deputy Minister and/or Chief of the Defence Staff
MEDEVAC
Medical Evacuation
MHz  
Megahertz
NATO
North Atlantic Treaty Organization
NFLD 
Newfoundland
NORAD
North American Aerospace Defence Command
OAG  
Office of the Auditor General
OCI
Office of Collateral Interest
OGDA
Other Government Departments and Agencies
Op 
Operation
OPI
Office of Primary Interest
R2MR
Road to Mental Readiness
RCAF 
Royal Canadian Air Force
RCN  
Royal Canadian Navy
Rdns 
Readiness
RP
Response Posture
S&T
Science and Technology
SAR
Search and Rescue
SMMS
SAR Mission Management System
TB
Treasury Board
UN 
United Nations
VCDS
Vice Chief of the Defence Staff

Back to Table of Contents

Executive Summary

This report presents the results of the Assistant Deputy Minister (Review Services) (ADM(RS)) evaluation of the Canadian Armed Forces (CAF) Operations, which was conducted in compliance with the Treasury Board (TB) Policy on Results between January 2019 and November 2019. The evaluation examined the relevance and performance of the program over a five year period, fiscal year (FY) 2014/2015 through 2018/2019. The evaluation had a principal focus on Search and Rescue (SAR) operations in response to a request from Commander (Comd), Canadian Joint Operations Command (CJOC).

Program Description

The CAF Operations, for the purpose of this evaluation, are comprised of the following three programs, as described in the Department of National Defence’s (DND) Program Inventory: 1.1 Operations in Canada; 1.2 Operations in North America; and 1.3 International Operations. Although there are other force employers, such as North American Aerospace Defence Command (NORAD) and Special Operations Forces, the operational programs are the collective responsibility of CJOC as program lead. CJOC is responsible for conducting domestic, continental and expeditionary defence operations for which it is the force employer. It is also responsible for providing operational support to Canadian safety and security operations.

Overall Assessment

  • CAF operations are required to defend Canada and Canadian interests.
  • The use of SAR capabilities for routine patient transport is not within the CAF mandate.
  • Resources assigned to CAF operations programs are fragmented, and prevent CJOC from understanding the full range of expenditures, hindering their ability to make assessments on economy and efficiency.

Relevance

Canada has a need for defence operations. The CAF conducts operations to meet Canada’s defence and security requirements. Global security environment trends undermine the traditional security once provided by Canada’s geography. Defending Canada and Canadian interests thus, not only demands robust domestic defence, but also requires active engagement abroad, both continentally and overseas.

Effectiveness

The evaluation found that CAF Operations have consistently met operational requirements when assigned to operations ranging from domestic and continental disaster relief and security operations to international combat missions.

SAR personnel are placed under a high level of stress due to the nature of their job, and have indicated that mental health is a serious and growing concern. SAR employees are not provided with pre-and post-operational support.

Ambulance services, including air ambulance services, are the responsibility of the provincial governments under the Canada Health Act, and these services are normally contracted by the provincial governments. While critical medical evacuations are considered DND’s humanitarian mandate, the use of SAR capabilities for routine patient transport is not within the CAF mandate.

Economy and Efficiency

The evaluation was unable to assess the economy and efficiency of the CAF Operations Programs. This highlighted the lack of oversight of the assignment of CAF Operations expenditures to Program segments. It is recognized that Level 1s (L1) are accountable for expenditures under their direct control but there is a lack of overview and accountability for Program activities delivered by multiple L1s. The department would benefit from clarifying allocations, monitoring and reporting of Program expenditures. It is suggested that regular L0 and/or L1 level discussions are held to: (1) ensure efficient utilization and monitoring of resources at the program level; (2) review resource utilization by multiple L1 organizations contributing to a particular Program; and (3) provide guidance to support Chief of Programme (C Prog) review efforts.

Key Findings and Recommendations

Table 1. Summary of Key Findings and Recommendations.

Key Findings Recommendations

Relevance

1. There is a continuing need for CAF Operations to support the federal government’s roles, responsibilities and priorities.

 

Effectiveness

2. Although the CAF achieves a stated 97.8 percent of its operational objectives, there is a lack of precision in the objectives and a lack of clarity in their measurement.

1. It is recommended that CJOC continue their efforts in reviewing and revising operational objectives to ensure they are clearly defined, use a combination of both quantitative and qualitative indicators that are measurable and meaningful, and to the extent possible consistent and coherent across the entire spectrum of operations.

3. CAF planning and preparation for a potential request for domestic aid increases CAF responsiveness to Canadians.

 

4. The provision of Provincial Air Ambulance services to Newfoundland is not considered part of the CAF’s SAR Mandate.

2. It is recommended that recovering the costs for the provision of Air Ambulance services for the province of Newfoundland be explored.

 

5. The two individual SAR response postures have not been reviewed since 1958. As a result, they have negative effects on force readiness, SAR crew quality of life and the ability to conduct training.

3. It is recommended that CJOC assess the need for two separate SAR response postures.

6. SAR lacks a data analytics capability.

4. It is recommended that CJOC identify the SAR analytics requirements, and ensure the new SAR Mission Management System is capable of capture and analysis of the required data.

7. SAR crews are not uniformly provided with pre- and post-operation support.

5. It is recommended that CJOC enhance the provision of proactive mental health practices to SAR personnel.

Economy and Efficiency

8. At this time, it is not possible to assess the economy and efficiency of CAF Operations programs. It appears that L1 organizations outside of CJOC are unclear in terms of the alignment of CAF Operations expenditures to program segments.

6. It is recommended that CJOC, as the lead for the CAF Operations Programs:

a) Develop and implement a robust process for: i) reviewing and adjusting current and future alignments to CJOC Program segments; and ii) ensuring that CJOC provide ongoing oversight for all expenditures related to CAF Operations.

b) Work with C Prog on developing guidance to support requests for new L1 segments in the Operations Programs.

Observation: Increased use of GPS-capable beacons will improve SAR efficiency.

 

Table 1. Summary of Key Findings and Recommendations. This table outlines the evaluation’s key findings along with their associated recommendations.

Table 1 Details - Summary of Key Findings and Recommendations.

Note: Please refer to Annex A—Management Action Plan for the management responses to the ADM(RS) recommendations.

Back to Table of Contents

1.0 Introduction

1.1 Context for the Evaluation

This report presents the results of the ADM(RS) evaluation of CAF Operations which was conducted between January 2019 and November 2019 in compliance with the TB Policy on Results. As per the TB policy, the evaluation examined the relevance and performance of the program over a five-year period, FY 2014/2015 through 2018/2019. The evaluation may be used to inform future senior management discussions regarding CAF Operations.

An ADM(RS) evaluation of CAF Operations was recently completed in November 2017. The completion of this evaluation, as well as seven other related evaluations and audits, provided an in-depth examination of the relevance and overall performance of the CAF operation programs. After a formal request from and consultations with Comd CJOC, the evaluation’s primary focus was on SAR activities.

The evaluation was supported by representatives from L1 organizations and specific divisions, including CJOC, Assistant Deputy Minister (Finance) (ADM(Fin)), Military Personnel Command, Royal Canadian Navy (RCN), Canadian Army (CA), and Royal Canadian Air Force (RCAF), providing advice and support to ADM(RS).

1.2 Program Profile

1.2.1 Program Description

The CAF Operations, for the purpose of this evaluation, are comprised of the following three programs, as described in DND’s Program Inventory: 1.1 Operations in Canada; 1.2 Operations in North America; and 1.3 International Operations. Although there are other force employers, such as NORAD and Special Operations Forces, the operational programs are the collective responsibility of CJOC as the program lead. CJOC is responsible for conducting domestic, continental and expeditionary defence operations for which it is the force employer. It is also responsible for providing operational support to Canadian safety and security operations.

The Operations in Canada Program involves detecting, deterring and defending against threats to or attacks on Canada, and assisting civil authorities, law enforcement agencies and non-governmental partners in responding to domestic emergencies including terrorism. This program also includes conducting SAR operations.Footnote 1  As a SAR service provider, DND/CAF is responsible for providing coordination of maritime and aeronautical SAR operations anywhere within Canada’s designated SAR area of responsibility. The objective of the SAR program is to prevent loss of life and injury though SAR alerting, responding and aiding activities using public and private resources.

The Operations in North America Program involves detecting, deterring and defending against threats to or attacks on North America in partnership with the United States, including through North American Aerospace Defense Command and meeting our North American Aerospace Defense Command obligations.Footnote 2

The International Operations Program involves leading and/or contributing to international peace operations and stabilization missions with the United Nations, North Atlantic Treaty Organization (NATO) and other multilateral partners. Operations include: coalition efforts to deter and defeat adversaries including terrorists; capacity building to support the ability of other nations to contribute to security abroad; supporting the security of Canadians oversees; and providing assistance and response to international disasters or major emergencies.Footnote 3

1.2.2 Program Objectives

The CAF Operations Programs support the following Departmental Results:

1.1 Canadians are protected against threats to and attacks on Canada;

1.2 People in distress receive effective SAR response;

1.3 Canada’s Arctic sovereignty is preserved and safeguarded;

1.4 North America is defended against threats and attacks; and

1.5 Canadian Armed Forces contribute to a more stable and peaceful world.

1.2.3 Stakeholders

The DND/CAF elements that may be stakeholders or partners, depending on the circumstances, include the RCAF, RCN, NORAD, CA and Canadian Special Operations Forces Command (CANSOFCOM), all of which may operate together in operations. Other key partners include the Assistant Deputy Minister (Materiel), Chief of Military Personnel (CMP), Assistant Deputy Minister (Infrastructure and Environment) and Assistant Deputy Minister (Information Management) (ADM(IM)). Allied forces and coalition partners in operations may also be considered stakeholders.

1.3 Evaluation Scope

1.3.1 Coverage and Responsibilities

An ADM(RS) evaluation of CAF Operations was recently completed in November 2017, which included an in-depth examination of the relevance and performance of the CAF operations. The completion of this evaluation, as well as seven other related evaluations and audits, and a formal request from and consultations with Comd CJOC led to a decision to focus this evaluation on SAR activities.

The evaluation covered the period of FYs 2014/15 to 2018/19 and examined the relevance and effectiveness of the program with a focus on the SAR capabilities of the CAF. The scope of the current evaluation focused on three main themes.

  1. Degree to which people in distress receive an effective SAR response;
  2. Degree to which the operational objectives are indicative of mission success; and
  3. Degree to which the CAF supports Canadians in need.

1.3.2 Resources

Table 2 denotes the expenditures for Programs 1.1, 1.2 and 1.3 for the two years that expenditures have been tracked, as Programs in the Program Inventory.Footnote 4  Planning for expenditures is challenging for CAF operations as this is dependent upon government decisions to assign operations to the CAF. This is particularly true for international operations. For example, planned expenditures for Program 1.3 for FY 2018 was $298.2 million but actual expenditures were $546 million.

Table 2. Total Program Expenditures.

Program

Actual Expenditures FY 2017/18

Actual Expenditures FY 2018/19

Planned FY 2019/20

1.1 Operations in Canada

$146,762,549

$130,861,641

$135,902,647

1.2 Operations in North America

$76,806,359

$61,518,816

$59,552,511

1.3 International Operations

$433,332,963

$546,028,875

$248,266,514

Total

$656,901,871

$738,409,332

$443,721,672

Table 2. Total Program Expenditures. This table denotes the Program expenditures for FY 2017/18 to FY 2018/19.

Table 2 Details - Total Program Expenditures.

1.3.3 Issues and Questions

In accordance with the Treasury Board Secretariat Directive on Results (2016),Footnote 5  the evaluation report addresses the evaluation issues related to relevance and performance. The methodology used to gather evidence in support of the evaluation questions can be found at Annex B. A logic model linking activities and results to outcomes is also provided at Annex C.

Back to Table of Contents

2.0 Findings and Recommendations

2.1 Relevance

Key Finding 1: There is a continuing need for CAF Operations to support the federal government’s roles, responsibilities and priorities.

Canada has a need for defence operations. The global security environment marks the shifting balance of power, the changing nature of conflict and the rapid evolution of technology. Increasingly, threats, such as global terrorism and those in the cyber domain, transcend national borders. These trends undermine the traditional security once provided by Canada’s geography. Defending Canada and Canadian interests thus not only demands robust domestic defence but also requires active engagement abroad.Footnote 6 

The CAF conducts operations to meet Canada’s defence and security requirements. In 2018, the CAF conducted 34 named operations, 24 of which were international. In addition, the CAF conducts activities that are not conducted under named operations such as SAR and defence and security support.

Defence is a core federal government responsibility as articulated in the Constitution Act, 1867,Footnote 7  which defines and outlines the responsibilities and duties of the federal government, including DND/CAF. Furthermore, Article 17 of the National Defence Act establishes DND and the CAF as separate entities, operating within an integrated National Defence Headquarters, as they pursue their primary responsibility of providing defence for Canada and Canadians. Moreover, theAct assigns the Defence Minister as the authority in charge of all matters relating to defence, including the Land, Naval and Air Services of Canada.

There is a continuing need for the CAF to conduct SAR missions, as mandated by the Government of Canada (GC). CAF SAR personnel prevent loss of life and injury through SAR alerting, responding and aiding activities through these missions.

SAR incidents are classified in accordance with five categories:

Over the period of FYs 2014/15 to 2018/19, SAR personnel responded to 2,204 Category 1 and 2 aeronautical, maritime and humanitarian incidents across the country (Table 6).

2.2 Performance—Achievement of Expected Outcomes (Effectiveness)

Key Finding 2: Although the CAF achieves a stated 97.8 percent of its operational objectives, there is a lack of precision in the objectives and a lack of clarity in their measurement.

The CAF conducts force employment operations upon direction from the GC. The Government liaises closely with the CAF prior to initiating operations to identify the CAF capabilities best suited to meet the demand. This close liaison is a key component in achieving an average of 98.4 percent of their operational objectives in the last three years, as illustrated in Table 3.Footnote 8  As the GC directly tasks the CAF at the strategic level, measuring the operational success of the CAF is a complex challenge, and efforts to ensure strong performance measurement continue to evolve.

Table 3. Results of operations.

Expected result

Performance Indicator

Target

2015/16

2016/17

2017/18

2018/19

The application of Canadian Defence and Security capabilities continuously protects the sovereignty of Canada, the values of Canadians and the interests of the GC against risks imposed by armed threats.

Percentage of Defence Combat and Support Operations that have successfully achieved their operational objectives.

 

90-100%

 

97%

 

98%

 

98.50%

 

100%Footnote 9 

Table 3. Results of operations. This table indicates the percentage of operations that achieved their operational objectives from FY 2015/16 to 2018/19

Table 3 Details - Results of operations.

An analysis of the objectives that were not achieved during the time period covered by this evaluation indicated that these objectives were beyond the control of CJOC and thus the CAF. For example, 60 percent of the objectives not achieved were, “to assist a nation in achieving national and regional peace and stability.”Footnote 10  This is a difficult objective to measure, let alone achieve, as there are multiple contributing factors outside the control of the CAF.

Operational objectives analyzed from FYs 2014/15 to 2018/19 were found to be limited. In 2018, there were 25 CAF operations with listed objectives. Of the 75 operational objectives listed, only one was quantitative. Therefore, 98.7 percent of the objectives were qualitative in nature, yet rated on a numerical scale from 1 to 10. This rating scale, however, was not standardized and based on the subjective opinion of the Comd of each operation. In addition, a methodology to support this rating scale does not exist and, as a result, the difference between values remains unclear. For example, for an operation’s objective, it was unclear how to differentiate a rating of 7 versus 8.

Measuring the success of operational objectives is a complex challenge, yet progress continues to be made. CJOC, through its ongoing performance and results framework optimization, aims to identify achievable and measurable operational objectives. New operational reports include tangible evidence of the success of operations, utilizing both qualitative and quantitative objectives. Having the ability to measure performance supports evidence-based decision making and allows the CAF to showcase success by providing justification for additional funding requests. In addition, identified challenges associated with a CAF Operation could be used for consideration of future operational objectives.

ADM(RS) Recommendation

Key Finding 3: CAF planning and preparation for a potential request for domestic aid increases CAF responsiveness to Canadians.

The CAF supports Canadians in the event of a Disaster Relief Operation (DRO) by providing a rapid, scalable response to federal, provincial and territorial requests for DRO assistance. The Whole of Government response for a DRO is coordinated by Public Safety through the Government Operations Centre. Operation (Op) LENTUS is the CAF contingency plan that outlines the joint response to provide support for Humanitarian Assistance and Disaster Response to provincial and territorial authorities in the case of a major natural disaster that overwhelms their capacity to respond.

During the reporting period, between the calendar years 2015 and 2018, there were 18 requests for CAF support. All requests were substantial and in circumstances posing a high risk to Canadians. As such, they were designated as Op LENTUS. An analysis of the Op LENTUS data during the evaluation period presents an upward trend of requests for CAF to support Canadians, as demonstrated in Table 4.

Table 4. Operation LENTUS expenditures from 2015 to 2019.

Year

Operations

Examples

Expenditures

2019 (until April 1)

6

New Brunswick, British Columbia floods, fires in BC and Northern Ontario and Kashechewan First Nations.

 $2,761,862

2018

4

New Brunswick ice storm, Ontario and Quebec floods and BC wildfires.

$14,350,300

2017

1

Alberta wildfires

$315,000

2016

3

Ontario floods and Saskatchewan and British Columbia wildfires

$1,269,000

2015

4

Ontario and Manitoba floods

$149,000

Table 4. Operation LENTUS expenditures from 2015 to 2019. This table lists the Op LENTUS expenditures from 2015 to 2019.Footnote 11 

Table 4 Details - Operation LENTUS expenditures from 2015 to 2019.

Based on evidence from the document review, it was found that the CAF is able to achieve success on operations in part due to their standard for high readiness, their ability to predict certain types of natural disasters, training and liaising with regional authorities. By anticipating and planning for requests for assistance, the CAF is able to respond rapidly and successfully. The CAF maintains high readiness units assigned to each Joint Task Force’s (JTF) area of responsibility, which requires the CAF to have a reconnaissance platoon depart within eight hours of receipt of orders from the JTF Comd. For example, a Chief of the Defence Staff (CDS) order was sent on April 20, 2019. CJOC quickly followed up and then the JTF Atlantic ordered a reconnaissance team, which was deployed to the incident command post and a platoon to Fredericton by 1600 hours that same day.

In addition, certain types of support requests can be predicted as they occur each year. For example, during flood and fire season, it is anticipated that more requests for support will be received. Analysis of the CA training documentation shows an alignment of the conduct of support exercises with the timing of potential support requests. The ability to utilize personnel already on exercise, especially reservists, reduces their activation time, increasing the ability of the CAF to respond to large scale support requests such as floods.Footnote 12  By planning and preparing for these seasons, the CAF is able to successfully respond to the requests for support. The CAF also enhances its responsiveness through close liaison with regional authorities to gain as much forewarning as possible of potential requests for support.

Memoranda of understanding and GC policies and directives also permit the CAF to provide support to other federal departments and agencies and provincial authorities that includes assistance to law enforcement agencies and the Department of Fisheries and Oceans.

Key Finding 4: The provision of Provincial Air Ambulance services to the province of Newfoundland is not considered part of the CAF’s SAR Mandate.

The CAF is mandated to respond to aeronautical incidents and to coordinate SAR activities. The use of SAR capabilities for routine patient transport does not fall within the CAF mandate. Ambulance services, including air ambulance services, are the responsibility of the provincial governments under the Canada Health Act. As routine medical evacuations or hospital-to-hospital transfers are beyond the mandate of a humanitarian incident, they are not considered to be SAR incidents, and should be dealt with by provincial or territorial authorities.Footnote 13 

While not mandated, CAF SAR units are called upon by the province of Newfoundland (NFLD) to provide routine patient transport, a costly and time consuming responsibility. To demonstrate this, Table 5 denotes the number of times the RCAF conducted patient transfers from one hospital to another in NFLD. During the time period covered by the evaluation, hospital transfers in NFLD accounted for an average of 33 percent of the SAR humanitarian flights. This evaluation found that this is a challenge only in NFLD, and does not appear to be prevalent in other Provinces and Territories.

Table 5. Number of RCAF Hospital-to-Hospital personnel transfers in Newfoundland.

 Year

Hospital-to-Hospital Medical Evaluations (MEDEVAC)

Total Joint Rescue Coordination Centre (JRCC) Halifax Humanitarian Missions

Hospital-to-Hospital MEDEVACs Percentage

2015

20

70

28.60%

2016

37

113

32.70%

2017

27

96

28.10%

2018

34

77

44.20%

Table 5. Number of RCAF Hospital-to-Hospital personnel transfers in Newfoundland. This table identifies the number of hospital transfers conducted in NFLD by the CAF from 2015 to 2018.Footnote 14 

Table 5 Details - Number of RCAF Hospital-to-Hospital personnel transfers in Newfoundland.

Often, requests for SAR air ambulance services are received when weather and visibility is too poor for the province’s own air ambulances. With limited SAR resources, there is also a concern within the CAF that these hospital transfers raise the risk of SAR resources not being available for their primary mission, which involves aeronautical SAR in support of people in distress within their area of responsibility.

The CAF currently provides patient transfer services without cost recovery. When NFLD is not asked to reimburse SAR services, there is no incentive for them to acquire sufficiently capable air ambulance services. This reliance on DND to provide this service could be perceived as placing DND in direct competition with Canadian industry. A possible solution that could address this issue is the identification of the air ambulance service as Op LENTUS. The provision of ambulance services as aid by the federal government to a province enables clear identification of the resources utilized, which aids in cost recovery activities.

ADM(RS) Recommendation

Key Finding 5: The two individual SAR response postures have not been reviewed since 1958. As a result, they have negative effects on force readiness, SAR crew quality of life and the ability to conduct training.

The CAF is mandated to conduct aeronautical SAR in support of people in distress within its area of responsibility. To fulfill this mandate, SAR crews are obligated to respond within set Response Posture (RP) standards. RP is measured as the time from when a tasking is received to the crew being airborne. During normal working hours, a fully operational aircrew is to be airborne within 30 minutes of receipt of an alert (RP30), and at all other times a fully operational aircrew is to be airborne within 2 hours of an alert (RP2hrs). These standards of RPs have existed since 1958Footnote 15  without being evaluated or assessed on credibility or relevance. The RPs identified were simply the RCAF readiness abilities at that time. No evidence could be found that an in-depth analysis was completed when these times were turned into standards.

Table 6 denotes that the CAF averaged 441 Category 1 and 2 SAR responses per year from 2014 to 2018.

Table 6. CAF SAR Responses.

CAF SAR Responses

2014

2015

2016

2017

2018

Average

Cat 1 and Cat 2 Aeronautical incidents

189

224

239

228

185

213

Cat 1 and Cat 2 Air SAR responses

51

74

66

62

47

60

Average Air SAR response time on RP30 (in min)

28

26

42

23

41

32

Average Air SAR response time on RP2hrs (in min)

69

69

78

73

71

72

Cat 1 and Cat 2 Air SAR responses to Maritime incident

215

215

248

216

216

222

Cat 1 and Cat 2 Air SAR responses to Humanitarian incident

129

128

183

176

178

159

Total Cat 1 and 2 Responses

395

417

497

454

441

441

Table 6. CAF SAR Responses. This table denotes the SAR responses and average response time by calendar year during the reporting period.

Table 6 Details - CAF SAR Responses.

Analysis of the SAR data during the time period covered by the evaluation found that the RP2hrs (120 minutes) averaged 72 minutes (refer to Table 6). Maintaining a threshold that is 48 minutes above the average response time and is continuously met barring mechanical issues is of questionable value as an indicator to measure performance. In addition, the requirement of two RPs with an actual delta of only 40 minutes requires validation.

Based on the SAR data analysed, the average time from a tasking to incident closure is 65.113 hours. Of 2,204 SAR responses during the reporting period, the average time to achieve RP30 and RP2hrs was 32 and 72 minutes. Therefore, the average time to achieve posture was only 0.82 percent and 1.8 percent of the entire SAR incident, respectively. Although only a small portion of a SAR operation, as the sole quantitative measurement within DND’s control, the positioning of these two RPs has been the subject of extensive analysis. CJOC has conducted trials where the RP30 timeframe was shifted to align with times of increased SAR requests based on historical distress callouts.

During interviews, all levels of CJOC staff articulated that shifting of RP times between RP30 and RP2hrs caused greater strain on an individual’s quality of life and ability to conduct training. As Table 7 details, the altering of the RP30 can affect as many as 690 CAF personnel in addition to contracted maintenance personnel. In 2013, the Office of the Auditor General (OAG) recommended that “National Defence should assign a sufficient number of SAR personnel to continue to meet operational needs and provide for the necessary training, professional development and leave of SAR personnel.” Footnote 16  The RCAF responded that a study was to be conducted and completed by January 2014, with the final model and analysis to be completed by spring 2014.Footnote 17  Although not yet finalized, this report should provide accurate SAR personnel data for SAR RP analysis. During this evaluation, ADM(RS) reached out to RCAF regarding the findings of this study and was informed that due to operational responsibilities the study was delayed but is underway.

Table 7. SAR Personnel required to maintain RP30.

Squadron

Type of Aircraft

Aircrew Personnel

Maintenance Personnel

Overall Personnel

103 Squadron Gander

CH149

34

0

34

413 Squadron Greenwood

CH149 and CC130

79

65

144

424 Squadron Trenton

CH146 and CC130

78

78

156

435 Squadron Winnipeg

CC130

78

119

197

442 Squadron Comox

CH149 and CC115

77

82

159

TOTALS

 

346

344

690

Table 7. SAR Personnel required to maintain RP30. This table denotes the SAR personnel required to maintain the 30 minute SAR RP.

Table 7 Details - SAR Personnel required to maintain RP30.

Interviewees have suggested multiple alternatives to the current posture. For example, using a single RP and providing the JRCCs the authority to select specific high-readiness postures as required, such as for the lobster fishery season. These suggested alternatives could be incorporated into the analysis of Canada’s SAR posture requirements.

ADM(RS) Recommendation

Key Finding 6: The SAR management system lacks a data analytics capability.

Being vitally important, SAR requires a data analytics capability to improve efficiency. In 2013, the OAG noted the requirement for more in-depth SAR performance metrics, and that the information management system used to manage SAR cases does not adequately support operational requirements and is nearing its breaking point. Except for a limited number of high-level indicators, such as the annual number of SARs conducted, SAR data analysis must be conducted by manually reviewing individual SAR events. The lack of an analytics capability limits performance measurement as the current environment requires significant manual effort by analysts to acquire and maintain performance data to perform the analysis. Interviewees stated that compiling the SAR data in response to specific analytic requests is also a time-consuming task.

Of note, a new SAR Mission Management System (SMMS) is in development and scheduled to be delivered in June 2021. Although the RCAF is identified as the project sponsor, CJOC is responsible for the SAR performance metrics. CJOC should ensure that SAR analytics data is clearly identified, and that the new SMMS contains a robust analytics capability that supports SAR performance measurement and results reporting.

ADM(RS) Recommendation

Key Finding 7: SAR crews are not uniformly provided with pre- and post-operation support.

The CAF manages each SAR as an individual standalone “mission”. When a SAR response is required, command of RCAF assets is assigned to the JRCC (CJOC) as the force employer to conduct the SAR. However, SAR personnel are not provided with pre- and post-operation support such as mental health services, contrary to the practice when operations are conducted.

SAR is a stressful 24/7 ‘no fail’ mission, in hazardous conditions, with an average 440 SAR responses per year (Table 6). Interviewees have stated that from SAR crews operating in hazardous conditions, to JRCC personnel talking to the families of lost individuals, to working in a basement room with no windows, SAR personnel are placed under a high level of mental stress. Interviewees indicated that the mental health of SAR personnel is a serious and ongoing concern.

The CAF has a robust proactive health system for operations. This includes providing proactive mental health support for personnel both before and after deployments. SAR positions are treated as a standard CAF posting with the standard reactive mental health support. The CAF should investigate how to provide mental health services to SAR personnel that are similar to those provided to other personnel employed in CAF operations.

ADM(RS) Recommendation

2.3 Economy and Efficiency

Key Finding 8: At this time, it is not possible to assess the economy and efficiency of CAF Operations programs. It appears that L1 organizations outside of CJOC are unclear in terms of the alignment of CAF Operations expenditures to program segments.

It is not possible to assess the economy and efficiency of CAF Operations. In fact, while the evaluation team received information from CJOC related to their expenditures on CAF Operations, it was not possible to obtain a complete understanding of the nature of the departmental spending allocated to CAF Operations as multiple L1 organizations code expenditures to this Program. This highlighted the lack of oversight of the assignment of CAF Operations expenditures to Program segments.

The Program leads do not have accountability over the entire Program and its associated expenditures which may involve multiple L1 organizations. The program-level oversight and guidance on Program expenditures would allow the department to better understand the resources allocated and expended in a Program with a view to adjust, if required.

It is recognized that L1s are accountable for expenditures under their direct control but there is a lack of overview and accountability for Program activities delivered by multiple L1s. The department would benefit from clarifying allocations, monitoring and reporting of Program expenditures. It is suggested that regular L0 and/or L1 level discussions be held to: (1) ensure efficient utilization and monitoring of resources at the program level; (2) review resource utilization by multiple L1 organizations contributing to a particular Program; as well as (3) provide guidance to support C Prog review efforts.

Figure 1. Programs 1.1 and 1.2 Expenditures by Year.

 Figure 1. Programs 1.1 and 1.2 Expenditures by Year. This figure denotes the L1 organization financial expenditures for Programs 1.1 and 1.2 as reported to ADM(Fin).

Figure 1 Details - Programs 1.1 and 1.2 Expenditures by Year.
Double Horizontal Bar Graph of the expenditures for programs 1.3 by year.

Figure 2. Program 1.3 Expenditures by Year. This figure denotes the L1 organization financial expenditures for Program 1.3 as reported to ADM(Fin).

Figure 2 Details - Program 1.3 Expenditures by Year.

ADM(RS) Recommendation

Observation 1: Increased use of GPS-capable beacons will improve SAR efficiency.

There are two types of Emergency Locator Transmitter (ELT) beacons: 121.5 Megahertz (MHz) and 406 MHz. Of the two beacons, the 406 MHz is far superior. The 121.5 MHz ELT was designed to alert other aircrafts flying overhead of a crash. A significant limitation is that another aircraft must be within range and listening to the 121.5 MHz to receive the signal. However, a 406 MHz ELT properly registered with the Canadian Beacon Registry provides vital contact information, such as the name of the owner of the beacon, the aircraft type, tail number and an emergency contact number. For false alarms, when the origin of a 406 MHz beacon is known, the JRCCs have a 97 percent closure rate with the aid of a simple phone call if it was accidentally triggered.Footnote 18  It reduces the number of times SAR crews conduct unnecessary searches, preserving primary SAR aircraft for actual emergencies, and reduces operational costs while decreasing the frequency in which the CAF responds to inadvertent ELT activations.Footnote 19  A properly registered 406 MHz beacon also allows SAR crews to quickly fly directly to an incident site where lives are potentially at risk.

In February 2009, the ability to track the signal processing of the older emergency 121.5 MHz beacons was discontinued. When a beacon activates, JRCCs have an enormous challenge finding a 121.5 MHz beacon or an unregistered 406 MHz beacon. Without the aid of the COSPAS-SARSAT system, finding the source of these beacons routinely requires the tasking of an RCAF or Civil Air SAR Association or other air assets.

Current Transport Canada regulations specify that only certain aircraft types are required to have an ELT. They do not specify that it must be a 406 MHz beacon. Transport Canada has not yet mandated the newer 406 MHz beacons as a required standard for private owners, thus as of end of December 2016, only 29 percent of registered Canadian aircrafts had a 406 MHz ELT. Between January 1, 2015 and December 31, 2018, there were 93 aircraft crash (Category 1) missions that involved an ELT search. Of these 93 missions, only 43 missions (46%) had a 406 MHz beacon.Footnote 20 

Although outside the control of DND, this is an operational impediment as Air SAR is time consuming and costly. Moreover, the limited utilization of modern ELT beacons has a direct impact on the efficiency of operations. Compared to closing an accidental trigger of a 406 MHz beacon with a phone call, when 121.5 MHz beacons are accidently triggered, SAR crews conduct unnecessary searches, increasing operational costs and reducing availability of SAR assets to respond to real emergencies. Of note, CJOC has stated that a change to the Canadian Aeronautical Regulations is in progress.

Back to Table of Contents

Annex A—Management Action Plan

Key Finding 2: Although the CAF achieves a stated 97.8 percent of its operational objectives, there is a lack of precision in the objectives and a lack of clarity in their measurement.

ADM(RS) Recommendation

Management Action

Action 1.1

Implementation of a Results Based Monitoring and Evaluation Framework to be used across all operations. This permits the analysis and production of individual operational goals, outcomes and outputs, the development of Measures of Performance and Measures of Effectiveness, and the resulting indicators. These form the basis of Annex OO to Operation Orders – the Assessment Plan. The data collection plan as part of the Assessment Plan will dictate the nature of the return required to address each indicator (Quantitative, Qualitative, Objective and/or Subjective). A proposed Results Based Monitoring and Evaluation Framework has been exercised and partially informed a Regional Assessment Board in March 2020.

Target Date: April 2021 Implementation

Action 1.2

CJOC Operational Planning Process and Op Design Standard Operating Procedures will be examined to determine the best means to translate Strategic Objectives to Operational Objectives and Decision Points, which permit realistic measurement. The issue has been identified at the Operational Assessment level. Unrefined Strategic Objectives and limited Op Design hamper the ability to ultimately formulate indicators and perform the measure and adjust functions. The Joint Operation Planning Group and resulting products will incorporate Operational level objectives and a full Op Design.

Target Date: April 2021 Implementation

Action 1.3

Examine defining characteristics, direction and objectives/outcomes between Operations, Missions and Activities. Determine current classifications and what must be measured. Supporting documentation (e.g., Initiating Directive, Operation Orders, Fragmentation Order) must be available, relevant and current to support the implementation of a Results Based Monitoring and Evaluation Framework, and the resulting measurement and analysis of Operational Objectives. The desired military effects will be weighed against objectives, and whether these are measurable. Each named operation/mission will be examined. Supporting documentation will be updated.

OPI: CJOC
Target Date: August 2021 for April 2022 implementation

Key Finding 4: The provision of Provincial Air Ambulance services to Newfoundland is not considered part of the CAF’s SAR Mandate.

ADM(RS) Recommendation

Management Action

Action 2.1

CJOC SAR will track the number of Air Ambulance missions in the province of NFLD and Labrador, and the costs associated with these missions. This will be reported to the Federal SAR Operations Governance Committee (FSAROGC). The first report will be submitted to the FSAROGC by December 2020 with subsequent reports to follow at each semi-annual meeting.

The FSAROGC will assess the situation and recommend courses of action to Comd CJOC regarding cost recovery. The correct mechanism for recovery to be determined, should recovery be directed.

OPI: CJOC
OCI: RCAF, ADM(Fin)
Target Date: December 2020 to the FSAROGC

Key Finding 5: The two individual SAR response postures have not been reviewed since 1958. As a result, they have negative effects on force readiness, SAR crew quality of life and the ability to conduct training.

ADM(RS) Recommendation

Management Action

Action 3.1

CJOC will initiate joint consideration of the postures between the RCAF and CJOC as the Force Generator and Force Employer. We will examine reasons for current postures and if changes to posture can improve service provision balanced with personnel and materiel limitations.

If a single posture is recommended, CJOC to prepare decision briefing to CDS for decision.

Decision prior to December 2021.

OPI: CJOC/DG Ops, SAR
OCI: RCAF
Target Date: December 2021

Key Finding 6: SAR lacks a data analytics capability.

ADM(RS) Recommendation

Management Action

CJOC agrees that having a robust, efficient data analytics capability for SAR operations is an absolute must. The current system is capable of capturing multiple data points but the manual processing of multiple demands is antiquated.

Action 4.1

CJOC, in cooperation with the RCAF and ADM(IM), will work towards determining if more or less data is required than is currently being captured. All SAR publications pertaining to statistics will be amended to any changes.

Action 4.2

Once data requirements are known, CJOC, in cooperation with the RCAF and ADM(IM), will engage the SMMS project team to determine the feasibility of capturing SAR mission data requirements and the ability to produce multiple outputs (reports and returns) from a variance of selectable criteria.

Action 4.3

CJOC will determine if additional staffing requirements with specific technical expertise will be needed in order to achieve this Management Action Plan.

CJOC will be satisfied that this Management Action Plan will be complete once all statistical requirements (products) can be achieved by the SMMS software with significantly less work hours than the current process, and that those products meet all needed reports and returns for ongoing internal evaluations of SAR operations.

OPI: CJOC/DG Ops, SAR
OCI: RCAF, ADM(IM), CJOC/DG Rdns, S&T
Target Date: If possible, to align with SMMS project full operational capability date

Key Finding 7: SAR crews are not uniformly provided with pre- and post-operation support.

ADM(RS) Recommendation

Management Action

Action 5.1

CJOC will confirm SAR Squadrons have implemented Road to Mental Readiness (R2MR) programs as part of post-mission debriefing when required or requested.

Action 5.2

CJOC will include reference to SAR Squadron R2MR programs in the CJOC SAR Directive, making the program part of mission debriefing, if required or requested.

Action 5.3

CJOC will consult with RCAF on status of the R2MR program annually.

Action 5.4

Inclusion of mental health provisions in 2021 CJOC SAR Directive.

OPI: CJOC/DG Ops, SAR, HSS
OCI: RCAF
Target Date: March 2021

Key Finding 8. At this time, it is not possible to assess the economy and efficiency of CAF Operations programs. It appears that L1 organizations outside of CJOC are unclear in terms of the alignment of CAF Operations expenditures to program segments.

ADM(RS) Recommendation

Management Action

Action 6.1

CJOC conducted a review, validation and modifications (where necessary) of all Program Segment alignments during FY 2020/2021 Annual Master Data Review for CJOC’s owned Operations financial structure. This undertaking confirmed that CJOC’s existing and new operations financial structure is properly aligned with the Operations core responsibility program segments.

Action 6.2

In order to ensure that the L1s operational financial structure is properly aligned to the Operations core responsibility program segments, CJOC J8 Plans and Policy section will work with other L1 stakeholders and C Prog to ensure that existing and future alignments to CJOC Program segments are reviewed and adjusted for the Annual Master Data review for FY 2021/22. This review and adjustment process will ensure that CJOC exercises oversight for the attribution of operational expenditures related to CAF Operations.

OPI: CJOC
OCI: VCDS C Prog, ADM(Fin), Applicable L1s/Comds
Target Date: March 2021

Back to Table of Contents

Annex B—Evaluation Methodology and Limitations

1.0 Methodology

1.1 Overview of Data Collection Methods

The evaluation team used multiple lines of evidence and complementary qualitative and quantitative research methods to help ensure the reliability of information and data to support evaluation findings. In order to ensure the validity of data captured through different methodologies, a data triangulation approach was used. The methodology established a consistent approach in the collection and analysis of data to support findings, conclusions and recommendations. Based on the evidence from available sources, the evaluation reviewed the achievement of expected program outcomes and the efficiency of the program activities within the Departmental Results Framework (DRF), to develop a proper baseline for the relevance and performance of the Programs. Information and data were correlated to each evaluation question and corresponding indicators.

1.2 Details on Data Collection Methods

Comparison of both qualitative and quantitative assessments was used to validate the overall analysis and to develop the evaluation’s findings and recommendations.

The following data collection methods were used to gather qualitative and quantitative data for the Evaluation:

1.2.1 Literature and Document Review

A preliminary document review was conducted as part of the planning phase of the evaluation to gain a foundational understanding of the CAF operations activities in the context of the Programs that constitute Defence’s Program Inventory. A comprehensive document review was undertaken as part of the conduct phase of the evaluation, focusing on the relevance and performance of the Programs.

1.2.2 Key Informant Interviews

Interviews were conducted in person and over the phone. Interviewees were provided with an interview guide in advance. During interviews, clarifying questions were asked. Notes were taken by the evaluators during interviews. These were later transcribed and compared for clarity and accuracy using an evidence matrix tool, designed to capture reoccurring themes in a common record.

Interviews were conducted in parallel with the Evaluation of the Command, Control and Sustainment of Operations. Organizations interviewed included the following: CJOC Headquarters; JRCCs in Trenton and Halifax; and 424 and 413 SAR squadrons.

1.2.3 Program and Financial Data Reviews

Program data relative to CAF operations were sourced from multiple sources. Challenges in linking the Program expenditures to activities, noted in finding 8, required the evaluation to compile financial data from multiple sources.

2.0 Limitations

Table B-1. Evaluation Limitations and Mitigation Strategies.

Limitation

Mitigation Strategy

The lack of visibility into the diverse organizations planning, assigning and managing resources to the Programs dedicated to delivering CAF Operations prohibited the evaluation of program economy and efficiency.

The evaluation requested information on the activities conducted by each L1 organization that assigned resources to the DRF programs.

The possibility that the interviewees would provide biased information and only denote positive components of their organization.

A comparison was made between interviewees and other people from the same organization or group, and from information provided by other sources (e.g., documents and files).

Limited performance measurement information.

The evaluation consulted multiple sources of quantitative and qualitative information to determine the performance of the program. However, although performance measurement data was limited for this program, data obtained was supplemented by primary data collection carried out by Director General Evaluation (DGE).

Table B-1. Evaluation Limitations and Mitigation Strategies. This table lists the limitations of the evaluation and the corresponding mitigation strategies.

Table B-1 Details - Evaluation Limitations and Mitigation Strategies.

Back to Table of Contents

Annex C—Logic Model

Flow Chart of the inputs, activities, outputs and outcomes of the program.

Figure C-1. Logic Model for CAF Operations. This flowchart shows the relationship between the program’s main activities, outputs and expected outcomes.

Figure C-1 Details - Logic Model for CAF Operations.

Back to Table of Contents

Page details

Date modified: