Summative Evaluation Of The Canadian Forces Health Services Response To The Covid-19 Emergency

Acknowledgements

The Directorate of Health Services Quality and Performance would like to thank the Canadian Forces Health Services (CFHS) Evaluation Steering Committee for their timely input and support throughout the process. Many partners in CFHS headquarters and in the field contributed, and we would especially thank those who hosted the field visit and facilitated data gathering. We also extend our gratitude to all the Defence Team members who took the time to provide their input and share their experiences.

Dr. Heather Galbraith, Family Physician
Royal Canadian Navy, 1 May 2020

Table of Contents

• Acronyms and Abbreviations
• 1.0 Executive Summary
  • 1.1 Background
  • 1.2 Purpose
  • 1.3 Methods
  • 1.4 Findings
  • 1.5 Conclusion
• 2.0 Context
  • 2.1 Historical Perspective
  • 2.2 Contingency Plan LASER
  • 2.3 Mounting Pressure
  • 2.4 Command and Control
  • 2.5 Ongoing Modernization of Governance in the CFHS
  • 2.6 Requests for Assistance
• 3.0 Key Findings
  • 3.1 Cluster I Institutional health services support of CAF operational capabilities
    • 3.1.1 Effectiveness
    • 3.1.2 Efficiency
    • 3.1.3 Governance
  • 3.2 Cluster II Logistics of the response
    • 3.2.1 Effectiveness: Epidemiological surveillance
    • 3.2.2 Effectiveness: Lessons Learned
    • 3.2.3 Effectiveness: Communications
    • 3.2.4 Efficiency: Epidemiological Surveillance
    • 3.2.5 Efficiency: Lessons Leaned
    • 3.2.6 Efficiency: Communications
    • 3.2.7 Governance: Epidemiological Surveillance
    • 3.2.8 Governance: Lessons Learned
    • 3.2.9 Governance: Communications
  • 3.3 Cluster III Institutional health support of civilian health system through RFAs
    • 3.3.1 Effectiveness
    • 3.3.2 Efficiency
    • 3.3.3 Governance
  • 3.4 Cross-cutting issues: Gender, diversity & inclusion
• 4.0 Conclusions
• 5.0 Summary of Recommendations
• Annexes
  • Annex A Terms of Reference
  • Annex B Logic Model
  • Annex C Evaluation Approach and Methodology
  • Annex D Conplan Laser
  • Annex E List of Definitions
  • Annex F List of Documents Reviewed

Acronyms and Abbreviations

ADM (PA)

Assistant Deputy Minister Public Affair

AFU

Air Filtration Unit

ARA

Authority Responsibility Accountability

ADM (RS)

Assistant Deputy Minister Review Services

BCP

Business Continuity Plan

BRP

Business Resumption Plan

B/WSurg

Base/Wing Surgeon

CAF

Canadian Armed Forces

CDS

Chief of the Defence Staff

CE

Construction Engineering

CF H Svcs Gp

Canadian Forces Health Service Group

CFHS

Canadian Forces Health Services

CFB

Canadian Force Base

CFHIS

Canadian Forces Health Information System

CDS

Chief of the Defence Staff

CDO

Chief Dental Officer

CDU

Care Delivery Unit

CFB

Canadian Forces Base

CFTPO

Canadian Forces Task Planning and Operation

CFJP

Canadian Forces Joint Publication

CMED

Central Medical Equipment Depot

CMP

Chief of Military Personnel

CO

Commanding Officer

CONPLAN

Contingency Plan

CoC

Chain of Command

C2

Command and Control

COM

Commander Canada Command

CJAT

Commander's Joint Assessment Team

CJOC

Canadian Joint Operations Command

CGP

Canadian General Population

CSM

Clinic Service Manager

CT

Contact Tracing

DAOD

Defence Administrative Orders and Directives

DentIS

Dental Information System

DDC

Dental Detachment Commander

DFHP

Directorate of Force Health Protection

DGHS

Director General Health Services

DND

Department of National Defence

DLLS

Defence Lessons Learned System

DLLP

Defence Lessons Learned Program

D HS Q&P

Directorate of Health Services Quality and Performance

DHSO

Directorate Health Services Operations

DSG

Deputy Surgeon General

GBA+

Gender-Based Analysis Plus

GoC

Government of Canada

GDMO

General Duty Medical Officer

GMH

General Mental Health

HHR

Health Human Resources

HSS

Health Service Support

HS

Health Services

HQ

Headquarters

IPAC

Infection Prevention and Control

IT

Information Technology

JTF

Joint task forces

LTCF

Long Term Care Facility

MHCP

Mental Health Care Provider

MMAT

Multipurpose Medical Assistance Teams

MPC

Military Personnel Command

NATO

North Atlantic Treaty Organization

Op

Operation

PCSM

Primary Care Services Manager

PHAC

Public Health Agency of Canada

PHM

Public Health Measures

PI

Pandemic Influenza

PLRD

Programme des leçons retenues de la Défense

PML

Preferred Manning Levels

PSS

Psycho-Social Services

PM

Program Manager

PSU

Primary Sampling Unit

QC

Quality Council

RFA

Request for Federal Assistance

RSurg

Regional Surgeon

SG

Surgeon General

SME

Subject Matter Expert

SOODO

Standing Operations Order for Domestic Operations

TBS

Treasury Board Secretariat

TOR

Terms of Reference

WO

Warrant Officer

WW

World War

Back to top

1.0 Executive Summary

1.1 Background

On March 2, 2020 the Chief of the Defence Staff of the Canadian Armed Forces activated the pre-existing, multi-phased Contingency Plan (CONPLAN) LASER^{Footnote 1} in response to a global pandemic of COVID-19 (coronavirus disease 2019), caused by the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) virus^{Footnote 2}.Operation LASER (Op LASER) provided the basis for a coordinated Canadian Armed Forces (CAF) effort to maintain operational capability and support civil authorities. Due to the health-centric nature of the emergency, Op LASER was predominantly designed to protect the health of the force and, therefore, significantly implicated the Canadian Forces Health Services (CFHS), the organization ordinarily responsible for the provision of health care to CAF members both in-garrison in Canada and abroad on operations^{Footnote 3} as well as force health protection.

As a part of the CFHS response, the Directorate of Health Services Quality and Performance (D HS Q & P) was tasked^{Footnote 4},^{Footnote 5} to implement a robust national strategy to capture institutional lessons learned (LL) for CFHS throughout all phases of Op LASER. Given the unprecedented scope and complexity of the efforts required of CFHS in response to COVID-19, it was determined that the LL activity could only achieve the necessary level of quality, objectivity and comprehensiveness if executed under the auspices of the CFHS Evaluation Program as a structured Summative Evaluation of the Canadian Forces Health Services Response to the COVID-19 Emergency.

1.2 Purpose

The Evaluation covers the period from March 2020 to September 2021 (see Term of Reference in Annex A) and focuses on key evaluation questions derived from the perspectives of CFHS executive members via a broad organisational scoping exercise. The exercise shaped a shared vision of a summative mapping design of the array of complex, phased, and clustered emergency activities that depict how the CFHS contributed to achieving CONPLAN/Op LASER strategic objectives (see Logic Model in Annex B).

The mapping exercise conceptualized CFHS's activities as falling into three main Clusters: (I) institutional health services support of CAF operational capabilities; (II) logistics of response and; (III) institutional health support of Canadian civilian health care system through Requests for Federal Assistance (RFAs). Performance in each of these Clusters was examined through the lenses of effectiveness, efficiency, governance (particularly clinical governance), and Gender-Based Analysis Plus (GBA+).

This report presents the findings and recommendations produced to address the key evaluation questions within each of the three Clusters of CFHS activity. It is intended to produce insights into what worked well and what can be improved in CFHS's response to COVID-19 with the anticipation that CFHS senior decision-makers can use these insights to improve the organization's ability to plan for and respond effectively and efficiently to future health emergencies.

1.3 Methods

This evaluation used a mixed method approach and techniques of data collection, driven by the specific needs for information linked to the key evaluation questions. Data was collected from a variety of primary (e.g. interviews, observation and focus groups) and secondary data sources (document and literature review as well as Canadian Forces Health Information System (CFHIS), Dental Information System (DentIS) and DND reports and publications databases). Detailed methodology can be found in Annex C. Collected data from multiple sources and lines of evidence were analysed, interpreted and triangulated by the evaluation team:

1.4 Findings

From the value criteria point of view of health interventions, the CFHS strategic objectives assigned within Op LASER such as maintaining the operational effectiveness of CAF, provision of health care to CAF members and support of the Government of Canada were achieved. Moreover, the CFHS retained the core operational capability to support all CAF missions – including pre-existing operations as well as those associated with Op LASER. While the extent to which the objectives were achieved does vary among the Clusters, no evidence of systematic failure was identified (see Figure 1). The CFHS was able to deliver on its mandate during all phases of the COVID-19 pandemic.

However, as indicated by the value criteria, the CFHS performed less well in terms of efficiency of the response, which imposed additional potentially avoidable burdens on a health system already stressed by pre-existing shortages of health care personnel (the CAF has less than 50% of required physicians available to deploy along with 75% of nurses and 65% of medical technicians). In short, the scores achieved do not reflect the heavy toll exacted from the personnel within the CFHS who made the extensive and sustained response possible despite the myriad of challenges and inefficiencies.

1.5 Conclusion

Based on the evidence examined during the Evaluation, CFHS uninterruptedly provided essential mission-critical health services, medical and dental direction and expert advice in support of public health protection and supplied health services forces to Joint Task Forces (JTFs) engaged in current operations in Canada and abroad. The personal health and safety of CAF personnel were effectively protected, allowing the CAF to maintain operational effectiveness, and readiness for missions during the COVID-19 pandemic. In-garrison health services and units rapidly implemented public health measures for infection control and ensured the safe continuity of care provision. Increased demand on the military health care system to ramp up a number of internal functions, such as epidemiological surveillance, as well as force generate an unprecedented number of personnel for RFAs was managed.

However, the Evaluation identified several factors that impacted the efficiency of the CFHS response to COVID-19, many of these adding to the organizational effort required for CFHS to achieve the successes it did. These included important gaps in planning and preparedness, most particularly the lack of a detailed and up to date medical CONPLAN and pre-existing human and other resource challenges. Similarly, there were findings related to pre-existing governance structures and processes that impeded CFHS's ability to execute required tasks in the context of the pandemic.

Notwithstanding these handicaps at the outset, the Evaluation also found CFHS personnel at all levels were able to develop and rapidly institute a number of innovations and adaptations to meet the multiple challenges and demands they faced. While largely effective, these measures were not without significant cost to the CFHS – particularly at the tactical level. As a result, CFHS should expect substantial challenges recovering from this extended surge posture.

Back to top

2.0 Context

2.1 Historical Perspective

"The fear of influenza was constantly in minds [of the military healthcare providers during WWII] as a result of an experience in the First World War." When the Canadian Army was preparing for deployment, it was estimated that roughly 266,000 soldiers lost over a million training days per annum due to respiratory illness^{Footnote 6}. Later, when the 1957 influenza pandemic (caused by an H2N2 variant) began, CFHS leadership and their civilian counterparts "pondered the best approach to mitigate the disease's potential ravages^{Footnote 7}. As with the 2009 H1N1 pandemic, when Op LASER was first set in motion, the CDS activated CONPLAN LASER immediately after the WHO declared the COVID-19 outbreak a pandemic, giving rise to the second activation of Op LASER.

2.2 Contingency Plan LASER

The CONPLAN LASER, which exists as an Appendix to Annex A of the CAF's Standing Operations Orders for Domestic Operations (SOODO), provides the "basis for a coordinated CF effort to maintain operational capability and support to civil authorities as directed by the CDS^{Footnote 8} ". It was a phased plan specifically developed to address a pandemic caused by influenza and served as the starting point for all activities and orders issued under Op LASER.

In accordance with the CONPLAN, the CAF's steady state is Phase I, which requires the CAF to maintain a perpetual state of pandemic readiness. A detailed summary of the aspects of the CONPLAN's design relevant to the matters examined in this Evaluation, along with a comparative analysis of the features of the COVID-19 pandemic and the 2009 H1N1 influenza pandemic – upon which the current CONPLAN was based - is contained at Annex D.

2.3 Mounting Pressure

With the promulgation of the COVID-19-related Op LASER, CFHS experienced unprecedented pressures from several fronts. Firstly, an already depleted workforce was being directly (i.e. getting sick) or indirectly (e.g. death of a family member or loved one) impacted negatively by the virus, further reducing the available workforce. Secondly, while the demand for in-garrison health services decreased in some respects as CAF training wound down (e.g., physiotherapy consult,), it remained stable or increased in others (e.g., primary care, mental health). Thirdly, compounding an already distressed workforce, CFHS was required to rapidly force-generate drawing from institutional healthcare providers as a secondary source to augment force-generating unit efforts. Lastly, CFHS was required to maintain the current levels of integral medical and dental support to operations with the added responsibilities of controlling the spread and protecting the safety of mission troops and associated allies.

2.4 Command and Control

CFHS exists as a Level 2 organization within the CAF, with all other major force elements existing as Level 1 organizations. With few exceptions (such as the Disaster Assistance Response Team), health services elements are deployed to support these other elements. Planning for health services support under normal circumstances can be inefficient due to the need to work through a higher-level L1 organization that does not have the capacity or expertise to contribute to such specialized aspects of operations. However, this is generally mitigated through CFHS's ability to rely on doctrine, experience, and having time to absorb the extra steps of working through the L1. CONPLAN LASER, and the ensuing Op LASER, departed from normal operations in having health and its preservation as the overarching mandate, yet simultaneously reflected traditional domestic operational doctrine, which injected additional layers of non-health services command and control in a time-compressed setting. As will be explained in the Evaluation's findings, this configuration had a negative impact on CFHS's response to COVID-19.

2.5 Ongoing Modernization of Governance in the CFHS

The 2018 ADM(RS) Evaluation of Military Healthcare^{Footnote 9} recognized the need for CFHS to strengthen its clinical governance, including clarifying the Surgeon General's and Chief Dental Officer's (CDO) accountabilities, responsibilities, and authorities (ARAs) for respective medical and dental professional technical matters within the CAF. This includes acting as the CAF's Chief Medical Officer of Health, providing definitive public health advice. CFHS efforts to action ADM (RS) recommendations to firmly establish SG and CDO ARAs as the core of an updated and integrated governance framework were still very much in process when the COVID-19 pandemic was declared. As a result, CFHS was subjected to a sudden and unprecedented surge in demands for public health advice, which created heavy burdens on the Force Health Protection functions and professional-technical networks that inform and distribute that advice. For CFHS, meeting this demand was further complicated by continually evolving knowledge about the SARS-CoV-2 virus and the need to align public health advice with multiple host jurisdictions within Canada and abroad.

2.6 Requests for Assistance

Throughout the SARS-CoV-2 pandemic, the CAF maintained critical operations while responding to the Government of Canada (GoC) Requests for Assistance (RFAs). These RFAs were launched under OP LASER, reflecting CONPLAN LASER's anticipation of the need for the CAF to provide forces to assist civilian authorities with the provision of essential public services impacted by a pandemic. From 19 February 2020 to 28 March 2021, the CAF supported 111 RFAs that varied in nature and complexity and included the repatriation of Canadians from abroad, support to identified Long Term Care Facilities (LTCFs) in Ontario and Quebec, as well as providing personnel to support the contact tracing efforts of the Public Health Agency of Canada (PHAC).

Throughout the RFA process, CFHS was the predominant CAF force generator for specialized positions responding to a high-profile, politically sensitive, startling situation in LTCFs impacted by the global pandemic whose effect and severity was changing rapidly.These missions comprised an unusual population and context for healthcare providers trained to support more typical military operations and were also complicated by important health regulatory jurisdictional considerations.

In sum, and in addition to the need to sustain essential healthcare in-garrison, Op LASER saw the largest number of health services personnel deployed concurrently in recent memory, requiring the CFHS to reach into clinics and lean heavily on Reserve Force members to fill the tasking.

Back to top

3.0 Key Findings

3.1 Cluster I Institutional health services support of CAF operational capabilities

3.1.1 Effectiveness: Is the intervention achieving its objectives?

Finding: Public Health Measures (PHMs) instituted within the CFHS were generally effective in controlling the risk of transmission of COVID-19.

A virtual audit of the implementation of recommended infection prevention and control measures conducted in June 2020 across CFHS clinics indicated a high degree of compliance with most PHMs in the majority of locations. This was confirmed in more detail through on-site visits of the sampled clinics during the Evaluation.

The CAF achieved infection rates lower than background Canadian population rates^{Footnote 10}. In terms of outbreaks originating in CFHS clinics, these were limited and involved staff-to-staff transmission, indicating Infection Prevention and Control (IPAC) measures and the Chief Dental Officer's Interim Clinical Directive: Dental treatment during the COVID-19 Pandemic at the patient interface were effective despite the inherent increased risk of exposure in the patient care setting. As the pandemic continued and the nature of the virus became better understood, CFHS implemented service delivery design adaptations and engineering controls that permitted increases in in-person healthcare delivery without increased risk of COVID-19 transmission.

The response to COVID-19 hastened the CFHS's development of IPAC-relevant capacities and innovations, such as: enhanced ability to perform contact tracing; improved engineering controls (such as Air Filtration Units for dental procedures); evidence-based decision-making aids; and a rapidly deployable communicable disease testing capability. As the pandemic continued, these service delivery adaptations and innovations permitted increases in healthcare delivery in-garrison and effective Health Services Support (HSS) in deployed settings without a significant increased risk of COVID-19 transmission.

Finding: Overall, CFHS was able to sustain essential in-garrison and deployed health services during the initial phases of the response to COVID-19. There is, however, evidence that the CFHS will struggle to meet both deferred and, in the case of mental health, growing healthcare demand while also providing support to increasing in-garrison operations over an extended CAF recovery and reconstitution period.

The institution of PHMs and the requirement for enhanced IPAC measures in the clinics significantly decreased their ability to provide in-person health services, particularly in the initial weeks of the response. Data collected from the CFHIS and DentIS indicate significant dips across all categories of health services delivered (see Fig 2). This was most marked for dental and physiotherapy services due to the higher infectious disease transmission risk and/or the lack of suitable alternatives to in-person service delivery to meet care needs.

Figure 2 CFHS health care services utilisation

Back to top

COVID-19 similarly disrupted ongoing deployed operations and a large proportion of the standard HSS plans and support processes had to be adapted to accommodate pandemic conditions, creating significant challenges early on. CFHS was able to adapt and continue to provide effective HSS to deployed operations (including communicable disease control support), largely due to the clear priority of filling additional taskings and providing other forms of support to these missions over other competing activities.

Clinics prioritized the care provided to ensure that urgent requirements were met while adaptations to service delivery design that would permit increased care delivery were being planned and implemented. Patients and Health Care Providers (HCPs) who were interviewed generally understood the requirement to scale down services and felt essential needs were met. Data for externally accessed primary care services (see Fig. 2) does show a decline in utilization during the first quarter after the pandemic was declared, suggesting that overall CAF member demand for healthcare – whether delivered directly by CFHS or externally in a civilian setting - was restrained due to limited accessibility of health care. This would be consistent with the trend in Canada in general due to curtailed levels of individual activity, lower incidences of other infectious illnesses during lock-downs, and fear of contracting COVID-19 from exposure to other patients in a healthcare setting.

The in-garrison demand for essential patient care being significantly higher in medical care than in dental services afforded medical clinics less time and capacity than their dental counterparts to respond to pandemic-related requirements and changes.

It was suggested by some patients and HCPs that PHMs were overly conservative in some situations, particularly where there was low community prevalence of COVID-19. Decisions to curtail in-person services in favor of no or virtually-delivered care in these situations may have conferred greater overall health risk than was offset by the decrease in risk of patients or staff contracting COVID-19 through in-person healthcare activities.

During the latter part of the evaluation period, data from CFHIS shows trends across most health services for higher levels of demand than existed pre-pandemic, with trajectories that were continuing to rise. While some of this was likely due to pent up demand for services that had been deferred earlier in the pandemic, there is evidence that the pandemic has resulted in increased incidences of mental health issues in the general population. Early indications are that this phenomenon might also apply to CAF members, which will further strain CFHS primary care mental health care capacity.

3.1.2 Efficiency: How well are resources being used?

Finding: CONPLAN LASER was largely ineffective in supporting an efficient CFHS in-garrison response to the COVID-19 pandemic.

CONPLAN LASER (October 2012) was outdated, untried, and contained insufficient detail and scope to inform the efficient mobilization of a system-level health services response to a pandemic. The complementary medical response plan referred to in Appendix 4, Annex A of the CONPLAN as a Phase I requirement^{Footnote 11} does not appear to have been developed, or, if developed, was not finalized, exercised and available to medical planners. Two of the three strategic objectives of the CONPLAN (to maintain essential health services for CAF personnel and to provide assistance to civil authorities), from the CFHS perspective, operate in direct competition with one another given that support to RFAs in the event of a health crisis should reasonably have been expected to divert finite health human resources from the ongoing tasks of force protection, delivery of essential care to CAF members, and preservation of HSS to ongoing operations. While this level of surge across multiple lines of CFHS operation may have been a feasible strategy during the brief influenza driven Phase III envisioned in the CONPLAN, it was not a realistic expectation for the sustained COVID-19 pandemic.

While the exigencies of an in-garrison care response to a pandemic were not contemplated in any detail within the CONPLAN, nor did they appear to be covered to any useful extent within most Business Continuity Plans (BCPs) that generally apply to base/wing level operations. This created an important planning gap and confusion when clinic "operations" became an explicit component of Op LASER. Clinic BCPs, where they existed, varied in quality and utility of health services-specific content. Further, they relied on the availability of resources, such as IT, that proved insufficient for the magnitude of the emergency.

Most evaluation informants were unaware of how to access the CONPLAN LASER and, therefore, unfamiliar with the specifics of its contents, even at the headquarters level. This, combined with the gaps between the CONPLAN and existing BCPs, led to a significant level of potentially avoidable effort expended in the planning and executing of the CFHS response while the emergency was unfolding.

Finding: CFHS rapidly instituted adaptations to services to permit continued provision of high priority care in a pandemic environment, but not without some cost to productivity. The safety and quality of some of the adaptations and the risk of deferred services have yet to be assessed.

A number of adaptations were identified through interviews and site visits conducted as part of the evaluation process and are displayed in Figure 3 below.

Figure 3 Mechanisms of sustainability of health care provision

Back to top

Overall, it was found that efforts to reduce the demand for care, redirect patients to other sources of care, outsource operations where possible, optimize capacity given available staff, and increase staff effort combined to make health care provision more sustainable both in-garrison and in deployed settings.

Guidance was provided from higher headquarters on prioritization of care, although this was clearer and timelier for dental than medical care. The ability to triage patients into categories of priority with confidence and consistency (e.g. dental or physiotherapy services) and the nature of the service lending itself to safe deferral of significant numbers of non-urgent patients were also factors.

Virtual Care (VC) – Significant and enduring advances in the use of VC to augment provision of healthcare in the CFHS resulted from necessity due to pandemic restrictions. Data collected through the evaluation period demonstrated a strong acceleration of VC use even when PHMs were eased and in-person care became safer. This rapid institution of VC - notwithstanding its use outside of CFHS for some time – resulted in varying staff comfort levels. Considerable variation was found in how well VC was leveraged at the tactical level to support continuity of services. In one location, where the mental health staff moved quickly to adopt VC, wait times for services steadily decreased from pre-pandemic levels without increasing reliance on referred-out services. In contrast the converse was true in locations where staff were more reluctant or not as well equipped and supported in their adoption of VC.

Some issues were identified with the rapid shift to VC in the CFHS context, where the inability to integrate VC software into the extant CFHIS and the logistics of its use in clinics without WiFi capability posed significant challenges. These included a shifting of administrative burden for the management of patient encounters from clerical staff to HCPs, and the need to extend appointment lengths to deal with this as well as unreliable technology. Additionally, team-based care was not easily coordinated in the absence of mature IT enabled VC business processes. Some concerns were also raised about the clinical appropriateness of VC use to such a significant degree, highlighted by reports of patient safety incidents in which VC use was identified a causal factor; however, investigating this is outside the scope of this evaluation.

Human Resources – At several sites, particularly early in the pandemic, most onsite healthcare professionals were Calian^©contractors as their contracts precluded their working outside of the clinic. Having a high proportion of contracted HCPs on staff pre-pandemic was, therefore, seen as facilitating efficiency in preserving health services. Some adaptations to HCP scopes of practice to optimize use of more available health care providers were developed and authorized by the Surgeon General centrally. In other cases, adaptations to clinical roles and scopes of practice were local. Most often, local adaptation saw the re-assignment of personnel whose primary functions diminished due to overall reductions in some health-related services (e.g., dental hygiene procedures that were deemed non-essential and highly restricted, dental personnel offering support to vaccination clinics as assistant immunizer or for immunization data entry). There was also a shift seen from in-person triage and screening by clinicians to telephone screening of patients performed by clerical staff to manage healthcare access. Any training and protocols to support the safe execution of this new duty were generated locally and Quality Assurance to monitor compliance with standards not evident. Some clerks were uncomfortable with this new responsibility.

Other Factors - Strong support from the Directorate of Force Health Protection (DFHP) personnel helped build communicable disease control and IPAC capability in local and regional CFHS staff and support clinics and the CAF in responding effectively and efficiently to the pandemic. This support took the form of epidemiological data, scientific knowledge products (such as the Fusion Centre Reports), IPAC guidance, risk stratified decision-making frameworks, and regular teleconferences to provide updates and answer questions from professional-technical authorities across the CFHS.

The joint DFHP / D Dent Svcs / RCDC COVID-19 Intervention Team study related to efficacy of air filtration units is another example of a scientific knowledge product as well as joint collaboration.

A number of training products were developed centrally to better prepare staff to work safely in a pandemic environment (e.g. PPE use instruction). These were well received; however, would have better contributed to system efficiency if they had been available before clinic leaders were required to develop training locally to meet the immediate needs.

The rapidity of the changes required to provide care in a pandemic environment as well as the relative immaturity of CFHS's Quality Improvement (QI) and Performance Measurement programming, however, meant that many pandemic related adaptations, particularly for medical services, occurred without the benefit of structured QI methods. Such methods would have enabled CFHS to better assess and refine the changes, systematically identify risks to the quality and safety of the healthcare provided, and approach scale and spread with deliberation to facilitate efficient standardization of good practices and minimize duplication of effort.

Finding: CFHS's ability to respond efficiently to the pandemic was impeded by the lack of several important resources and enablers.

The demands of Op LASER exacerbated pre-pandemic system-wide and/or local challenges. Several factors were found to have impeded the efficiency of the clinics' response including:

• Pre-existing personnel shortages or issues
• Insufficient IT equipment and supports to implement BCPs at baseline and slow augmentation at most locations
• Pre-existing infrastructure limitations
• Lack of pre-existing platforms and capacities to facilitate efficient passage of information, both command and control and professional-technical
• Limited or delayed guidance and training supports from headquarters
• Limited IPAC program capacity
• Limited structures for consistent and effective collaboration between medical and dental services

Human Resources – Pre-pandemic shortfalls of health care personnel eroded the robustness, efficiency, and sustainability of the response to COVID-19. CFHS was well below Preferred Manning Levels (PML) in key health services trades and was actively staffing public service vacancies and working with Calian to fill open contractor positions before the declaration of the pandemic. Data from the last quarter of FY 19/20 indicates seven (7) of the 16 health services occupations were below PML, with Medical Officers at the Captain (working) rank at 55 % of PML. There was no surge capacity to accommodate additional work or compensate for diminished productivity and loss of staff needing to attend to pandemic-related social impacts. Senior Medical Authorities, already working at capacity, had significant workload added to their plates as linchpin public health advisors for their clinics and supported bases and units.

Attrition – always a risk for highly employable health services personnel – was cited by some as increasing during the pandemic, with its impact further exacerbated by an inability to move public service staffing processes forward in the context of overarching DND business continuity challenges.

Finally, it must be noted that some clinics also contributed personnel to both new pandemic-driven and ongoing operational taskings, regardless of CFHS efforts to minimize pulling personnel from clinics.

Efficient mitigations for personnel challenges were complicated by the asymmetrical impact of the pandemic across different healthcare service areas and across different categories of worker within CFHS's blended workforce. Contract HCPs were not able to be accommodated with paid leave or part-time remote work in the same way as other team members and were heavily relied on by medical clinics to sustain services. Conversely, in some dental clinics, contracted HCPs were placed in abeyance or their hours reduced. Unequal access to COVID-19 vaccination was also a significant concern for contractors. In short, there were perceptions of significant unfairness stemming from differences in how Defence Team members were managed during the pandemic. This perception, combined with the weight of overwork and personnel concerns associated with COVID-19 as a personal health threat, has negatively impacted team cohesion and diminished morale.

Back to top

Information Technology –IT resource constraints significantly exacerbated the human resources challenges described above. Questionnaires, interviews and site visits revealed a variable range of information technology available to support health services, which directly impacted the clinics' ability to provide and sustain essential services efficiently. For example, clinic business continuity plans modified for the COVID-19 crisis relied heavily on IT support. With the sudden shift to teleworking (IAW CDS direction) by CAF/DND, the demand for proper C2 capability overwhelmed the IT and IM resources. Some organizations had laptops and priority DWAN access, while others did not. Only one clinic reported being fully equipped by the supporting base to facilitate healthcare business continuity as a priority. The short and mid-term IT shortages resulted in workarounds, some of which were sanctioned while others were not. Some resorted to using unsecured freeware, cross-platform, centralized instant messaging (IM) and voice-over-IP (VoIP) technology such as WHAT's App^©and personal devices to maintain a modicum of communication.

Information management was also challenging. Pre-pandemic, there was no comprehensive, nationally accessible, and consistently used Information Management platform solidly in place for CFHS. Much of how the high volume of information important to coordinate the CFHS response to COVID-19 was collected, disseminated, and stored was created 'on the fly'. Tactical units found there was an overwhelming amount of information, at times, that needed to be sorted through to determine actions to be taken. Medical clinics needed to individually integrate information from a variety of sources early in the pandemic, resulting in a response tailored to local conditions but also diminishing efficiency. Similarly, they were largely on their own to interpret more general information and provide detailed guidance to their staff, often in the form of Standard Operating Procedures developed by each clinic in parallel.

The requirement to provide numerous ad hoc reports and returns up to headquarters in the absence of pre-existing databases also detracted from efficiency.

Infrastructure – The rapid and consistent institution of public health measures within CFHS spaces so that services could be safely sustained was also challenged by variations in clinic infrastructure, particularly in older establishments awaiting recapitalization. Infrastructure and base Construction Engineering support figured significantly as an attenuator or amplifier of SARS-CoV-2 spread in different CFHS settings. Differences in operatory room design (open vs. closed) across dental clinics impacted the ability of these clinics to institute required IPAC measures and, in turn, their ability to sustain or resume service delivery (see Figure 4). For medical clinics, each had differing capacity to adapt patient flow in ways that would minimize risk of transmission of infection based on the physical spaces they had to work with (see Figure 5).

Figure 4 Operatory rooms design

Figure 5 Pre-existing variations in clinic infrastructure

IPAC capacity – System-wide availability of IPAC expertise was significantly limited but of critical importance in the CFHS response to COVID-19. IPAC was not formally recognized and established as a program at CFHS headquarters (HQ) and there was only one IPAC certified Nursing Officer within CFHS who functioned as an IPAC advisor when needed prior to 2020. Further, IPAC functions in the clinics had generally been a secondary duty and not consistently performed. Of the clinics that were sampled, there was a correlation noted between having had on-site IPAC trained staff prior to the onset of the pandemic, and the effectiveness and efficiency of response (see Figure 6). Numerous informers credited their ability to access IPAC expertise as critical to being able to respond to COVID-19 and expressed concern that CFHS's capacity is only "one deep" given the importance of IPAC in delivering safe healthcare every day and not just in the setting of a pandemic. On the other hand, the CDO's Interim Clinical Directive was developed, evolved, and was implemented across Dental clinics to enable a safe clinical operating environment, while accounting for local conditions. These directives facilitated both patient and occupational safety.

Figure 6 Example of implemented IPAC measures in CFB clinics

Back to top

Other - Dental clinics had some pre-existing capabilities and resources that contributed to a generally more efficient response than their medical counterparts. These consisted primarily of a database providing information about dental service operations, a high degree of pre-existing familiarity with use of PPE among their staff, and an operational level HQ that streamlined information flow and developed and promulgated common COVID-19 related Standard Operating Procedures for clinics via a pre-existing and well-established national SharePoint platform.

It was noted that medical and dental services worked in silos at tactical levels, with few structures and processes in place to facilitate sharing of information and coordination of effort. This was a barrier to potential efficiencies – such as leveraging dental personnel's familiarity with PPE to assist in training medical staff.

Finding: CFHS overcame insufficient pre-existing inventory, an inefficient inventory management system and global supply chain disruptions and was able to sustain the supply of necessary medical PPE reasonably well to effectively meet COVID-19 response requirements.

Centralized pandemic stockpiles of PPE were not at target levels in amount or quality prior to the pandemic. Unavailability / unpredictability of funds prior to the start of the pandemic precluded having "hard" contracts in place to keep the stockpile up to date as well as continue the supply through the pandemic. Holdings at the Central Medical Equipment Depot (CMED) included expired products as there was insufficient up to date stock to replace it (see Figure 7). Best possible use of the PPE stockpile was, however, achieved through certifying expired stock for safe use, careful monitoring of PPE holdings and 'burn rates', having clinics take advantage of local supply arrangements, and reserving central stock for augmentation of clinic supplies only as required.

Figure 7

Some quality issues were due to disconnects between points of responsibility within CFHS for setting clinical standards for PPE and for its procurement.

At the tactical level, PPE supply was one of several factors limiting healthcare provision early in the pandemic. Perceptions of PPE sufficiency were varied: some medical clinics reported early shortages, while dental commentary from the same locations tended to be that supply was adequate but that this was due to judicious PPE management more than generous amounts of stock or a low burn rate. Overall, whether PPE supply was deemed adequate or not correlated with a clinic's ability to adapt services to decrease demand, leverage local supply arrangements, and/or access IPAC expertise to make optimal use of the stock that they did have and minimize wastage. Overall, there was no evidence of negative outcomes pertaining to the health or safety of CAF members or CFHS personnel due to lack of medical PPE, although worries about PPE adequacy, parameters for safe use, and availability were a stressor and contributor (in some cases) to mental health issues.

Pre-existing supply arrangements proved ineffective for procuring PPE on a large scale due to global supply chain disruption. The efficiency of the PPE supply chain was further decreased by an initial lack of coordination amongst federal organizations competing independently for limited contracts. This was resolved, but some contracts did not deliver PPE that met the expected level of quality. There also appeared to be a lack of coordination between medical and dental clinic supply chains, which may have lessened opportunities to optimize supply management.

Inconsistent N95 fit testing status of CFHS personnel and the stockpiling of different brands of masks with different fit profiles also detracted from the efficient supply of N95 masks to personnel. Another factor impeding efficiency of the response was the use of outmoded inventory management software that was not accessible at the tactical level, requiring ad hoc reports of PPE use in clinics to be generated at each clinic, transmitted, collated centrally and then reconciled with the centralized system.

3.1.3 Governance: To what extent was an effective governance framework well incorporated into the CFHS COVID-19 response portfolios?

Finding: Clinical governance within the CAF and CFHS was strengthened through the CAF's need to respond effectively to COVID-19.

The necessity of a whole of Defence Team response to COVID-19 as a health emergency solidified the role of the Surgeon General as the CAF's advisor on health. The Surgeon General was afforded increased direct access to high level CAF decision makers and was able to provide timely and relevant advice (supported by CFHS SMEs providing relevant decision support products such as epidemiology data and Fusion Centre reports). Similarly, the Director of Health Services Operations (DHSO) became more closely integrated into the Senior Joint Staff (SJS) to provide timely and unfiltered medical advice to strategic level decision-makers.

The approach taken by DFHP to provide verbal information to all operational, regional, and tactical clinical authorities concurrently on a frequent and regular basis, supplemented by published guidance as it became available, was effective and efficient. Most medical leadership teams indicated these regular teleconferences were one of their most important sources of HQ guidance.

On the other hand, Dental provided/received their own briefings (which included fusion slides) at the regional and tactical levels. This was supplemented by the RCDC COVID-19 Monitoring tool and CDO's Interim Clinical Directives.

Base/Wing Surgeons (B/WSurgs) played a critical role in integrating information from CFHS headquarters Subject Matter Experts (SMEs) and various civilian national, provincial and local public health and health regulatory organizations. While this process lacked the efficiency of the more standardized approach taken across dental clinics, it allowed more responsiveness to different local levels of COVID-19 risk.

Of the clinics sampled, a correlation was seen between an efficient and effective response and a strong integration of the local command/management and clinical governance functions (i.e. a high functioning B/WSurg and CO team in a medical clinic, or the combination Dental Detachment Commander role in a dental clinic), supported by pre-existing and well defined structures and processes for collaboration between service areas (e.g., primary care and mental health/psychosocial services). Clinics where one or the other of these two factors was less well established had more difficulty navigating the challenges of the pandemic.

Clinical governance of HSS on deployed operations worked well for the majority of missions sampled. Effectiveness and efficiency of the response was facilitated by a clear and straightforward clinical governance framework outlined through Op Orders and Med Plans, even if the mission pre-dated the pandemic's onset. The Canadian Joint Operations Command (CJOC) Surgeon and supporting health services planners were seen as effective in supporting CFHS's response to COVID-19 in deployed settings.

Finding: A strong operational level HQ overseeing a single line of effort across all tactical locations was a factor that contributed to consistency, synchronization, and efficiency in responding to COVID-19.

Although official designated a unit, 1 Dental Unit functions as an operational level organization exercising both command and clinical authority over all CFHS in-garrison dental clinics. Its headquarters is structured and equipped to take strategic HQ guidance and transform it into implementable direction and support for its subordinate clinics. This arrangement significantly facilitated a dental services response to the pandemic that was consistent, synchronized, and efficient.

In contrast, operational level command and control of CFHS medical clinics is split between two regional Health Services Groups (HSGs), except for the Ottawa clinic which stands alone as a National Level Unit reporting directly to the strategic level (and later Division) headquarters. The HSGs are also charged with significant force generation responsibilities, overseeing the Regular and Reserve Force Field Ambulances in their regions. Further, the organization and establishment of the HSGs were initially provided limited resources for the operational level oversight of in-garrison care program delivery. HSG limitations included a de-linking between command and clinical authorities at that level, with SMAs distributed regionally as Regional Surgeons without a single "Group Surgeon" integrated directly into the HSG command suites.

Overall, the design (including capacity and capability) of the operational level governance structure impeded the efficiency, timeliness, and consistency of the response of in-garrison medical clinics to the pandemic, when compared to the single and much more in-garrison care focused 1 Dental Unit HQ, the design of which facilitated a more consistent and efficient approach across the dental clinics.

The introduction of the CFHS Division concept a few months into the pandemic clarified the distribution of responsibility between strategic and operational levels of CFHS. Further consolidation of this concept may mitigate the fragmentation of the medical clinic command and control, as well as facilitate collaboration across medical and dental lines of operation to realize system level efficiencies.

Back to top

3.2 Cluster II Logistics of the response

Description: For the purpose of this evaluation, the Logistics of the response can be categorised into three sectors: 1. Epidemiological surveillance of COVID-19 in CAF, provided by the Directorate of Force Health Protection; 2. Institutional Lessons Learned as outlined in the Defence Lessons Learned Program^{Footnote 12} and provided through a number of different functions within CFHS; and 3. Emergency Communications provided by CFHS Public Affairs and the Directorate of Health Services Operation via the Fusion Center. Each sector had its own process, purpose, and time-bound deliverables.

3.2.1 Effectiveness: Epidemiological surveillance

Finding: CFHS effectively produced the epidemiological information required by the CAF to inform its response to COVID-19.

Epidemiology reports started sluggishly evolving to provide valuable and up-to-date information for decision-makers and clinical advisors. In addition, the sections within DFHP collaborated to develop new policies, provide evidence-based guidance and advice, and liaise with civilian counterparts at the national, territorial, provincial, and regional levels. Ever-increasing demands for reporting (e.g., daily reports of cases, outbreak investigations, advisories and situation updates on particular outbreaks), at times, exceeded capacity. However, no discernable public health harm resulted as additional or more timely information would not likely have changed recommended management.

Several (N15) different epidemiology reports were reviewed. The contents of the documents were relevant and provided end-users with information assisting the chain of command and professional technical personnel in COVID-19 decision-making and the passage of correct information. The array of documents in the cross-sectional sample contained surveillance reports, case counts, outbreak summaries, critical advisories, background information, and SARS-CoV-2 spread population modelling.

3.2.2 Effectiveness: Lessons Learned

Note: Lessons learned" (LL) terminology was used extensively by interviewees, found in many key documents reviewed, and referenced or described in published works. However, the term or definition was poorly understood and often used interchangeably to describe other constructs related to organizational learning. Such imprecise and inconsistent use of the term does not reliably convey whether the lesson was observed, transmitted, recorded, shared, or implemented. Instead, the term "lessons learned" is often used to explain the positive and negative outcomes of the organizational response to an incident and does not fulfill the expectations that a lesson was actually learned. For example, when Canada experienced an outbreak of Sudden Acute Respiratory Syndrome (SARS), a series of recommendations described as lessons learned to mitigate the "traumatic stress among frontline healthcare workers^{Footnote 13}" were not applied to COVID-19 stress-related challenges. In other words, they were not lessons learned, but rather, lessons observed or recommendations not actioned.

Finding: While activities were undertaken by CFHS to support organizational learning from the COVID-19 experience, a coherent, comprehensive, collaborative, and current lessons learned program tying these activities together and ensuring optimal effectiveness of the full LL function was lacking.

DOAD 8010-0, Lessons Learned (updated 22 July 2016), provided the most current published direction on Lessons Learned relevant to the Evaluation period. The DAOD, in conjunction with Canadian Forces Joint Publication (CFJP) A2, provides a well-defined and customizable framework supported by "how to" guidance on the proper application of the program's core principles. The evaluation team found, however, that although the Defence LL Program(DLLP) was technically implemented, CAF/DND did not have a coherent Joint LL function at the Level 0 (L0) to coordinate, standardise, and provide oversight and training for the L1 LL community [CJOC, CJAT and LL Branch Op LASER Initial Observations]. Due to its operational focus, the CJAT and the processes that support it are not designed to provide nuanced and contextual observations, best practices, and recommendations for most health services activities, including those of CJOC-generated areas of interest.

Despite the lack of a fully implemented DLLP, CFHS direction to conduct LL activities were inserted into official orders from the chain of command from the L0 level and down through cascading orders to the responsive directorates and clinics. Currently nascent and residing in D HS Q&P, the LL program had begun to develop a health services-grounded organizational learning framework serving operational, domestic and in-garrison health services-related activities. How the traditional military LL forms and processes would integrate into these broader organizational learning functions in place within D HS Q & P was still a work in progress.

As the pandemic progressed, health services personnel gathered information oft described as lessons learned, but were more properly categorized as lessons observed (after-action reports, reports and returns, and the result of SME (and sometimes not) opinion). Health services personnel reported receiving little guidance or training on correctly collecting field observations, let alone their subsequent incorporation into a structured lessons-learned framework. Perhaps the most published and scrutinized LL related documents generated by health services personnel were the raw observations from collective and individual experiences in the Long Term LTCFs. Meaningful direction, understanding and training on writing medical observations may have prevented the inclusion in the reports of content described as reflecting significant behavioural biases (such as mind-reading, emotionally charged hyperbole, reporting innuendo as fact) and unsubstantiated allegations lacking important evidenced-based data (e.g., stating that a resident had died of neglect, dehydration or malnutrition without a coroner's inquest or report^{Footnote 14}).

Notwithstanding the above, CFHS did undertake a number of activities under the auspices of other programs that contribute to organizational learning as conceptualized in DLLP doctrine. These included the reporting, analysis (e.g., the Rapid Assessment Protocole) and implementation of recommendations stemming from patient safety incidents related to COVID-19 adaptations, the application of Quality Improvement processes to the rapid roll-out of virtual care, and the active solicitation of observations related to mass vaccination administration which were fed directly into planning for Op VECTOR and captured in a published report. Finally, this Evaluation represents a robust and comprehensive approach to organizational learning from the COVID-19 experience.

Back to top

3.2.3 Effectiveness: Communications

Description: Public Affairs (PA). As outlined in CONPLAN LASER, the CF PA plan for response to a PI was expected to be active internally and passive externally as coordinated by ADM (PA) with Public Safety/Public Health Agency (PHAC) of Canada in order to maintain consonance with the overall federal government's message. Regional Joint Task Force HQs were to prepare PA plans IA W guidance provided at Annex X of the SOODO. [para 3 – coordinating instructions CONPLAN LASER]

Finding: CFHS PA was effective in collecting, contributing, co-ordinating, verifying, and publishing information internally.

CONPLAN LASER directs Public Affairs to take internal active and external passive roles during pandemic influenza. Given the high profile and political sensitivity of OP LASER, media content approval authority for Op LASER-related events rested with the Deputy Minister of Defence while the CFHS public affairs office adopted a passive posture, ensuring that approved PA products were communicated through the most appropriate outlet (e.g., Twitter, Facebook, main media outlets, CAF App, and CAF webpages). In addition, the office appeared to be COVID-19 current - collaborating with the offices of the SG, DSG, DFHP, and DHSO to be pandemic prepared passively. This passive posture negated, for the most, the need for a complex PA strategic and framework.

Evidence of effective internal communication can be seen in the multiple and timely social media posts. CFHS PA had multiple collaborative relations with other departments and directorates, ensuring that published information was positive, informative, and accurate. There were no noted corrections required following publications. It was not entirely clear if some of the ideas were initiated by CFHS PA or if it was suggested to CFHS PA, or if they were made aware of items that would be good to pursue. No data submitted reflected any changes in posture brought on by the pandemic.

Multiple mechanisms and types of communication platforms such as the Defence COVID Management Committee for rapid flow of information, social media platforms to CAF, Townhall communication and feedback, and Surgeon General Messages, appeared to be effective. They appeared to catalyse high vaccination rates in CAF members. The high efficacy of the mRNA vaccines and the high vaccination rate contributed to reduced morbidity and mortality among CAF members and civilian personnel, impeded transmission of COVID-19 in CAF facilities and sustained operational readiness of the force.

Finally, it is noteworthy that knowledge gathering, analysis and sharing improved dramatically with forming of the Fusion Centre within D HSO. Fusion Centre products facilitated informed decision-making both internal to CFHS and within the larger CAF.

Finding: CFHS PA's limitation to a passive media role with little authority for engaging external media likely resulted in some missed opportunities to highlight CFHS contributions to mitigating the impact of COVID-19 in a timelier way.

Given the high profile and political sensitivity of OP LASER, media content approval authority for Op LASER-related events rested with the Deputy Minister of Defence. With high-level strategic staff applying significant control at the tactical level, the CJAT assessment of Op LASER noted that "media engagement and connection with the public was infrequent and out-of-date information was sometimes reported when communications were issued at the higher level." For example, Public Affairs took a passive approach to publicizing CAF assistance to LTCFs and may have missed opportunities to foster more positive relations with the civilian population.

3.2.4 Efficiency: Epidemiological Surveillance

Finding: The dynamic nature of the pandemic, combined with the constant demands from multiple sources for epidemiology expertise (i.e., reports, advice and information), resulted in excessive demands on components of DFHP.

Universally reported by key interviewees was the incessant demand for epidemiology products and information. The exigences were often repetitive and redundant, with requests for data coming from all directions. To the staff within CFHS responsible for generating epidemiological data and products, it seemed that demands were not mitigated or filtered by senior leadership.

The epidemiology section within D FHP entered the pandemic incompletely staffed and all core business within DFHP needed to be stopped to meet the pandemic-related demands. Personnel fatigue (described as bordering on burnout) and resultant increased attrition exacerbated the impact of the pre-pandemic human resource shortages. They perpetuated the challenge of responding to the constant calls for epidemiology products.

The ever-increasing demands for epidemiological products (e.g., daily reports and updates, sitreps, outbreak reports, advisories, etc.) stretched the capability of staff. However, due to limited staff (e.g. secondment, burn-out, vacancies), the demands were not always fulfilled. Still, there was no evidence of public health harm as unmet information requests would not likely have changed recommended management. From the site visits, it was evident that health authorities were proactive and creative in using epidemiological data to its fullest.

3.2.5 Efficiency: Lessons Leaned

Finding: The lack of a systematic analytical framework for the management of LL observations impeded the efficiency of the LL function within CFHS.

As described in section 3.2.2 above, the CFHS lacked an overarching framework for LL activities and had not implemented several essential elements including LL training. This likely had a negative impact on the speed and efficiency of organizational learning related to COVID-19.

3.2.6 Efficiency: Communications

Finding: Both the CFHS PA function and the Fusion Centre were found to be efficient in the performance of their mandates.

Despite the limited resources, time constraints, and direction to adopt a passive external communication posture, the CFHS PA cell was able to use efficiently established communication channels to inform CAF members, external partners and the general public. To this end, essential up-to-date and refined information were provided to inform decision-making and policy.

3.2.7 Governance: Epidemiological Surveillance

Finding: Communication between DFHP experts, the professional technical network, and the chain of command was clear and effective.

Directorate of Force Health Protection (DFHP) Advisories 6695-09 and 6636-80 represented key documents for senior medical authorities and CoC, particularly at the tactical (i.e., clinic) level. The advisories were referenced consistently in clinic BCPs / BRPs. In addition, weekly DFHP-sponsored prof-tech teleconferences and voluminous email exchanges from prof tech net authorities and CoC provided meaningful opportunities for further DFHP input while providing feedback from target audiences. Lastly, DFHP used positional mailboxes monitored daily; guidance and policy SARS-CoV-2-related documentation was posted on a DFHP SharePoint and in OneNote.

The immense dis-coordinated daily demand for epi products created undue pressure on those responsible for responding to these demands. A common theme among respondents was the frustration with these demands that seemly came from "all directions" with little filtering from DFHP CoC. However, despite the pressures, the content of the epidemiology products was, for the most part, scientifically accurate, clear, complete, and concise.

3.2.8 Governance: Lessons Learned

Finding: A foundation for effective governance of the LL process within CFHS is in place at the strategic level but is not yet mature.

CFHS had made strides in implementing an Integrated Governance Framework, including establishing Quality Council, the terms of reference for which contain responsibilities that contribute to the DLLP process. However, the different processes supporting the collection and analysis of data relevant to organization learning are not all consistently in place and producing findings and recommendations for QC review and referral for action. Further, structures and processes to integrate QC functions with similar functions to support governance of LL at the operational and tactical levels are not yet in place.

Specific to LL through the evaluation process, evaluation program ARAs are well integrated. All governing bodies such as Evaluation Steering Committee (ESC) and Quality Council (QC) are in place, operationalised and provide guidance for the D HS Q&P to perform its function under the Directorate's evaluation mandate. This clear governance structure will support the completion of mandated COVID-19 LL through the evaluative process.

3.2.9 Governance: Communications

Finding: CFHS Public Affairs governance was a limiting factor to some of their activities but did not materially detract from their overall effectiveness.

CONPLAN LASER directs Public Affairs to take a passive role during Op LASER. As a result, the communication framework was straightforward and constrained. Approval authority rested with the Deputy Minister's Office, often resulting in delayed or inaccurate media and social content. Directed control over media and social media content negated the need for an elaborate communication template. Email analysis showed that PA remained agile and responsive to assist with product content when asked. That said, developing a communication management strategy would help track PA products in terms of the target audience, the objective of the material, who is most responsible, what the key message should be, which vehicle was best, and the timing of the released material.

Back to top

3.3 Cluster III Institutional health support of civilian health system through RFAs

Description: This aspect of the evaluation examines the support provided by CFHS to identified Long Term Care Facilities (LTCFs) in Ontario and Quebec, and the provision of personnel to support the contact tracing efforts of the Public Health Agency of Canada. Institutional support to the LTCFs was primarily focussed on providing nursing and personal care to dependant residents - often having several co-morbidities - and supporting infection control practices.

3.3.1 Effectiveness: Is the intervention achieving its objectives?

Finding: Overall, the CFHS was effective in force generating for the LTCF RFA and the assistance provided by CFHS personnel contributed to mission success.

Contributing to the conditions in these facilities and the decision to request assistance from the CAF was the significant negative impact that COVID-19 had on staffing, volunteers, and family members that typically assisted in the care of the residents. The deployment of CFHS and supporting personnel into LTCFs provided a much-needed stabilizing force until the LTCFs could resume operations under its own independent manning. The RFAs started with requests to support five (5) LTCFs in Quebec, which very quickly expanded to requests from the province for 1000 additional troops, while at the same time, Ontario submitted an RFA to assist in five (5) LTCFs.

Given the nature of the SARS-CoV-2 virus and the atypical nature of the healthcare mission compared to the kinds of military medical scenarios CFHS personnel train for, health services were pressed into becoming a more agile learning culture. Creativity and ingenuity were required to fill these requests and those anticipated to come in the future.

Finding: IPAC practices and PHMs implemented through the LTCF RFA mission were effective at mitigating the risk of COVID-19 to CAF operational effectiveness.

Despite working in LTCFs with very high rates of infection, the infection rate among CAF members working in LTCFs was low. This suggests that personnel were disciplined and successfully applied IPAC and public health protocols.

Finding: CFHS was effective in meeting the requirements of the RFA to support PHAC in contact tracing.

Thirty members from 1 Dental Unit were trained and force generated in response to the Health Canada/PHAC request for contact tracing assistance. Ultimately, as events unfolded, the supply of contact tracers exceeded a demand that fell short of anticipated levels. As a result, the call volume capacity of the assigned members of the CAF was not fully challenged and assessed. The objective of reducing data collection burden on PHAC was, however, successfully achieved by CAF members conducting 1270 calls to 1405 civilian contacts.

3.3.2 Efficiency: How well are resources being used?

Finding: The novel nature of the mission, lack of pre-existing plans, short timelines, and lack of timely and high level CFHS input into mission design and C2 led to inefficiencies throughout the RFA mission.

There was no CAF doctrine or organizational structure in place purpose-built to support novel requests for specialist or large-scale medically focussed domestic operations such as were seen during Op LASER, nor was there an easily deployable CFHS command and control (C2) element. Further, CFHS's position as an L2 impeded efficient operational planning and CFHS's ability to influence the design of command and control structures and functions that would be best suited to Op LASER's predominantly health and health services related activities and effects. The requirement to cascade Orders from CDS to MILPERSCOM in advance of CFHS – a significant force element - issuing Orders to its formations and units, created some inefficiencies. Parallel Orders coming down through CJOC to CFHS elements created some problematic overlap. From the perspective of some personnel on the ground, information from the CoC appeared to lack the appropriate staff work. Instead, it was seen as voluminous, overly detailed, and sometimes contradictory.

CFHS forces were not trained for nor familiar with caring for the patient demographic in the LTCFs. Just-in-time training and guidance were arranged to fill gaps in the knowledge and experience for those being deployed into the LTCFs. This training was more effective where civilian facility staff could conduct complete orientations. Many did not feel prepared to work in this capacity, however, comfort increased over time.

Similarly, training was required for those assigned to contact tracing as this specialised skill set and experience is not encompassed in the most health care occupations.

Finding: CFHS employed some innovative strategies to increase the efficiency of responding to the LTCF RFA in a resource-constrained context.

In order to maximize the ability to fulfill multiple and potentially varied tasks, Multipurpose Medical Assistance Teams (MMAT) were conceptualized and established. Significant numbers of Health Services Reserve Force members were mobilized, some of whom had clinical experience relevant to the LTCF setting. Within the LTCFs, shortfalls in the number of nursing and medical technicians to support the personal care of the residents were partially offset by employing other health occupations such as physiotherapy and dental personnel. Extraordinary clinical practice privileges were authorized by the Surgeon General for CFHS HCPs to optimize their ability to assist in providing care. Non-health care occupations were also engaged in general assisting and supplemental tasks with clear guidance provided on the scope of their duties.

Finding: A number of human resource related challenges caused inefficiency in force generating and sustaining CFHS personnel for the LTCF mission.

These challenges included:

• Pre-existing shortfalls in accessible information about the CFHS workforce;
• An initial failure to appreciate the importance of French language ability caused inefficiency in force generating CFHS personnel for the LTCF mission;
• A shortage of qualified personnel;
• The inability to employ all CFHS healthcare providers to their full scope of practice;
• HSS and logistics; and,
• No clear scheme to assess and prioritize health human resource requirements on an ongoing basis.

CFHS had no information technology enabled tool in place to holistically track numbers of available health human resources (HHR), their clinical credentials and qualifications, and readiness status, making the determination of deployable assets available cumbersome and dependent on the force-generating units to collect and provide the required information. The medical DAG process for this domestic operation was also not well understood or articulated. Many waivers were provided, resulting in members not being screened for fitness to deploy on this operation. Further, the Op Orders did not provide direction to ensure enhanced post-deployment screening.

The rapid execution of this large-scale domestic health services -centric deployment to Quebec did not duly consider the full scope of limitations that mismatched language abilities would have in working in a French language health care system. For example, Anglophone healthcare personnel faced challenges functioning in a Francophone environment – many raising concerns regarding liability, understanding facility guidelines, policies, and medical charts, and communicating with staff and residents. In addition, some Anglophone officers grew concerned about losing nuanced direction due to rough translation or not receiving translated written directives at all. Due to language issues, many members were transferred from Quebec to the main staging area in Borden, Ontario, resulting at times in a "surplus" of members awaiting the assignment of tasks at Camp Blackdown.

There was a general overall decrease in the deployable pool of human health resources due to the personal toll of the pandemic on households, children, family, school etc and the simultaneous loss of childcare and other typical family supports, again due to the global scale of the pandemic and the resultant restrictions. In addition, Health Services Reserve Force personnel became less available over time due to other commitments and cumbersome administration for securing or extending Class C contracts. Other military units could force-generate more significant numbers of general duty soldiers from non- health-care occupations, but this was limited by and dependent on the number of CFHS personnel that could be force generated in parallel.

Surprisingly, integral medical support was not incorporated into the deployment to LTCFs and, as a result, medical care was difficult to access for shift workers without considerable disruption to the team. Also, there was no evacuation plan with the appropriate COVID-19 protections in place for those deployed into LTCFs who required medical repatriation to their home base for outpatient-related illnesses.

Frustration and fatigue among CFHS personnel deployed to LTCFs were consistently reported. Some contributing factors appeared to be uncertain end date of deployment and unanticipated re-deployment to other facilities "hurry up and wait" scenario, confusion with regard to the chain of command, unfamiliar and at times hostile working environments, feeling ill prepared, re-defined or unclear scope of practice, inconsistent IPAC adherence within the facilities, PPE requirements and difficulty obtaining initial supplies, voluminous information from multiple reporting chains (i.e. prof tech / chain of command etc.), shiftwork with little down time, inconsistent quarantine protocols among various groups of deployed personnel, and finally while many providers felt positive about their experiences and impact in the LTCF, others struggled with the experience. There was an underestimation of the emotional and mental health toll of working in these LTCFs that were in poor condition due to the pandemic pressures.

There was a perception at the tactical level that some governments or institutions were using military personnel as "free specialist resources". It was clear as time went on that CAF members were deployed into LTCFs where the crisis was over. Moreover, certain healthcare providers were assigned to perform "general duty" tasks (non-clinical): washing, dressing, feeding, mobilizing, and toileting. This resulted in many personnel performing tasks well below their usual scope of practice - particularly the Med Techs - and negatively impacted overall efficiency.

Back to top

3.3.3 Governance: To what extent was a clinical governance framework incorporated into the CFHS COVID-19 response portfolios?

Finding: Insufficient involvement of CFHS in the development of the LTCF RFAs detracted from their effectiveness as foundations for the governance of the CAF's response.

RFAs exist to facilitate the provision of federal assistance in responding to emergency situations that are normally the responsibility of provincial / territorial or municipal governments to manage, but that are of sufficient magnitude that local capacity to respond is overwhelmed. Requests are mediated through Public Safety Canada, and the process is intended to ensure interdepartmental consultation and coordination of all departments involved in the federal response. For RFAs involving the CAF, the consultation takes place at the L0/L1 level and unfolds with aggressive timelines and limited time for staff checks if the organization with the relevant expertise is not directly involved.

In the case of the LTCF RFAs – in particular, the initial RFA with Quebec - the evaluation team found that the CFHS was not effectively represented at the RFA negotiation table to inform and shape the technical specifications and other obligations among partners in RFA. RFAs are intentionally designed to be quick to activate and to lack excessive detail that could fetter effective responses to developing situations; however direct inclusion of CFHS in the higher-level planning would have allowed for cross-jurisdictional health care provider licensing and regulatory issues to be negotiated at the outset and clarified in the RFA documentation^{Footnote 15}. This would have permitted much better use of limited resources (such as ensuring provincial recognition of Med Techs as healthcare providers in the case of the Quebec RFA). CFHS involvement in negotiating the terms of the RFA may also have resulted in better understanding of the importance of including well-defined parameters to prioritize facility requests, objectively assess their needs, and inform a realistic exit strategy from chronically understaffed LTCFs (e.g. desired effects, measurable standards, etc).

Finding: The initial Command and Control of the LTCF mission as drawn from the CONPLAN LASER was ill-designed to support the specialized nature of the LTCF mission.

While the CONPLAN LASER was ostensibly designed to facilitate response to a primarily health threat, it outlined a model of governance consistent with a more typical domestic operation (providing general manpower support to fires, floods, storms). Command and control structures and Orders issued to support the execution of Op LASER, therefore, borrowed heavily from existing SOODOs rather than from doctrine supporting fundamentally much more similar missions such as DART.

From the perspective of those on ground in the LTCFs, the chain of command was cumbersome and confusing, resulting in frustration with uncertainty of direction, increased workloads, and parallel reporting. This was more evident in the deployment of troops into Quebec where there were multiple reports of frustration and increased stress resulting from the rapidly changing and often little-understood changes in C2 structure after the mission started. The Ontario LTCF mission, initiated after those in Quebec, adopted a modified C2 structure whereby the CFHS was more or less the supported element. This construct appeared to alleviate some confusion regarding reporting and chain of command issues and helped ensure that personnel on the ground had unfettered access to the professional-technical guidance they needed. However, important disconnects remained at higher levels as evidenced by the inappropriately late engagement of CAF medical authorities (the Surgeon General) in advising on the management of the reports of clinical care and patient safety concerns coming from CFHS personnel in the LTCFs

Finding: Governance of the contact tracing mission was complicated by the absence of any formally structured, overarching agreement to clearly outline agreed objectives and terms.

As it was an agreement between federal departments, it did not fall under the RFA process. Nor, because the personnel were military, did the secondment of personnel from the CAF to Health Canada fall under established civilian human resources processes. As best as could be determined by the Evaluation team, the agreement for CFHS to provide personnel to HC for contact tracing was informal and not in accordance with any established procedures. The terms of the arrangement were never made explicit in a formal agreement. This resulted in some internal C2 confusion between the roles and responsibilities of SJS, CJOC, DHSO, and 1 Dental Unit. External clarification was required between Health Canada and DND. The absence of clear mission objectives and parameters likely contributed to the overstaffing of the requirement.

3.4 Cross-cutting issue GBA+: To what extent was the gender diversity, equity & inclusion mainstreaming policy incorporated into the CFHS COVID-19 response portfolios?

Background/Description: Gender Based Analysis Plus is an "analytical tool to support the development of responsive and inclusive initiatives, including policies, programs, and other initiatives^{Footnote 16} " that federal agencies, including the Department of National Defence, have been directed to use when developing policies, programs, or directions and when decision-making. According to Canada's Defence Policy (Strong, Secure, Engaged), routine consideration of gender and other identity factors can "increase operational effectiveness by drawing on the strengths of Canada's diverse and multicultural populations^{Footnote 17}.

Figure 8: Gender-Based Analysis Plus (GBA Plus) Diagram

In applying GBA+, many factors should be considered. The needs and experiences of different groups of people are influenced by intersecting parts of their identity, the context they are in and their lived experiences. Figure 8 highlights a range of factors that can work together to shape these experiences and outcomes. In this way, GBA Plus is an intersectional analysis.

As the research on the nature of COVID-19 increased exponentially during the first year of the pandemic, significant risk factors for contracting COVID-19 were identified, including age, but biological sex or other identity factors encompassed by GBA+ were not among them. However, as the pandemic progressed, evidence emerged suggesting that biological sex and gender differences do exist in areas such as mental health impacts of the pandemic and the natural history of Long-COVID.

The degree to which GBA+ was incorporated into the thought process of CFHS leadership as they developed guidance related to the pandemic circumstance is difficult to measure. No metric can determine the extent to which GBA+ was used implicitly or when an analysis identifies no issues but was not referred to directly as having been completed. It is also relevant to note that CFHS comprises a much higher proportion of women than the rest of the CAF, representing over half the health services workforce. This increases the likelihood that the perspectives and needs of women in CFHS's approaches to COVID-19 response were the starting point of the planning and not a consideration layered on through a GBA+ type analysis. In other words, micro-applications (i.e. tactical level documents) of GBA+ (i.e. within tactical level documents) are difficult to identify when they were designed for a known gender diverse population.

Finding: Few GBA+ related issues in the CFHS response were identified, although no formal mechanism existed in CFHS to apply GBA+ analysis systematically and consistently to relevant aspects of the COVID-19 response or to monitor GBA+ issues in the organisation.

Except for L0 and 4 HSG direction, the lack of explicit GBA+-specific guidance in the critical health services documents reviewed and commentary from strategic leadership, suggests no formal mechanism exists in health services to systematically monitor GBA+ issues in the organisation.

Institutionalizing a formal GBA+ process could have reduced the issues caused by large initial deployment of unilingual Anglophones to the LTCFs in Quebec, which resulted in an inefficient redistribution of this personnel out of the facilities and back to staging areas and the need to urgently force generate, unilingual Francophone or bilingual replacements. Apart from deployment inefficiency, effectiveness on the ground was also negatively impacted as described in Cluster III findings above.

Despite the lack of a formal COVID-19 specific GBA+ analysis framework, most of the defence team members interviewed or completing a questionnaire did not identify any GBA-related oversights. Further, the Evaluation team captured multiple actioned GBA+ consistent practices that could be used as a starting point to shape an institutional framework:

• Mandatory GBA+ training for each CAF member;
• Periodic health assessment questionnaire to include gender identification question;
• Clinic PPE stock reflected gender and demographic profiles of the population;
• Sampled clinks accommodated the needs of both staff and patients with young children via flexible work arrangements and schedules in extreme conditions (ie. imposed lockdowns, quarantine etc.) while preserving unit operations;
• Bilingual pamphlets to address women's health issues related to COVID-19 and services during the pandemic; and,
• Gender sensitive clinic design (gender neutral bathrooms, breastfeeding space).

Finding: CFHS lacked data on a number of identity factors in its workforce and patient population.

Reported epidemiology data, such as incidence and prevalence, were reported by biological sex and not gender. The inability of CFHIS and other personnel datasets to capture biological sex and gender may impede CFHSs' ability to understand and respond effectively and efficiently to less acute or direct health impacts of a diverse CAF population.

Similarly, the lack of data on identity factors beyond sex and gender may have positioned CFHS poorly to recognize and respond to GBA+ issues in its response to COVID-19, an issue which would have extended to the availability of GBA+ relevant evidence for the purposes of the Evaluation.

Back to top

4.0 Conclusions

Based on the sample, the Evaluation concludes that, overall, the response of CFHS to COVID-19 was timely and effective. The CFHS delivered its mandate during all phases of the pandemic.

The progression of SARS-CoV-2 in the CAF was effectively monitored through CFHS health surveillance and translated into knowledge products for use by local, regional or international CAF senior health authorities in formulating their advice on the health protection of CAF/DND personnel. Decision-making was supported by the timely development of a range of LL products, including epidemiological reports, communications platforms, and profound analysis of the health portfolio activities via the evaluation process. Different recommendations reflected the phases of the response and shaped strategic directions in uncertainty.

With the content of PA products resting at the DCDS level, CFHS PAs were limited in their response. That said, PA staff continued offering CFHS senior executives' advice and consistently post approved products on platforms such as Facebook^©and Twitter^©.

Multiple approaches to the generation of Lessons Learned were put in place, with varying degrees of success; still, they were able to capture and analyze complex information necessary for decision-making support relevant to the planning and execution of the sequential phases of the pandemic response and the improved planning for future pandemic-related emergencies.

OP LASER was a primarily medically focused Operation which would have benefitted from increased CFHS presence at the planning table and integrated into command and control structures, notwithstanding its L2 status and current doctrine for more generic domestic operations. Although hampered by a C2 structure that was not task-tailored for a predominantly health services mission, institutional support of F/P/T civil authorities through Requests for Assistance contributed to minimizing, mitigating or preventing the spread and impact of SARS-CoV-2 virus on the Canadian population and civilian health care systems.

The Evaluation found that the provision of health care in-garrison alongside the support of operational effectiveness of CAF was adequately preserved throughout the evaluation period. Several mechanisms were employed to ensure the sustainability of healthcare provision in a resource-limited setting. These included reducing and controlling the demand for care through better governance, redirecting patients to other sources of care, outsourcing operations where possible to optimize capacity given available staff, and having defence team members put in extra efforts (e.g. overtime).

However, the Evaluation identified several factors that negatively impacted how efficiently the CFHS was able to meet the demands of the pandemic and the governance of its response.

There were inadequacies in the CONPLAN LASER from a health services perspective as well as gaps between what was included in the CONPLAN and base/wing level operations Business Continuity Plans (where they existed) that led to a significant level of potentially preventable effort required to plan and execute the CFHS response while the emergency was unfolding. Further, they relied on resources, such as the availability of Information Technology, Personal Protective Equipment, and health human resources, that proved insufficient for the magnitude of the emergency.

Most importantly, the Evaluation found that the significant and prolonged force generation requirement and unrelenting demand for military healthcare services during the pandemic has exhausted personnel as well as negatively impacted the health and well-being of CFHS personnel – particularly front-line healthcare providers. This can potentially exacerbate pre-existing CFHS attraction and retention issues and further weaken the CAF's vulnerable military health care system. The Figure 9 below depicts the different sources of staff fatigue and burnout uncovered by interviews conducted during the Evaluation. These factors are identified and expanded on throughout the preceding sections of the report; however, the Evaluation team believed it was essential to highlight this as a pervasive and cross-cutting issue that, unless ameliorated, will imperil the CFHS's ability to realize the rich lessons to be learned from this pandemic and effect positive organizational change.

Figure 9

Back to top

5.0 Summary of Recommendations

The findings from this evaluation have resulted in 22 recommendations, summarized in the table below:

Back to top

Annexes

Annex A - Terms of Reference

Annex B - Logic Model

Annex C - Evaluatino Approach and Methodology

1.0 Evaluation Governance

This evaluation was guided by the principles and standards outlined by the Treasury Board Secretariat (TBS) Policy on Results^{Footnote 18}, Department of National Defence (DND) Ethical Policy for Evaluation^{Footnote 19} and Defence Administrative Orders and Directives (DAOD) on Lessons Learned^{Footnote 20}.

Moreover, for quality assurance, the evaluation plan was developed and implemented under the guidance of an Evaluation Steering Committee and the overarching oversight of the CFHS Quality Council and Health Services Board as demonstrated in figure below:

Diagram 1 Evaluation governance flow chart

*Note: Design aligned with CF H Svcs Gp modernisation initiative and demonstrates integrated governance framework:Authority, Responsibility, Accountability.

Back to top

1.1 Evaluation scope

A participatory approach, engaging military leadership along with diverse institutional representatives, was applied to develop Key Evaluation Questions (KEQ) to investigate how the evaluation criteria have been achieved throughout the most critical aspects of the CFHS response to COVID-19. In total, 327 potential KEQs were collected through broad scoping exercises with military and civilian Subject Matter Experts (SMEs) and prioritized validated by the Evaluation Steering Committee's (ESC) executive members. The final KEQs informed what information would be collected and how the criteria would be interpreted and analysed. The scope of the evaluation was balanced based on ESC members' perceptions on where potential lessons were likely most concentrated against Surgeon General strategic priorities, which highlighted areas under active improvement where lessons learned from the pandemic could be of most value to effect positive change. Organizational capacity to conduct the evaluation in a timely manner was also a consideration. For these reasons, the Operations Globe and Vector were not included in this evaluation. The diagram below contains summative information about the evaluation scope:

Diagram 2

Health portfolio by evaluation criteria. The evaluation assessed the performance in terms of four criteria^{Footnote 21}: effectiveness; efficiency; governance; and gender diversity and inclusion (see Table 1 below). Each criterion is a different lens or perspective through which the institutional COVID-19 response can be viewed; together, they provide a more complete picture of the response, the process of implementation, and the results.

Effectiveness: Is the intervention achieving its objectives?

The extent to which the intervention achieved, or is expected to achieve, its objectives, and its results, including any differential results across groups.

Efficiency: How well are resources being used?

A measure of how economically and timely resources/inputs (funds, expertise, time, equipment, etc.) are converted into results.

Governance: To what extent was a Clinical Governance Framework incorporated into the CFHS COVID-19 response portfolios?

Governance is the decision-making framework of an organization that clearly identifies the appropriate authorities, responsibilities and accountabilities of those who have been entrusted with making decisions related to ensuring the core objectives are met at various levels within institutions or organizations.

Gender & diversity: To what extent was the gender & diversity mainstreaming policy incorporated into the CFHS COVID-19 response portfolios?

The extent to which the gender mainstreaming policy incorporated into multiple CFHS programmes or special institutional health support operations. Gender mainstreaming is defined as "the (re)organisation, improvement, development and evaluation of policy processes, so that a gender equality perspective is incorporated in all policies at all levels and all stages, by the actors normally involved in policy-making".

1.2 Sampling strategy

The evaluation utilised a systematic purposive sampling approach. The sampling units were selected on the basis of priority activities and ensuring inclusive participation as had been defined in the scope of this evaluation. The selection also was informed by the portfolio analysis and CFHS Evaluation Steering Committee. Other selection criteria included availability of and access to relevant information.

The medical and dental clinics in the sample represented all three services (Army, Navy, and Air Force) and also included a Training Base, and CFHS Headquarters (HQ). The sites varied in size and scope of services and had a broad geographic representation from across Canada (see Diagram below).

Current CAF deployed operations with a component of institutional health support that were selected for inclusion in the sample included four international operations and one operation in the Canadian North.

The logistics of response Cluster sample included the Public Affairs section, the Directorate of Force Health Protection (DFHP) Epidemiologic Unit, the Directorate Health Services Quality and Performance (D HS Q&P) Lessons Learned programming and the Directorate Health Services Operations (DHSO) COVID-19 Fusion Center.

Institutional health support of the Canadian civilian health care system through RFAs included a sample of four Long Term Care Facilities (LTCFs) and the Contact Tracing augmentation mission.

Diagram 3

Back to top

1.3 Data collection and analysis

The evaluation used a combination of data collection methods and gathered both primary (mainly qualitative), and secondary (qualitative and quantitative) data. The main body of evidence was collected through the following methods: expanded desk reviews and analysis of strategic/operational HQ and tactical level key documents, Commander's Joint Operations assessments^{Footnote 22} and documents, relevant ADM (RS) audit reports^{Footnote 23}, CFHIS/ DentIS databases and published guidance in relation to specific activities; interviews with key informants; focus groups and; site observation. The figure below describes the five data collection methods and their respective data sources (Figure 3).

Indicators - documentable and/or measurable pieces of information regarding critical aspects of Op LASER activities - were developed for each evaluation question. Standardized data collection tools such as an analytical matrix, data collection protocols, questionnaires and an evidence rating grid (see Appendix 1) were designed to ensure the collection of valid and credible evidence and enhance inter-rater reliability. Structured approaches were put in place to document and synthesize the information gathered and a rating was assigned to each indicator under each question per criterion, using guidelines established to classify findings from "good" to "unsatisfactory". Each rating is shown in the Appendix 1 Analytical rubric for evaluation of CFHS response to COVID-19. Overall project ratings were obtained by calculating the average of all project indicator ratings.

Measurability: all data-points were evaluated by SMEs, using standardised templates, against established and measurable quality dimension criteria: availability, completeness, timelessness, relevance and perception of use during the observation.

1.4 Conceptual approach

The funnel diagram below shows 3 Level filtering processes to narrow down search and elimination of irrelevant information in order to generate the evidence. Each level demonstrates independent multistep analytical process:

I. Prioritisation & stratification of information by evaluation criteria via KEQs exercise

II. Stratification of unstructured data & single pieces of evidence by quality dimension criteria via Analytical Rubric (Appendix 1)

Analytical Rubric divides a project/program or performance into essential elements that are judged separately; it used for the appraisal of the quality of individual piece of evidence (document, Interview, observation) to gain rich understanding of broader perspectives of people and groups. The process includes:

1. Sorting out the information
2. Identification of significant findings
3. Quantification of evidence according to quality dimension criteria

III. Evaluation of structural evidence on given topic/evaluation criterion via Triangulation Rubric

This final process includes Synthesis and Quantification of structural data produced in Level II.

Diagram 4

1.5 Limitations and bias control

Given the size and complex interplay of personnel, structures and processes involved in CFHS's response to COVID-19, it was a challenge to attribute results to specific activities or factors and accurately ascertain the degree to which any given factor contributed to the achievement of a particular result. Further, it is important to note that the evaluation team's findings and recommendations have been formulated on the basis of evidence available to them. We cannot dismiss absolutely the possibility that evidence relevant to the Evaluation existed but had not been identified or evidence reviewed was not captured fully or with complete accuracy by the current evaluation methodology. We assess this limitation as low given the sampling methods and the broad array of expert Key Informers interviewed.

Even though the response under the CONPLAN and later Op LASER was goal oriented, baseline data had not been collected for majority of sampling units reviewed, nor had measurable targets against which to assess the achievement of those goals been established. In this circumstance, it was difficult to measure with precision the impact on target areas.

To control reliability of the evaluation, a detailed assessment of the processes and factors which may potentially contribute to a deviation of the result was conducted at the initial stage of planning and mitigations identified and incorporated in the evaluation plan. The following table outlines main anticipated biases along with a control strategy for each:

Back to top

Annex D CONPLAN LASER

1.0 Background

CONPLAN LASER exists as the CAF's contingency plan for domestic response to a viral pandemic – anticipated at the time it was developed to be caused by influenza (Pandemic Influenza (PI)). The last activation of Op LASER occurred in 2009 in response to the H1N1 influenza outbreak, and lessons learned served as the basis for the current iteration of the CONPLAN. As a result, CONPLAN LASER was updated in 2012, underpinning a response designed specifically to address another influenza-like pandemic.

1.1 Planning Assumptions

As a plan designed to respond to pandemic influenza, its Key Factors were based on the anticipated epidemiology of a PI virus, and associated assumptions were embedded throughout the CONPLAN. These included important assumptions around favorable timelines and availability of an influenza vaccine modified to address the PI and, effectively, end the pandemic. The predominant impact of PI on CAF readiness was anticipated to be due to the disease itself without much apparent consideration of the possibility and subsequent impact of an extensive and sustained imposition of public health measures on a global scale.

The table below compares 2009 and 2020 responses to the H1N1 and SARS-CoV-2 pandemics, respectively. It highlights some of the significant differences that resulted in a CONPLAN based on a flu pandemic that was insufficient to respond to the COVID-19 global crisis.

By the first year of each pandemic, there were 127 times more confirmed cases of COVID-19 than influenza caused by H1N1. The COVID-19 pandemic continues with spanning almost three years, while the influenza pandemic lasted approximately 13 months. PPE shortage pressures did not exist in 2009. Mask-wearing was mandated for only those working in healthcare facilities. SARS-CoV-2 was two to five times more contagious and prone to super-spreading events. Anti-virals that effectively controlled the spread and severity of H1N1 infection did not work for SARS-CoV-2. With a shorter pandemic course (i.e., approximately one year and two waves for H1N1 versus almost three years with 7 to 8 waves for SARS-CoV-2), the 2009 pandemic had minimal negative impacts on supply chains, commodity shortages, and the economy as a whole. These starkly contrasting differences between the two events likely led to a sense of false security that CONPLAN LASER and its resulting Operation were rigorous and tested enough to succeed against the impact COVID-19 was expected to have on Canada.

1.2 Timeline of the COVID-19 pandemic

The diagram below demonstrates the timeline of the COVID-19 pandemic and core activities associated with activation of CONPLAN LASER.

Diagram 5

Back to top

1.3 Scope and Mandate

CONPLAN LASER is described as "the overall CF medical Force Health Protection plan to ensure an operational capability to respond to the requirements of the Government of Canada is preserved" in the event of PI. The three strategic objectives of the plan are described as:

1. CF operational effectiveness and readiness maintained in Canada and overseas;
2. Essential health services are maintained for CF personnel by the CFHS; and,
3. Assistance to civil authorities for the maintenance of essential public services is provided as necessary, as part of the federal government response.

Specific to the objective of maintaining essential health services is included within the scope of the CONPLAN. However, this part of the plan would ostensibly be achieved through existing structures and processes (i.e. in-garrison CFHS clinics), which are part of normal day-to-day health services "operations" under CFHS command and control and not normally under the auspices of an operational force employer as is the model for most domestic and international operations.

Apart from being a stated objective of the CONPLAN, there is no additional content related to how CFHS clinics would need to define and be prepared to deliver essential health services within the context of a pandemic: the CONPLAN's focus is almost entirely on the requirement to manage and prevent influenza infection in CAF members and preparing to respond to requests for assistance from civil authorities.

1.4 Preparedness

As a contingency plan, the purpose of CONPLAN LASER was to enhance the CAF's operational preparedness for a pandemic. To do so effectively, the Phase I of the plan outlined everything that would need to be in place to achieve the state of preparedness effective and efficient activation of the remaining phases of the plan would require.

Review of the CONPLAN identifies as number of capabilities and capacities that the Op LASER required but were either not considered in the CONPLAN or included in the CONPLAN but not actually put in place.

Requirements for detailed information about case counts and distributions (not just deaths and hospitalizations) were not foreseen or planned for. Similarly, there was a Phase II requirement to be prepared to implement formal personnel strength monitoring and reporting but in the absence of a database capable of doing this.

Delays and shortages of critical medical supplies, including PPE, were foreseen, although having adequate stock to mitigate this was not identified as a key Phase I requirement.

Finally, it is worthwhile noting that Phase I required the development and validation (by exercising) of region-specific contingency plans for the response, as well as periodic review of Business Continuity Plans. Also included was a requirement for CFHS to "prepare, exercise and maintain a PI plan for CFHS". The evaluation team found no evidence that these requirements were met.

1.5 Command and Control (C2)

While the CAF has reorganized its force employment structure, the model of C2 outlined within the CONPLAN is consistent with CAF doctrine that requires a shift of resources from force generators to the force employer. This left CJOC as the supported Command for all aspects of Op LASER, with the important apparent exception of in-garrison healthcare, which remained under command and control of MPC. This left CFHS as a force element critical to the achievement of the CONPLAN's strategic objectives but under the command and control of two completely different L1s – MPC and CJOC. Without in-garrison care for provision of essential healthcare under the CONPLAN being identified as a "supported" priority to other L1s, there was no requirement for prioritization of clinic support from their Bases/Wings in terms of fulfilling IT or CE needs.

This complexity is reflected in Appendix 4 to Annex A of the CONPLAN, where even though the CONPLAN recognized LASER as a "unique operation" requiring a "unique C2 and reporting structure", it abided by standard C2 doctrine wherein CFHS would no longer be in command of its day to day in-garrison care mission but would be in support of Canada Command (now CJOC) as the 'supported command' that assumes responsibility for the whole of the "CF response to a pandemic influenza".

C2 outlined for health services was particularly convoluted in the CONPLAN's era of multiple operational commands that were later abandoned for the more straightforward CJOC concept. Under the CONPLAN, CFHS found itself providing aspects of HSS to the response through CANCOM, CANOSCOM, and CEFCOM^{Footnote 30} as well as under CMP's authority. Confusingly, there is a shift from Phase II, where HSS would appear to be primarily delivered via CMP, to Phase III where CANOSCOM would assume control of all CFHS tasks with RJTFs simultaneously becoming the primary C2 structure of the response under CANCOM and the responsibility for essential healthcare as a plan objective included in the ARAs given to CANCOM under the 'CF response to PI'.

Back to top

Annex E List of Definitions

Back to top

Annex F List of Documents Reviewed

ADM(RS). Audit of the Defence Team Response to COVID-19. (August 2021).

ADM(RS). Evaluation of Military Healthcare. (November 2018).

ADM (Mat). Procurement Administration Manual: PAM - Chapter 4 - EXECUTE PROCUREMENT

Barclay E. Predicting the pandemic. The Lancet World Report. (20 September 2008).

CANCOM (2012). CONPAN LASER Canadian Forces Response to an Influenza Pandemic.

CFHS. V 160707 GP CONPLAN (2007). Influence -Response to Pandemic Influenza.

CFHS.11 CF H Svcs C. Business Continuity Plan (BCP) COVID-19 Response.

CFHS.14 Wing Business Continuity Plan (BCP) Change 2. (Approved 22 March 2020)

CFJP. A2 providing amplifying detail on DAOD 8010-0, Lessons Learned.

CDC.1918 Pandemic Influenza Historic Timeline. (Accessed October 2022).

CFHS. 2 Fd Amb Clinic-CDU PPE Inventory Supply – Petawawa. (April 2020).

CDC. 2009 H1N1 Pandemic Timeline.

CMP CFHS (2020). Virtual care trial preliminary report.

GoC. Military response to COVID-19: Canadian Forces Health Services,

GoC. Military Operations. Operation LASER.

GoC, Ethical Policy for Evaluation. Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans – TCPS 2 (2018) t, article 2.5: Activities Not Requiring Research Ethics Board Review

CFHS - FRAG ORDER 002 – OP LASER, 20-01.

CDS- TASKORD 004-OP LASER 20-01-CAF RESUMPTION OF ACTIVITIES. (10 June 2020)

CFHS. 26 CF H Svcs C Business Continuity Plan (BCP) in Response to COVID-19.

CFHS. Aide-Memoire, Training in the Pandemic Environment – HSS Advice to 4 Cdn Div Bases/Units. Version 1.1.(25 June 2020).

CJAT. Assessment Report: Canada-Op LASER. (September 2020).

CJAT. Op LASER: Initial Observations. (Spring 2020).

CFHS. Business Resumption Plan Order for COVID-19, Petawawa. Version 4. (August 2020).

CFHS (2019). CFHS Quality Council Terms of Reference.

CFHS (2020). COVID-19 Pandemic Rapid Assessment Protocol (RAP).

CFHS. Chief Dental Officer's Interim Clinical Directive (additions2-6).

CDS. TASKORD for CAF Response to COVID-19 Outbreak. (2 March 2020).

CDS. TASKORD002-OP LASER 20-01 Force Employment.

CDS/DM. Directive on DND/CAF Operating and Reconstituting in a Persistent COVID-19 Environment. (February 2022).

CDC. Long COVID or Post-COVID Conditions. (July 2022).

CDC. Similarities and Differences between Flu and COVID-19.

CFHS.COVID-19 Surveillance Reports [2021-08-23 - N1883, 2021-05-25- N1804,2022-04-04-N6233-4)].

CFHS Evaluation Steering Committee. Meeting I & II Agenda and Record of Decision. (June 2020, February 2021).

CFHS. COVID-19 FUSION CENTRE – CAF Overview, Report [2021-09-13-N072,2021-06- N069].

CFHS. Summary of Discussions Canadian Forces Health Services (CFHS) Environmental Scan Threats, Weaknesses, and Strengths (TOWS), (1080-1 (HS Strat) (17 February 22).

CFHS. Aide-Memoire, Training in the Pandemic Environment – HSS Advice to 4 Cdn Div Bases/Units. Version 1.1.(25 June 2020).

CFB Esquimalt. Dental Unit PPE Stock Management Board (2022).

CFB Petawawa, 2 Fd Amb Base Clinic. CDU PPE Supply Inventory. (20 Mar 2020, 26 Mar 2021).

CMP CFHS (2020). Virtual care trail preliminary report.

DAOD 8010-0. Lessons Learned [Date of Issue 30 April 2004, Last Updated 22 July 2016].

1600-1 (Dent Det Edm) Op LASER TASKORD 002.After-Action Report.

DFHP. Infection Prevention and Control in Canadian Forces Health Services Centres and
Detachments. (15 June 2017).

DFHP. Summary of three COVID-19 outbreaks in Canadian Army (Autumn 2020, 19 November 2020).

DFHP. Advisory 6636-80. Novel Coronavirus (2019-nCoV): Current Situation, Surveillance and Reporting requirements and Infection Control Measures for CAF Health Services Units. (18 March 2020).

DRDC (2020). COVID-19 Defence Team Survey.

DFHP. Advisory 6638-80 Force Health Protection Recommendations. (30 April 2020).

DFHP. Advisory 6695-09 Force Health Considerations for Resumption of Business in DND/CAF Facilities in the COVID-19 Context. (10 June 2020).

DFHP. Summary of three COVID-19 outbreaks in Canadian Army. (19 November 2020).

DRDC. Weekly COVID-19 Information Digest. (17 April 2020).

4000-1 (Op Med), DSHO. Instruction – Information Management COVID-19. (25 March 2020).

CFHS. Evaluation Steering Committee Meeting II Agenda. Operationalization of Accountability: Summative Evaluation of the Delivery of Canadian Forces Health Services in COVID-19 Emergency. (February 2021).

European Centre for Disease Prevention and Control. SARS-CoV-2 Variant Classifications and Definitions. (2022).

European Centre for Disease Prevention and Control. SARS-CoV-2 variants of concern as of 8 December 2022. (2022).

Feasby, W.R. Official History of the Canadian Medical Services 1939-1945, Volume 2., Edmond Cloutier, C.M.G. O.A., Queen's Printer and Controller of Stationary. (Ottawa, 1953).

GoC. Frequency and impact of longer-term symptoms following COVID-19 in Canadian adults. (2022-10-20).

GoC. Introduction to GBA Plus.

Heymann et al, The next pandemic – when could it be? Chatham House. (23 February 2020, December 2022).

Health Canada (2022). Frequency and impact of longer-term symptoms following COVID-19 in Canadian adults.

Health Canada. COVID-19: Main modes of transmission.

3350-1 (J3) 1 Health Services Group. Task Order, Operation LASER. (12 March 2020).

Hirabara, S.M., SARS-COV-2 Variants: Differences and Potential of Immune Evasion. National Library of Medicine (NIH): Frontiers in Cellular and Infectious Microbiology. (December 2022).

JTF-U. Op. NANOOK ROTO 8 End Tour Report.

JTF-J5. Op UNIFIER ROTO 10 End Tour Report.

JTF Op IMPACT. Admin Instruction – Change of Command Handover.

JTF Op IMPACT ROTO 3. Dental Section End of Tour Report.

JTF Op IMPACT. Training Direction and Guidance.

JTF-U Op. REASSURANCE ROTO 8. End Tour Report.

Last J, A. Dictionary of Public Health

Last J, A. Dictionary of Epidemiology

MARPAC, CFB Esquimalt. COVID-19 Epi prevalence reports. (2020-2022).

Morissette, P. Public health practice within Canada's military healthcare system. Canada Communicable Disease Report, Vol.46, No.9, (3 September 2020).

Miami University. Differences in disease: How COVID-19 compares to SARS and H1N1.

Maunder R. (2004). The experience of the 2003 SARS outbreak as a traumatic stress among frontline healthcare workers in Toronto: lessons learned.

National Defence. Strong Secure Engaged: Canada's Defence Policy. (2017).

Newman, C. What can the Defence Medical Services learn from the COVID-19 pandemic in order to be ready for the future? BMJ. (4 August 2022).

6600-1 (NPME). Evaluation Directive of the Canadian Forces Health Services Response to the COVID-19 Pandemic. (26 October 2021).

NATO. NATO Field Manual for Medical Lessons Learned, 1st Edition. (November 2016).

OECD.: Revised Evaluation Criteria Definitions and Principles for Use. (2019).

Office of the Department of National Defence and Canadian Forces Legal Advisor: Contract definition of RFA for Contact Tracing.

Ontario Superior Court Of Justice. Electronically issued / Délivré par voie
électronique : 29-Jun-2020 Court File No./N° du dossier du greffe: CV-20-00643205-00CP

3350-Op LASER 20-01 (COS). Op LASER – JTFC Observations in Long Term Care Facilities in Ontario. (14 May 2020).

3350-Op LASER. 20-01– Lecons Apprises. (2 juillet 2020).

PHAC. COVID-19 pandemic guidance for the healthcare sector. (22 April 2020)

Rawling, Bill. The Myriad Challenges of Peace. (Ottawa, 2004).

Simpson, C.R. et al. Effectiveness of H1N1 vaccine for the prevention of pandemic influenza in Scotland, UK: a retrospective observational cohort study. Lancet Infectious Disease. (2012 Sep)12(9):696-702.

Surgeon General/Director General Health Services. Annual Strategic Planning Guidance – 2021/22.

3350-1 (SSO DHSO) Canadian Forces Health Services Group -FRAG ORDER 001- Op LASER 20-01. (17 March 2020).

TBS. Policy on Result 2.5 (1 July 2016).

WHO. Tracking SARS-CoV-2 variants. (2022).

WHO. Canada: cumulative COVID-19 cases, deaths and vaccination rates. (2022).

WHO. Severe Acute Respiratory Syndrome (SARS). (2022).

WHO. Coronavirus Disease (COVID-19) Dashboard with Vaccination Data.

WHO. Strengthening WHO preparedness for and response to health emergencies – Director General Report. (2022).

WARNING ORDER (WO)- MILPERCOM LESSONED LEARNED PROGRAM (5 October 2017).

WARNING ORDER. 1 DU Support to Health Canada (2 April 2020).

Back to top