Evaluation of Military Health Care

Alternate Formats

PDF Version
(636 KB, 40 pages)

Assistant Deputy Minister (Review Services)

ADM(RS): Assistant Deputy Minister (Review Services)
ARA: Authorities, Responsibilities and Accountabilities
B/W Comd: Base/Wing Commander
CAF: Canadian Armed Forces
CDO: Chief Dental Officer
CFHIS: Canadian Forces Health Information System
CF H Svcs Gp: Canadian Forces Health Services Group
CF Mil Pers Instr: Canadian Forces Military Personnel Instructions
CMP: Chief of Military Personnel
CO: Commanding Officer
DAOD: Defence Administrative Orders and Directives
D Med Pol: Directorate of Medical Policy
DND: Department of National Defence

FHCPS: Federal Health Claims Processing Service
FY: Fiscal Year
HLIS: Health and Lifestyle Information Survey
HSG: Health Services Group
MEL: Medical Employment Limitation
MPC: Military Personnel Command
OCI: Office of Collateral Interest
OPI: Office of Primary Interest
PAA: Program Alignment Architecture
PEQ: Patient Experience Questionnaire
PIP: Performance Information Profile
QR&O: Queen’s Regulations and Orders
R2MR: Road to Mental Readiness
SG: Surgeon General
SSE: Canada’s defence policy: Strong, Secure, Engaged

Overall Assessment

Military Health Care fulfills an ongoing need and aligns with government roles and priorities
Structural and functional gaps, including lack of formal authorities, may affect the delivery of health services
CAF members are satisfied with their health care
Overall cost effectiveness of Military Health Care cannot be assessed, but some good practices have been introduced

This report presents the findings and recommendations of the Evaluation of Military Health Care within the Department of National Defence (DND) and the Canadian Armed Forces (CAF). The evaluation was conducted by Assistant Deputy Minister (Review Services) (ADM(RS)) as a component of the DND Five-Year Evaluation Plan, in accordance with the Treasury Board Policy on Results (2016). The evaluation examined the relevance, effectiveness and efficiency of in-garrison health care and advice provided to Regular Force CAF members for fiscal years (FY) 2010/11 to 2017/18. Evaluation methods included interviews, surveys, document and literature review, and financial data analysis.

Program Description

Military Health Care is provided by the Canadian Forces Health Services Group (CF H Svcs Gp). CAF members access health services through in-garrison centres and through external health care providers, by referral. CF H Svcs Gp comprises approximately 3,900 military (Regular and Reserve Force), and public service personnel in 44 locations across Canada, the US and Europe. Total expenditures for the Military Health Care program were $721.5 million in FY 2016/17.

Relevance

This evaluation supports the continued relevance of the Military Health Care program. Regular Force CAF members need access to health services as they are excluded from provincial health plans. These services contribute to their medical fitness, a contributing factor towards the CAF’s operational effectiveness. Military Health Care is aligned with federal roles and responsibilities, as per the Canada Health Act, the National Defence Act, and Queen’s Regulations and Orders (QR&O). The delivery of health services by CF H Svcs Gp is aligned with departmental priorities. Canada’s defence policy: Strong, Secure, Engaged (SSE), 2017 places substantial focus on and provides an investment in an inclusive approach to health. This approach, known as Total Health and Wellness, aims to ensure the CAF Health System meets the unique needs of its personnel with efficient and effective care.

Effectiveness

Certain structural and functional gaps may affect the delivery of the Military Health Care program. Notably, the Surgeon General (SG) and the Chief Dental Officer (CDO) lack formalized authorities, responsibilities and accountabilities (ARA) for the clinical standards and quality of health services. Health-related policies, directives, instructions, and performance measurement system for Military Health Services are outdated and inadequate. Despite these challenges, CF H Svcs strives to ensure patient safety and quality of service through a number of internal and external mechanisms.

Evidence in favour of the effectiveness of the health care system can be found in the satisfaction of the users of that system. Surveys consistently show that most CAF members, their commanding officers (CO) and base/wing commanders (B/W Comd) are satisfied with the CAF health care system. Some areas of concern include wait times for appointments and communications between units and CF H Svcs Gp personnel, in particular relating to difficult medical employment limitations (MEL).

Efficiency and Economy

The CAF has largely been economical in its health system expenditures. However, its overall cost effectiveness cannot be assessed by this evaluation due to financial coding issues and lack of a satisfactory benchmark for comparison. Although expenditures for health services have risen over the seven-year evaluation period, this growth is much less than that for health care nationally. The CAF health system growth is in the form of increased spending on services from health-care providers external to the CAF. The evaluation was unable to determine the cause of the lower-than-expected growth in overall expenditures and whether there has been any impact on health service outcomes as a result. CF H Svcs Gp has successfully implemented some cost- saving measures, but the existence of high-risk transactions indicates possible inefficiencies in payments to civilian health care providers.

Key Findings and Recommendations

Table 1. Summary of Key Findings and Recommendations. This table provides a consolidation of report findings and recommendations.
Key Findings	Recommendations
*Relevance*
1. Regular Force CAF members need access to health services.	-
2. Military Health Care is aligned with federal roles and responsibilities.	-
3. The objectives of Military Health Care are aligned with departmental priorities.	-
*Effectiveness*
4. The Surgeon General and the Chief Dental Officer lack the proper authorities to deliver their required responsibilities as the heads of the medical and dental professional branches in the CAF and DND. Efforts have been underway to rectify this for some time.	1. Formally establish the authorities, responsibilities and accountabilities of the Surgeon General and the Chief Dental Officer. OPI: Commander, Military Personnel Command OCI: CF H Svcs Gp
5. CF H Svcs Gp has some mechanisms in place to ensure safety and quality of service, but more could be done to sustain efforts and monitor service effectiveness.	-
6. Health-related policy documents and directives for the CAF are not comprehensive and up to date.	2. Finalize the complete set of policies, directives and instructions on MELs. Examine all other policy areas to identify other urgent policy creation/revision needs, and develop a plan for the timely completion of these. OPI: CF H Svcs Gp
7. A formalized performance measurement system is not fully operational.	3. Develop and implement a performance measurement framework for the CAF health system, including the finalization of the Total Health Care Performance Information Profile (PIP), compiling performance indicators from across all program activities. OPI: Commander, Military Personnel Command 4. Dedicate human and other resources to performance measurement. Once the PIP is finalized, identify challenges to data availability and develop a plan to address any gaps. OPI: CF H Svcs Gp
8. Overall, CAF members, their COs and B/W Comds are satisfied with health care and advice received.	-
9. Wait times, likely caused by personnel shortages, are a concern to some in the CAF.	5. Prepare a report to CMP showing progress towards meeting targets for all occupations for which shortages exist. For occupations where sufficient progress is not being made, the report should describe how the recruitment and retention strategy will be adjusted to correct this. OPI: CF H Svcs Gp
10. Communication between units and CF H Svcs Gp personnel, such as through MELs, could be improved.	See Recommendation 2
*Efficiency and Economy*
11. Military health care expenditures have increased slightly, both in dollar terms and relative to total DND expenditures, over the evaluation period.	-
12. There is insufficient evidence to conclude on the cost efficiency of CAF health services, due to financial coding issues and lack of a satisfactory benchmark for comparison. CF H Svcs Gp has successfully implemented some cost-saving measures, but there may be some inefficiencies in payments to civilian health care providers.	-

Table 1 Details - Summary of Key Findings and Recommendations

Back to Table of Contents

1.0 Introduction

1.1 Context for the Evaluation

In accordance with the Treasury Board Policy on Results and the DND/CAF Five-Year Evaluation Plan, ADM(RS) has conducted an evaluation of the Military Health Care program. This evaluation examines the relevance, effectiveness and efficiency of in-garrison health care and advice provided to Regular Force members of the CAF from FY 2010/11 to FY 2017/18. A previous evaluation of Military Health Care was conducted by ADM(RS) in 1999. ADM(RS) also conducted an evaluation of Medical Support to Deployed Operations in 2014.

In addition, six recent internal and external audits and other reviews have included elements of Military Health Care:

ADM(RS) Audit of the Health Services Centre Recapitalization Project (June 2011);
Office of the Auditor General Transition of III and Injured Military Personnel to Civilian Life (2012);
ADM(RS) Audit of Canadian Forces Health Services (2014);
ADM(RS) Audit of the Canadian Forces Health and Physical Fitness Strategy and Canadian Forces Fitness Program Delivery (February 2014);
ADM(RS) Follow-up on Audit of Canadian Forces Health Services (forthcoming); and
ADM(RS) Advisory Report on Information Management in the Federal Health Claims Processing Service Contract (forthcoming).

Back to Table of Contents

1.2 Program Profile

1.2.1 Program Description

This evaluation covers the Military Health Care program, identified as element 4.1.8 in the DND/CAF Program Alignment Architecture (PAA).^{Footnote 1} Health care and health advice for the CAF is the responsibility of the CF H Svcs Gp, whose mandate is threefold:

Deliver health services;
Provide deployable health services capabilities; and
Provide health advice.

The CF H Svcs Gp and the SG are responsible for delivering CAF health services to CAF personnel, whether in garrison or deployed.^{Footnote 2} The CAF Spectrum of Care provides comprehensive medical, dental, and mental health care services, as well as comprehensive drug coverage, to all Regular Force members from time of enrolment to their release. This is in accordance with a clearly articulated set of guiding principles, and with very few coverage limitations.^{Footnote 3}

CF H Svcs Gp personnel provide outpatient care and treatment through medical and dental centres located across Canada and abroad. Members who require additional or extended care can be referred to civilian or private providers. Costs for approved procedures are paid for by CF H Svcs Gp through the Federal Health Claims Processing Service (FHCPS), administered by Medavie Blue Cross.

When deployed, the CF H Svcs Gp provides treatment and care by leveraging the human and material resources from its clinics, HQs, Field Ambulances, Field Hospital, and Central Medical Depot. Support to deployed operations is excluded from the scope of this evaluation.

CF H Svcs Gp provides health advice to the CAF chain of command on topics such as medical standards and fitness to complete required tasks, public and occupational health, operational force health protection and readiness, health aspects of operational planning, and maintenance or enhancement of human performance.^{Footnote 4}

With the exception of oral health advice provided by the CDO, health care and advice in the CAF is led by the SG, who plays several roles. The SG is Commander of CF H Svcs Gp and the functional authority on all matters related to medical care, carrying interconnected administrative authorities as Director General of Health Services.

CF H Svcs Gp comprises:

Approximately 3,900 military and public service personnel;^{Footnote 5} and
42 units and 62 detachments in Canada (as well as two detachments in the US and four detachments in Europe). This includes health services centres, dental detachments, field ambulances (mobile medical units), a field hospital, two schools, a trauma training centre, two medical simulation centers, one aeromedical evacuation flight, a research establishment, and a medical equipment depot.

It is one of the largest employers of paramedics in Canada, and Canada’s largest dental organization focused on a single-patient population.^{Footnote 6}

The CF H Svcs Gp has a centralized headquarters in the National Capital Region. Two regional formations support CF H Svcs Gp Headquarters: 1 Health Services Group (HSG) for western Canada and 4 HSG for eastern Canada. These formations manage and respond to domestic requirements and force generate military health services and equipment for deployed operations.

1.2.2 Resources

Table 2 shows the Military Health Care program expenditures, total DND/CAF expenditures, and total number of Regular Force CAF members, over the seven-year evaluation period.^{Footnote 7} In FY 2016/17, expenditures on health services were $721,477,049. The expenditures shown for Military Health Care cover the entirety of the program, including expenditures in areas excluded from the scope of this evaluation, and expenditures outside of the CF H Svcs Gp.^{Footnote 8}

Table 2. Military Health Care expenditures, total DND/CAF expenditures, and Regular Force CAF members, FY 2010/11 to FY 2016/17. This table displays the expenditures related to PAA element 4.1.8 Military Health Care, expenditures for total DND/CAF, and total number of Regular Force CAF members, for FYs 2010/11 to 2016/17.
-	FY 2010/11	FY 2011/12	FY 2012/13	FY 2013/14	FY 2014/15	FY 2015/16	FY 2016/17
Military Health Care Spending ($M)	693	711	708.5	698.6	694.7	694.4	721.5
Total DND/CAF Spending ($M)	20,298	20,219	19,978	18,764	18,454	18,666	18,606
Regular Force CAF Members	68,251	68,760	67,686	67,139	66,130	65,879	66,096

Table 2 Details - Military Health Care expenditures

Table 3. List of stakeholders. This table lists some of the key external stakeholders for Military Health Care.
-	Stakeholders
Other government departments	Veterans Affairs Canada Health Canada, Public Health Agency of Canada Statistics Canada Public Works and Government Services Canada
Other organizations in Canada	Provincial health care systems Private medical/dental clinics Medical and dental associations Accreditation Canada Commission on Dental Accreditation Canada Canadian and Provincial/Territorial medical and dental associations Provincial/Territorial regulatory authorities (medical, dental, pharm, nursing, etc. for licensure of our personnel) Canadian College of Health Leaders
International	North Atlantic Treaty Organization Other Allied Nations

Table 3 Details - List of Stakeholders

Back to Table of Contents

1.3 Evaluation Scope

1.3.1 Coverage

The evaluation covers the Military Health Care Program (4.1.8) within DND’s PAA, which, along with civilian health and wellness programming, falls within program 3.3 – Total Health Care – in Core Responsibility 3 (Defence Team) of the new Program Inventory.^{Footnote 9}

The purpose of the evaluation is to assess the relevance, effectiveness and efficiency of in-garrison health care and advice provided to Regular Force CAF members, covering the seven fiscal years from 2010/11 to 2017/18. Support to deployed operations is not covered in this evaluation as it was the subject of a previous evaluation.^{Footnote 10} Additionally, the following components of the Military Health Care program, which may be the subject of a future evaluation, are also excluded from the scope of this evaluation:

Health care to Reserve Force members
Health research and training
The relevance or appropriateness of the spectrum of care, meaning health care coverage

The evaluation’s scope focuses on activities 1 to 3 of the program logic model (Annex C) – that is, Advice, Governance and Planning, and Provision of Care. Assessment of program effectiveness was limited to the following two areas:

The extent to which CF H Svcs Gp has the appropriate authorities and governance to achieve its mandate; and
The extent to which in-garrison care and advice meet the need of the CAF.

1.3.2 Issues and Questions

The methodology used to gather evidence in support of the evaluation questions can be found in Annex B. The evaluation questions and corresponding indicators addressed by the evaluation are provided in Annex D.

Back to Table of Contents

2.0 Findings and Recommendations

2.1 Relevance

Key Finding 1: Regular Force CAF members need access to health services.

There is an ongoing, demonstrable need to provide health services to CAF members. Fit and healthy personnel are required to carry out the CAF mandate. As Regular Force CAF members are not covered under provincial health plans, a need exists to provide members with these services.

Health services, both in garrison and on operations, are needed in part to support the medical fitness of members. Medical fitness is defined as being free from illness, the ability to function efficiently and effectively, to enjoy leisure and to cope with emergencies.^{Footnote 11} The operational effectiveness of the CAF requires each member to ensure they are medically fit.^{Footnote 12}

Like all Canadians, CAF members require access to health services. However, while civilians in Canada have access to health services through provincial and territorial health care plans, the Canada Health Act explicitly excludes CAF members from these plans.^{Footnote 13} Therefore a need exists to provide them with access to health care services. Most Canadians also have supplementary health insurance coverage, often through a group plan from an employer, union or professional association.^{Footnote 14} The CAF’s health services fulfill the roles of both provincial and employer-sponsored health plans. This is important because a CAF member’s duties, whether deployed or not, can lead to illness and/or physical and mental injuries, which may require specialized care. Compared to the larger Canadian population, for example, military personnel experience higher rates of depression and post-traumatic stress disorder.^{Footnote 15}

Key Finding 2: Military Health Care is aligned with federal roles and responsibilities.

Military Health Care directly aligns with the role and responsibility of the Government of Canada, dating from the 1867 Constitution Act. Under the Act, Parliament has constitutional authority over the “militia, military and naval service, and defence.”^{Footnote 16} The National Defence Act establishes DND’s authority for the organization, efficiency, administration and good governance of the CAF. It does not explicitly mention health care as one of these responsibilities, although a proposed amendment aims to make this clarification. In place of this, QR&O establish CAF members’ entitlements to medical and dental care.^{Footnote 17}

Key Finding 3: The objectives of Military Health Care are aligned with governmental and departmental priorities.

The delivery of health services by CF H Svcs Gp is aligned with the priorities of the government and DND in the following ways:

SSE outlines a more inclusive approach to health known as Total Health and Wellness. According to this policy, DND will invest $198 million over ten years to implement the Total Health and Wellness Strategy.^{Footnote 18} SSE will “augment the Canadian Armed Forces Health System to ensure it meets the unique needs of our personnel with efficient and effective care, anywhere they serve in Canada or abroad.”^{Footnote 19}
DND’s Departmental Plan for 2018/19 commits to pursuing this policy direction by augmenting the CAF Health System, including initiating the incremental growth of the Medical Services Branch by 200 personnel.^{Footnote 20}

Back to Table of Contents

2.2 Effectiveness

2.2.1 Extent to which CF H Svcs Gp Has Appropriate Authorities and Governance to Achieve Its Mandate

Key Finding 4: The Surgeon General and Chief Dental Officer lack the proper authorities to deliver their required responsibilities as the heads of the medical and dental professional branches in the CAF and DND. Efforts have been underway to rectify this for some time.

The authorities of the SG and CDO have not been clearly articulated, and while amendments have been proposed to the QR&O related to medical and dental services, these amendments have not yet been approved.

As described under Key Finding 2, the CAF health system lies within the federal mandate and, therefore, the provincial/territorial regulatory schemes governing most aspects of health care in Canada do not apply. It is the QR&O that set out the regulatory framework for the CAF’s health care system.^{Footnote 21} However, the ARA of the senior-most positions in CAF’s medical and dental services – the SG and the CDO, respectively – are not set out in the QR&O, or in any other legislative or regulatory document. Some CAF policies and directives establish limited scope authorities for these positions, such as the authority to approve new CF H Svcs Gp policy and direction documents.^{Footnote 22} Furthermore, some publications provide a description of CF H Svcs Gp’s organization and governance structure, such as the Canadian Forces Medical Clinic. Interviewees have noted that this doctrine publication will be updated as part of the ongoing modernization of CF H Svcs Gp.^{Footnote 23}

Amendments to the QR&O have been proposed to clearly establish the authorities and roles of the SG and the CDO, and are awaiting legal review by the National Defence Regulations Section. The proposed amendments would ensure that the SG and CDO have the explicit legal authority to carry out their assigned functions. Notably, they would allow them to:

Authorize personnel to carry out clinical activities that may be restricted under provincial laws and regulations;
Promulgate clinical and professional policies and standards that apply to personnel who perform clinical duties within the CAF, regardless of their command relationship with the CF H Svcs Gp; and
Establish clinical and professional performance standards that are binding on civilian contractors providing health services within the CAF.^{Footnote 24}

CF H Svcs Gp personnel note that the amendments are currently in the hands of the National Defence Regulations Section within the Department of Justice, and there is currently limited capacity to advance them.

ADM(RS) Recommendation

Key Finding 5: CF H Svcs Gp has some mechanisms in place to ensure safety and quality of service, but more could be done to sustain efforts and monitor service effectiveness.

The safety and quality of services offered by the CF H Svcs Gp are ensured in several ways: through internal governance committees and programs, and through external validation through a non-profit independent organization, Accreditation Canada.

Several committees exist in CF H Svcs Gp to ensure safety and quality of health services. For example, the Surgeon General’s Clinical Council is responsible for making recommendations to the SG on clinical standards and quality assurance.^{Footnote 25} A similar Dental Clinical Council also exists. The Quality and Patient Safety Advisory Committee assists CF H Svcs Gp Leadership in overseeing and ensuring the quality and safety of health services provided throughout the organization. It also provides oversight and guidance to the CF H Svcs Quality and Patient Safety program.^{Footnote 26} This program contributes to safe and high-quality care and is “committed to activities that build excellence and innovation through improvement initiatives, prioritizing a culture of safety for patients, being fully compliant with internal and external standards of care, and measuring the quality of care.”^{Footnote 27}

Despite these committees and programs, CF H Svcs Gp has difficulty sustaining quality assurance and improvement activities, and struggles to ensure all committees meet regularly and fulfill their mandates. For example, the Quality and Patient Safety Advisory Committee has not met since 2016 due to the need to re-evaluate its mandate and membership. This effort has been delayed due to competing priorities. CF H Svcs Gp personnel note that the Q&PS program is struggling to maintain and evolve services to keep pace with civilian health system standards, after the loss of Q&PS personnel in the clinics during a period of departmental budget cuts.

CF H Svcs Gp does not sufficiently monitor the success of quality and safety initiatives, due to ongoing challenges regarding its performance measurement system (see Finding 7). CF H Svcs Gp has identified this and recently created a Directorate of Quality and Performance which will consolidate and develop clinical audit and program evaluation, special investigations, performance measurement, quality improvement and patient safety, and risk management. This additional capability is not yet in place as it is reliant on anticipated growth as part of the Canadian Forces Health Services Modernization.

Accreditation Canada is an independent external organization which assesses health care services against established standards for quality health care. It reviews clinics through eight quality dimensions – Population focus, Accessibility, Safety, Work life, Client-centered Services, Continuity of Services, Appropriateness, and Efficiency. In a 2017 report, Accreditation Canada found CF H Svcs Gp clinics in compliance with all eight areas with a rate of 81 percent to 100 percent and granted the CAF health system “Accredited” status.^{Footnote 28}

In their 2013 report Accreditation Canada outlined the need for CF H Svcs Gp to “pursue the development of standardized clinical outcome measures that can be reported and compared at the unit level and across the units. This will allow for clinically relevant indicators and improve the opportunity to leverage some of the best practices which may be in place on some bases.”^{Footnote 29} The 2017 Accreditation Canada report noted progress in this domain:

Research analysis and literature reviews are used to identify best practices and develop clinical practice guidelines for the health providers. The results and outcomes are monitored thorough a clinical audit process.^{Footnote 30}

Specifically, CF H Svcs Gp is in the early stages of conducting a clinical audit of the treatment of low back pain in clinics across the CAF to assess how well they are meeting standards and to measure consistency across clinics. This audit will provide information on efficiency as CF H Svcs Gp will be able to assess whether clinicians are following best practices and the resulting effect on resources. For example, CF H Svcs Gp will be able to determine the standard, or best practice, for when a patient should be referred for diagnostic imaging, and how actual referrals align to the standard. After this initial audit of low back pain, CF H Svcs Gp intends to continue to audit the management of other important health conditions, contingent on Directorate of Quality and Performance’s capacity to do so.

Key Finding 6: Health-related policy documents and directives for the CAF are not comprehensive or up to date.

Document review and interviews conducted for this evaluation revealed a few gaps in health-related policies and direction documents, and a need to update many of those available.

Policy and direction on health-related matters in the CAF are set through a number of vehicles (cumulatively referred to here as policies). These documents follow the hierarchy below, in order of descending precedence:

Acts of Parliament (e.g., National Defence Act)
Regulations (e.g., QR&O)
Orders (e.g., Defence Administrative Orders and Directives (DAOD))
NDHQ Instructions (e.g., Canadian Forces Military Personnel Instructions (CF Mil Pers Instr), Canadian Forces Medical Orders)
CF H Svcs Gp policy and direction documents

Although these policies should be kept up to date, they often are not. DAOD and CF H Svcs Gp policies must be reviewed at least every five years, while CF Mil Pers Instr must be reviewed annually.^{Footnote 31} However, a substantial number were last revised more than five years ago, including all ten DAOD and 16 out of the 17 CF Mil Pers Instr within CF H Svcs Gp’s remit.^{Footnote 32} It is important to note that a review does not necessarily result in a revision; as a result it is possible that some of these policies are up to date despite the lack of revision.

By contrast, CF H Svcs Gp policies appear more up to date. Approximately two-thirds of these documents were revised within the last five years.^{Footnote 33} However, there is variability within these policies. For example, more than 60 percent of the “1000 Series” – policies related to dental care – are more than five years old.

A key example of the need for updated policy is in the area of suicide prevention. The Canadian Forces Administrative Order on suicide prevention from 1996 has not been updated despite recommendations arising out of expert panel reports in 2010 and 2016.^{Footnote 34} The CAF/VAC Joint Suicide Prevention Strategy identifies an updated policy in this area as an action item, with a target date of 2019.^{Footnote 35}

In addition to the need for updating existing policy instruments, interviews and document review revealed the need for new policies in areas where gaps exist. Two examples are as follows:

Governance: As mentioned in Finding 4, changes to the governance structure of CF H Svcs Gp are coming as part of ongoing modernization. An attempt to revise the QR&O to establish the authorities of the SG and CDO are in progress. CF H Svcs Gp policies, such as 3100-04 on the Group’s organization and role, are under review and currently unavailable. A description of the governance structure can be found in certain documents, such as the Canadian Forces Medical Clinic, but this doctrine publication is over a decade old and interviews have revealed that it also is in need of significant updating as part of an ongoing modernization initiative.^{Footnote 36}
Medical Employment Limitations: MELs are the means by which medical professionals communicate to the chain of command any limitations on a CAF member’s ability to carry out their tasks and duties, resulting from a health issue. According to the Canadian Forces Medical Standards, “the importance of clearly stated MELs cannot be overstated.”^{Footnote 37} There are a few policies relating to MELs, but the Medical Standard publication identifies five instructions on MELs that are currently in development.^{Footnote 38} There are currently no instructions that set minimum standards for clarity and timeliness of MELs. As discussed in Finding 10, the timeliness, clarity and consistency of MELs is an area of concern among many unit COs, who must act on the information contained in MELs.

Interviews have revealed that, due to workload constraints and the specialized nature of health services policy, CF H Svcs Gp currently has limited capacity to create and revise policy. CF H Svcs Gp intends to address this capability deficiency in its modernization initiative, however capability will remain an issue in the near future.

ADM(RS) Recommendation

Key Finding 7: A formalized performance measurement system is not fully operational.

A functional and implemented performance measurement system is required for the overall governance and management of CAF Health Services. Previous reviews and this evaluation found that this system, including performance measurement data collected through the system, is in need of improvement. Correspondingly, the Military Health Care program does not yet have a final Performance Information Profile (PIP), as required by the Treasury Board Policy on Results (2016).

A 2014 ADM(RS) audit of CAF health care found that reporting of performance measurement data has largely been ad hoc and much of the data is incomplete and insufficient to support decision making.^{Footnote 39} Similarly, the 2017 Accreditation Canada report mentions the need for improved performance measurement, and encourages CF H Svcs Gp to “increase its emphasis on performance measurement with indicator development, monitoring and evaluation at all sites.”

In the follow up to the ADM(RS) audit, CF H Svcs Gp has reported their performance measurement framework has continued to evolve, albeit slowly due to limited capacity and the disruption of the recent transition to the Departmental Results Framework. Logic models and draft indicators have been developed for CF H Svcs Gp programs. However, document review and interviews confirmed there is a continued lack of evidence-based performance measures in the implementation and overall management and delivery of the CAF health system. In particular, a performance management framework focused on the health system’s strategic objectives at the level of Comd CF H Svcs Gp/Director General of Health Services remains in development.

CF H Svcs Gp does not have an effective information system to support performance measurement in clinical service program areas. Canadian Forces Health Information System (CFHIS) and its dental equivalent, the Dental Information System, were on the leading edge of medical record keeping in Canada when launched in 1999. However the systems were never designed with the idea of statistical analysis or cross referencing with other data sets and survey tools. Interviews with CF H Svcs Gp personnel found that improvements and advances in similar technologies have surpassed the capabilities of CFHIS, and it is now considered obsolete. Attempts to extract performance measurement data from these systems for the evaluation were unsuccessful. Replacement with a modern system could, among other things, integrate medical and dental information, improve access to health care, support best practice clinical decision making at the point of care, and facilitate data analysis to support decision making.^{Footnote 40}

A formalized performance measurement system is also needed to support departmental reporting as a requirement of the new Policy on Results. Performance Information Profiles are required for all programs outlined in the Departmental Results Framework. At present the Total Health Care program (which includes Military Health Care) does not yet have a final PIP.

ADM(RS) Recommendation

2.2.2 Extent to which In-Garrison Care and Advice Meet the Needs of the CAF

Key Finding 8: Overall, CAF members, their COs and B/W Comds are satisfied with health care and advice received.

Surveys of commanding officers (COs) and base/wing commanders (B/W Comds), and of CAF members more broadly, show that the majority are satisfied with the health care and advice they receive from CF H Svcs Gp. Patients – the beneficiaries of the health services – appear to have greater satisfaction than health care providers.

CAF satisfaction with health services was assessed through three surveys – two implemented by CF H Svcs Gp, and one carried out for this evaluation. The Health and Lifestyle Information Survey (HLIS) is a quadrennial survey of the CAF, carried out by Directorate Force Health Protection within CF H Svcs Gp, which focuses on health status with some questions on health care utilization. The most recent such survey was conducted in FY 2013/14. The second CF H Svcs Gp survey is the Patient Experience Questionnaire (PEQ), administered in FY 2017/18 to patients visiting a CAF medical or dental unit. To get perspectives from those in positions of command, a survey of B/W Comds and COs was carried out as part of this evaluation. Details on the methodology for this survey are described in Annex B.

Overall, the surveys show the majority of respondents are satisfied with the medical, dental, and mental health care provided by CF H Svcs Gp. As shown in the chart below, B/W Comds and COs are mostly satisfied with health care delivery in their units. Both the HLIS and the PEQ report that in excess of 80 percent of respondents were satisfied with the health services they received overall.^{Footnote 41} The lower ratings for mental health seen in the survey of B/W Comds and COs were not observed in the surveys of the CAF population.

**Figure 1. Satisfaction with operating hours, wait times, and effectiveness of health care.** This figure shows the percentage of B/W Comds and COs surveyed who agreed or strongly agreed with the following statements: 1) operating hours are appropriate; 2) wait times are acceptable; and 3) services are effective in improving health/readiness.

Figure 1 Details - Satisfaction with operating hours, wait times, and effectiveness of health care

Satisfaction appears to be greater among patients than among health care providers. Across survey questions, respondents from Military Personnel Command (MPC) were the least satisfied of all the L1 organizations. Most MPC respondents were CF H Svcs Gp personnel.

In addition to health care, the survey of B/W Comds and COs also asked about the advice they received from CF H Svcs Gp. A large majority of respondents, across all sub-groups, agree that CF H Svcs Gp provide them with advice and support to ensure their unit’s training and work activities minimize the risk of injury. The large majority of B/W Comds, in particular, agreed that CF H Svcs Gp understands their unit’s work. An area of concern, particularly for COs, was in communications around ill and injured members. This is described in more detail in Finding 10.

Key Finding 9: Wait times, likely caused by personnel shortages, are a concern to some in the CAF.

Despite the satisfaction with health services overall as per the previous finding, many CAF members have concerns about wait times for health care appointments. Since CAF wait times are believed to be linked to personnel shortages, CF H Svcs Gp’s plan to increase attraction and retention may help shorten them over time. With the new defence policy, there is anticipation of important CF H Svcs Gp personnel growth over the coming years. However, the extent, nature and sufficiency of the growth is not known at this time.

Evidence on satisfaction with wait times is mixed, but is generally lower than satisfaction with other areas of health care. According to a survey of Regular Force CAF members, roughly half reported being satisfied with wait times for health and dental care appointments, with another quarter being neutral.^{Footnote 42} In contrast, another questionnaire administered to CAF members found respondents mostly agreed they could book an appointment in a reasonable time.^{Footnote 43} The reason for this discrepancy is unclear, but may be due to the fact that the latter survey included Reserve Force members (as opposed to just Regular Force for the HLIS), or because of differences in response options provided. The survey of B/W Comds and COs conducted for this evaluation found greater satisfaction for medical wait times than for dental or mental health. While about three quarters of COs and B/W Comds agreed that medical wait times are acceptable, B/W Comds were significantly less satisfied with wait times for dental and mental health, with only about half agreeing that these are acceptable.^{Footnote 44}

Survey results on wait times cannot be corroborated with actual wait times. With a few exceptions, such as mental health services, CF H Svcs Gp does not currently track wait times for appointments.^{Footnote 45} However, the HLIS provides some evidence that wait times may be longer for CAF members than for the Canadian public. The HLIS found that more than half of Regular Force personnel waited two weeks or more for an appointment for a routine, ongoing, non-urgent health problem, compared to a two-day wait period for the average Canadian.^{Footnote 46} CAF wait times have not changed significantly between 2008/09 and 2013/14, the two most recent years of the HLIS. Almost three-quarters (73 percent) waited two weeks or more for non-urgent dental problems. However, this data is self-reported; CF H Svcs Gp may wish to adopt metrics of wait times into its performance measurement system.

Of particular concern, according to B/W Comd and CO survey respondents, are wait times for specialists (such as dental specialists and psychologists), sick parade, and aircrew medicals. Respondents noted that long wait times can result in disruptions to the work day and inability of CAF members to deploy, and can impact the efficiency of units. This can be a particular challenge for the Royal Canadian Air Force because of a directive that “grounds” aircrew who do not get their required biennial health assessment on time.^{Footnote 47} According to respondents, this often results in flight delays or insufficient staff on flights.

A number of studies external to the CAF have associated wait times for health care services with negative health outcomes, including mortality.^{Footnote 48} However, a Statistics Canada study found the risk of death from all causes to be lower among the CAF than for the general Canadian population.^{Footnote 49} If wait times are affecting the health of CAF members, there is no evidence of this in terms of mortality. Impact on patient morbidity and/or timeliness of return to medical or dental fitness is unknown.

Many interviewees and survey respondents for this evaluation, as well as a review by Accreditation Canada, linked these delays to personnel shortages in CF H Svcs Gp.^{Footnote 50} Again, both interviews for this evaluation and previous reviews have found that recruitment and retention have been key challenges, due in part to bureaucratic requirements.^{Footnote 51} In the three years to FY 2015/16 (the three years for which data are available), the combined military and civilian staff complement in CF H Svcs Gp shrank by 7 percent, to 3,901.^{Footnote 52} This may be causally related to the increase in use of external civilian health care providers as discussed in Finding 11.

Staff shortages have been linked to other challenges beyond wait times. For example, B/W Comd and CO survey respondents noted medical services are often unavailable on site and/or insufficient, resulting in the need to travel for medical services and, consequently, to disruptions to the work day and/or operations. Some respondents also commented that CF H Svcs Gp lacks the capacity to address the special operational requirements of high-readiness units.

CF H Svcs Gp has recognized personnel shortages in several occupations including medical officers, pharmacists and social workers and have identified possible attraction and retention solutions to address the shortages.^{Footnote 53} Over the longer term, augmenting the CAF health system is a priority initiative of the new defence policy, including growing the Medical Services Branch by 200 personnel.^{Footnote 54}

ADM(RS) Recommendation

Key Finding 10: Communication between units and CF H Svcs Gp personnel, such as through MELs, could be improved.

There is a concern that consultation and communication between CF H Svcs Gp and B/W Comds and COs is insufficient. In particular, B/W Comd and CO survey respondents reported issues with the quality and frequency of communication between the medical officers and COs. According to respondents, clinicians do not provide timely feedback on medical assessments to B/W Comds and COs, and little to no additional information and clarification is provided when medical leave is granted. This lack of consultation may negatively affect the units and impede the facilitation of members’ recovery. Survey respondents expressed a particular concern about the lack of consultation with the chain of command in the process of assigning MELs.

While the majority (58 percent) of B/W Comd and CO survey respondents were satisfied with the timeliness and clarity of MELs, it was the area of lowest satisfaction of all areas surveyed. Many respondents stated that MELs are not timely and the process needs to be streamlined. Perhaps because of lack of formal guidance (Finding 6), respondents felt that MELs are often unclear, vague, and inconsistent across bases. Many respondents commented that there seems to be no standardized method of assigning MELs, and individual MELs are sometimes confusing or contradictory. Additionally, some respondents felt that MELs are inappropriate or impose too many limitations, which they linked to the failure of medical personnel to fully understand the needs of CAF members and units, leading to potential system abuses.

It is, however, impossible to determine whether these perceptions are accurate. Commanders, personnel managers and CAF leaders in general receive no training or instruction on how to manage routine personnel matters related to illness or injury. Outside of the CFP 154 Medical Standards publication, there are few instructions on how to conduct this important command responsibility. In such a void, it is understandable why disproportionate importance may be placed on the clarity of MELs and sick leave.

CF H Svcs Gp and the units need a more formal and institutionalized reporting process to ensure B/W Comds and COs get sufficient and timely feedback and clarifications on medical assessments and MELs. Survey respondents indicated the need for more proactive engagement from the CF H Svcs Gp, and more direct and regular contact and discussions between medical officers and B/W Comds and COs. Respondents also emphasized the need for the right balance between patient privacy restrictions and feedback to and communication with the chain of command in order to reduce any unnecessary operational restrictions. Personnel shortages and increased reliance on short-term or part-time contracted health care providers may contribute to these MEL and communication issues as well.

ADM(RS) Recommendation

**Figure 2. CAF health care expenditures, dollar value and percentage of total departmental expenditures, FY 2010/11 to 2016/17.** This figure displays expenditures on Military Health Care (PAA sub-sub-program 4.1.8) in both dollar terms and as a percentage of total departmental spending from FY 2010/11 to 2016/17.

Figure 2 Details - CAF health care expenditures, dollar value and percentage of total departmental expenditures, FY 2010/2011 to 2016/2017

DND spent $721.5 million on health care in FY 2016/17, an increase of 4 percent over FY 2010/11 expenditures (see Figure 2).^{Footnote 55} Over the same period, the CAF reduced in size: total expenditures declined by 8 percent, and the number of Regular Force members shrank by 3 percent. As a result, Military Health Care expenditures came to represent a greater share of DND/CAF outlays.

However, the growth of health care in the CAF was small in comparison to cost increases outside the CAF. Annual growth in CAF health services expenditures over this period averaged 0.7 percent, much less than the 3.3 percent average annual growth in national health care expenditures over this time period.^{Footnote 56}

The entirety of the dollar increase in CAF health spending can be explained by payments made to civilian health service providers, through the Federal Health Claims Processing Service (FHCPS). These payments, totalling $144 million in 2016/17, grew by a third since 2010/11; all other health services expenditures combined decreased by 1 percent in that period.^{Footnote 57} This supports interviewee comments that CF H Svcs Gp has been compensating for internal staffing shortages by relying more heavily on external providers.

Key Finding 12: There is insufficient evidence to conclude on the cost efficiency of CAF health services, due to financial coding issues and lack of a satisfactory benchmark for comparison. CF H Svcs Gp has successfully implemented some cost-saving measures, but there may be some inefficiencies in payments to civilian health care providers.

It was not possible to assess the cost efficiency of CAF health care. CF H Svcs Gp has implemented some measures that will likely reduce costs in the long term. However, another review identified possible high-risk transactions to civilian health care providers.

Previous examinations of the efficiency of CAF health care had mixed results. In a 2007 report the Auditor General stated that the CAF health system spent almost double the Canadian average, per person.^{Footnote 58} In response, CF H Svcs Gp commissioned a study that found that when costing only comparable services, the CAF services were marginally less expensive than the care provided to Canadian citizens by provincial and territorial health care systems.^{Footnote 59}

For this evaluation, an attempt was made to compare costs for CAF health services to those of other comparable jurisdictions. Such a comparison was not possible, due to limitations in current financial accounting and differences across comparators.

Financial data for Military Health Care are not in a format that can allow a calculation of cost efficiency. Although CF H Svcs Gp prepares financial plans that break costs down by activity, current financial coding does not capture the activity with which a given expenditure is associated.^{Footnote 60} CF H Svcs Gp may wish to explore the possibility of incorporating activity areas into the chart of accounts.

Differences across potential comparators also hindered the cost-efficiency assessment. The following factors, for example, make comparisons to Canadian civilian health care problematic:

Spectrum of care: CAF health care covers almost all necessary treatments and procedures, including both prescription and non-prescription drugs and full dental care. This coverage therefore surpasses that available to many civilians through both provincial and private insurance.^{Footnote 61}
Additional activities: CAF health care involves a number of activities that are not present in civilian health care. Firstly, while both CAF and civilian care exist to meet the health needs of their patient populations, CAF health care must also contribute to operational effectiveness.^{Footnote 62} This second purpose requires a number of activities not present in civilian health care, including regular medical fitness assessments, communication and collaboration with unit commanding officers on health needs of members, and other programs and advisory activities. Secondly, CF H Svcs Gp is responsible for some activities, such as education and research that would not necessarily be recognized as health care system expenditures in the civilian health care context.
Geographic spread: Of the 14 health care jurisdictions in Canada, the CAF has the 11th smallest patient population but the largest geographic coverage, since CAF health services are provided across the country and abroad. Geographic spread affects the number of clinics needed per patient, travel and transportation expenditures, and other costs. It is a primary reason why costs per patient in the territories are the highest in Canada.^{Footnote 63}

Despite being unable to assess overall cost efficiency, this evaluation finds that CF H Svcs Gp has implemented measures that lead to longer-term cost savings. By promoting prevention, these health measures can reduce the costs of health care in the future due to illnesses and injuries avoided. Some examples of these programs are:

Strengthening the Forces: Strengthening the Forces is the CAF health promotion program designed to enable CAF members to increase control over and to improve their overall health and well-being.^{Footnote 64} Studies on workplace health promotion programs outside of the CAF have shown that these programs often bring benefits, including reduced health care costs and increased productivity that outweigh the program costs.^{Footnote 65}
Road to Mental Readiness (R2MR): R2MR aims to increase early awareness of indicators of distress and encourage early access to resources, while providing CAF personnel with the knowledge and skills required to manage the demands of military service including deployment. The CAF’s Director General Military Personnel Research and Analysis is currently carrying out a randomized control trial to test the efficacy of the R2MR program, but there is evidence that similar programs have been cost effective both in Canada and elsewhere.^{Footnote 66}

In contrast to these cost-saving measures, data analytics by DND uncovered some potential high-risk areas for health services. The analytics, carried out on billing data for external health care provider services paid through the FHCPS, found the following:^{Footnote 67}

High-risk transactions: Significant numbers of possible high-risk transactions were found, including what appear to be duplicate and miscoded transactions, fees in excess of maximum allowable amounts, and transactions for non-eligible beneficiaries.
Differences in costs of treatments: Amounts paid for services varied significantly across the country; and‎ in the province of Ontario rates were found to be on average twice the rate set by the provincial government for civilians receiving the same treatment.

These apparent overpayments are of particular concern because the FHCPS expenditures are assuming an increasing share of total health service expenditures (see Finding 11). The advisory report presenting the above findings provided recommendations to the CF H Svcs Gp to develop data analytics capabilities and to strengthen internal control measures.

Back to Table of Contents

Annex A—Management Action Plan

Key Finding 4: The Surgeon General and the Chief Dental Officer lack the proper authorities to deliver their required responsibilities as the heads of medical and dental professional branches in the CAF and DND. Efforts have been underway to rectify this for some time.

ADM(RS) Recommendation

Management Action

Commander MPC considers the establishment of ARA for the Surgeon General (SG) and Chief Dental Officer (CDO) as a high priority for resolution. It is recognized that pivotal to resolving these ARA and other related service delivery challenges (as outlined in multiple MAPs) lies in a fulsome review and amendment of the related QR&O (QR&O 34 and 35). As the new ARA are being developed for CFHS, that cost effectiveness and other domains of quality are appropriately captured for the key personnel, including Surgeon General, Chief Dental Officer, and CFHS Commanders.

The amendments of the QR&O have been made a top priority and will be pursued in a phased approach as follows:

Phase 1 – Data Collection and coordination – CFHS has already commenced this phase and will have relevant prioritized documentation ready for review.

Target Date: September 2018

Phase 2 – QR&O Analysis and Writing Plan established – By December 2018, a dedicated MPC analyst from the newly established MPC Policy Division will commence detailed review of the QR&O with a view to providing a prioritized writing plan.

Target Date: May 2019

Phase 3 – Modernization of the CFHS structure with associated ARA - This was directed by CDS and is ongoing. Aligning of the new ARA into the QR&O amendments will occur during all phases, as applicable. The modernized structure is estimated to be implemented over the next two FYs with specific timelines for priority positions being achieved starting in APS 19. Overall modernization implementation will be timed with final QR&O completion.

Target Date: Fall 2020

Notes: MPC has collected the key information and documentation on the ongoing efforts by CF H Svcs Gp in their attempts to prioritize changes to the QR&O and to establish clarity on the ARA of the key positions and, by extension, clarity on the obligations of the CF H Svcs Gp to provide Health Services to all entitled personnel. These key foundational documents being updated will enable ARA clarity in mandate for the SG and CDO, and set the conditions for a renewal of multiple subordinate policies needed to guide the CF H Svcs Gp in the execution of the overall established mandate. Equally important, the QR&O renewal and the clarification of any larger ARA issues will inform (and be informed by) the on-going Modernization efforts of the CF H Svcs Gp.

Supporting the internal process through all phases, will be the new Policy drafting organization being stood-up in the JAG structure. While the details remain unclear, it is assessed that this new section will function similar to NDRS which was removed from DND/CAF by DOJ leaving little drafting capacity resident in the department.

OPI: Commander, Military Personnel Command

Key Finding 6: Health-related policy documents and directives for the CAF are not comprehensive and up to date.

Key Finding 10: Communication between units and CF H Svcs Gp personnel, such as through MELs, could be improved.

ADM(RS) Recommendation

Management Action

CF H Svcs Gp is responsible for providing health care to eligible CAF members, maintaining a deployable health delivery capability and providing medical employment advice to the commander. The ADM(RS) Evaluation Report identified gaps in CF H Svcs Gp policy and deficiencies in communication Medical Employment Limitations (MELs) to the Chain of Command. Actions to address these specific areas for improvement are as follows:

CF Health Svcs Gp will hire a full-time physician (with experience in military occupational medicine) to update and maintain the policies and regulations regarding Medical Categories and MELs, specifically detailed in CFP 154. This policy vehicle is available online to all providers, members and CoC and is the foundation of the CAF MEL/Category system. Target date: December 2018.
CF Health Svcs Gp has a Policy Working Group with senior decision makers from all Directorates as its members. CFHS intends to increase the frequency with which the WG meets (from semi-annual to quarterly) in order to prioritize, collaborate, and synchronize the development of CFHS policies in a systematic manner that is responsive to the needs of the CAF and considers the clinical impact or significance. Quarterly meetings are intended to begin September 2018.
CF Health Svcs Gp will increase the size of its policy staff in order to meet the demands of policy management from within CFHS and from external stakeholders. Target date: July 2020.
CF Health Svcs Gp Directorate of Medical Policy (D Med Pol) will conduct bi-weekly teleconferences with Base Surgeons, and other key clinicians, to provide timely updates on changes to policies and MELs. This teleconference will also provide D Med Pol the opportunity to hear issues/concerns from the tactical level and conduct analysis that will influence healthcare policy development. Target date to begin teleconferences: September 2018.
CF Health Svcs Gp (D Med Pol) will conduct annual SAVs to all CAF Clinics to provide (refresher) training to all Clinicians on the appropriate assignment and communication of MELs and to the supported units and to provide strategic education to the locally supported units about Medical standards, MELs and the Category system and communication between supervisors and clinicians. Target date to begin SAVs: April 2019.
CF Health Svcs Gp (D Med Pol) will develop a communication plan regarding health policies to the CAF’s leadership and to all CAF members. Focus will be on highlighting changes in healthcare polices, creating greater general awareness of healthcare policies, and education on how to safely and effectively employ members with MELs, and to effectively communicate with the member and the supporting medical advisor. Target date: March 2019.

OPI: CF H Svcs Gp
Target Date: July 2020

Key Finding 7: A formalized performance measurement system is not fully operational.

ADM(RS) Recommendation

Management Action

The 2017 Surgeon General’s Integrated Health Strategy has identified seven Health Strategic Priorities; one of which is to conduct and leverage Performance Measurement (PM) to provide decision-making tools that are useful to all levels of the CF H Svcs Gp, the CoC and CAF personnel. We also recognize that achieving both efficiency and effectiveness in the delivery of health services requires a strong and integrated capability for PM, Quality Improvement, Utilization Review, Evaluation and Risk Management. The CF H Svcs Gp’s recent Governance Review demonstrated a lack of structure and capacity to perform and integrate these functions effectively, and also identified challenges in the availability of data and data analytics capability to support PM. These observations have helped shape our Modernization plan.

As a vanguard Modernization initiative, the Strategic Initiating Directive for the stand-up of a Directorate of Quality and Performance was recently approved, with the intent to integrate and enhance CF H Svcs Gp capacity to monitor and improve its performance as a Military Health System but also to lead the way in developing an organization-wide culture of quality, safety, and performance with a focus on clinical activities. The stand-up will occur in five phases, the first of which – identification of an interim Director to stand up an Initial Operating Capability (IOC) – is complete, and the second of which – staffing to IOC level – is underway.

CF H Svcs Gp is extremely limited in its current capability to dedicate human and other resources to performance measurement as recommended. Full implementation of the recommendation is tied to full implementation of the Directorate of Quality and Performance and is contingent on support for the additional positions required. Timelines for full implementation are anticipated to extend beyond two years.

There is a current version of the PIP for 3.3 Total Health Care. It is not reflective of any changes that will be required as a result of the C142 governance review that is not finalized. As an interim measure (relying on existing PM staff only), CF H Svcs Gp will work with CMPC and C Prog staff to comply with all of the requirements and schedules outlined by C Prog in relation to PIP development. That includes the work that has been directed by C Prog for the summer of 2018 to confirm segments and complete the financial alignment to those segments, and confirm indicator methodology, all in advance of April 1, 2019. These segments will be under the current version of the PIP.

Target date: Segment, financial alignment and methodology verification fall of 2018 (per C Prog direction under current version). Further revision will be required when C142 Governance review is completed and Performance Measurement Framework documentation is completed and evolution of with direction from L0 and L1 is provided. Estimated initial revision based on C142 requirements is September 2019.

OPI: Commander, Military Personnel Command
Target Date: September 2019

Key Finding 7: A formalized performance measurement system is not fully operational.

ADM(RS) Recommendation

Management Action

The 2017 Surgeon General’s Integrated Health Strategy has identified seven Health Strategic Priorities; one of which is to conduct and leverage Performance Measurement (PM) having the goal in sight to develop, validate and implement a Performance Measurement Framework (PMF) that is aligned with the Quadruple Aim and provides decision-making tools that are useful to all levels of the CF H Svcs Gp, the CoC and CAF personnel.

CF Health Svcs Gp, through the newly formed Directorate of Quality and Performance, will promulgate an Initiating Directive intending to integrate and enhance CF H Svcs Gp’s capacity to monitor and improve its performance as a Military Health System. Target date: July 2018.
CF Health Svcs Gp will develop a performance-based strategy focused on quality care, safety, and performance through evidenced-based best practices. Target date: December 2020.

OPI: CF H Svcs Gp
Target Date: July 2020

Key Finding 9: Wait times, likely caused by personnel shortages, are a concern to some in the CAF.

ADM(RS) Recommendation

Management Action

Whilst wait times are not indicative of the level of access to care for CAF members, or of the quality of care they receive, they can be indicative of efficiencies in the manner in which healthcare is delivered. Generally, CAF wait times are higher than those experienced elsewhere in Canada, and that shortages in healthcare personnel are considered a contributing factor to increased wait times. To address personnel shortfalls, CF H Svcs Gp will:

Identify information requirements (beyond what is available in existing reports, such as AMORs and SIPs) to develop the recommended report. Target date: October 2018.
Establish quarterly reviews of recruiting and retention strategies for MOSIDs considered in distress, that includes the challenges to recruitment and retention and elaborate on the actions underway to address the challenges. Reviews will be presented to CF H Svcs Gp’s Executive by the Directorate of Health Service Personnel with MOSID Advisor in attendance. Target date to commence: September 2018.
Establish working group with Canadian Armed Forces Recruiting Group (CFRG) in order to develop recruiting campaigns targeting (future) healthcare providers. Target date: July 2020.

OPI: CF H Svcs Gp
Target Date: July 2020

Back to Table of Contents

Annex B—Evaluation Methodology and Limitations

1.0 Methodology

1.1 Overview of Data Collection Methods

The evaluation findings are based on analysis triangulation or multiple lines of evidence. The data collection methods were selected based on the data required to address the performance indicators. The evaluation team used the following data collection methods to gather qualitative and quantitative data:

Document and literature review
Key informant interviews
Financial analysis
Survey

1.2 Details on Data Collection Methods

1.2.1 Document and Literature Review

The document and literature review identified key issues and trends relating to the relevance and performance of Military Health Care. Document review included program documentation specific to Military Health Care including DAOD, policies, plans, reports, directives and orders. In addition to program-specific documentation, other more general documentation relevant to Military Health Care was also reviewed. These included QR&O, Reports on Plans and Priorities/Departmental Plans, Departmental Performance Reports/Departmental Results Reports, the Speech from the Throne, the Canada Health Act, and the National Defence Act. The literature review included external documents consisting of health research reports, medical journals, and other military reports and websites.

1.2.2 Key Informant Interviews

The team conducted interviews with DND employees and CAF members who play a key role in delivering Military Health Care. Interviews provided a key source of qualitative information to support the evaluation of the relevance and performance of the program. Interviews were conducted with the following:

CF H Svcs Gp senior officers;
CF H Svcs Gp HQ staff;
CF H Svcs Gp HSG staff;
CF H Svcs Gp medical clinic staff;
CF H Svcs Gp dental clinic staff; and
CF H Svcs Gp Field Ambulance staff.

1.2.3 Financial Analysis

The team reviewed financial data for Military Health Care to determine the efficiency and economy of the program. Financial data from FY 2010/11 to 2016/17 was obtained from the DND/CAF financial system.

1.2.4 ADM(RS) Survey

A survey was developed in collaboration with CF H Svcs Gp to measure the satisfaction of B/W Comds and COs with the in-garrison care provided to their units and the advice they receive from health services personnel. A total of 315 responses were received, generating an estimated response rate of 41 percent. Responses were weighted by environment and function (i.e., B/W Comd vs CO) to support representativeness of the survey results.

Reserve Force units and deployed units were removed from the scope of the survey.

Reponses were analysed by base type,^{Footnote 68} rank, command and whether they identified as a B/W Comd or CO of a unit.

2.0 Limitations

Table B-1. Evaluation Limitations and Mitigation Strategies. List of the limitations of the evaluation and the corresponding mitigation strategy.
Limitation	Mitigation Strategy
The possibility that the interviewees would provide biased information and only positive stories about their Program.	A comparison was made between interviewees and other people from the same organization or group, and information from other sources, such as documents and files.
Lack of performance measurement information	The evaluation consulted multiple sources of quantitative and qualitative information to determine the performance of the program. However, performance measurement data was limited for this program. Data obtained was supplemented by primary data collection carried out by DGE.
Older data collection	Because of delays in the evaluation, some of the data collection, such as interviews, were carried out two years prior to report publication. Every attempt was made to ensure all evidence presented in this report is current.

Table B-1 Details - Evaluation Limitations and Mitigation Strategies

Figure C-1 Details - Logic Model for Evaluation of Military Health Care

Figure C-1 Summary: This logic model shows the relationship between the program’s main activities, outputs and expected outcomes.

Inputs

Canada Health Act (1984)
Provincial Dental Regulatory Acts
National Defence Act
CFDS, DPR, RPP, SIP, AMOR process
Medical and dental supplies
Logistics
Infrastructure

Activities

Advice Research
Governance and Planning
Provision of Care
Research and Analysis
Training and Education
Operation

Outputs

Briefings to senior leaders, external and internal stakeholders
HS Decisions and orders provided to L1s/CAF
DAODs
CANFORGENS
Policy, directions and organizational orders
Clinical quality and patient safety oversight
Accreditation process
Medical regulatory function
Credentialing of HCPs
Standards of practice/regulatory oversight and clinical audit
Clinical Practice Guidelines
Medical, dental assessments
Wait time reports, patient satisfaction reports
Contracts, agreements
Prevention and promotion advice and materials
Medical and dental supplies and logistics
Oversight of all clinical research and surveillance
Knowledge translation
Environmental health research
Epidemiology analysis
Performance Measurement
Training courses/programs, MCRP
Advice on SAR TECH training
Student ratios
Joint exercises, operations
After action reports, lessons learned
Supplies and logistics
Intelligence reports
IT/IM capabilities
Tracking of medical and dental supplies

Immediate Outcomes

CF HS is the recognized authority on all CAF/DND health care issues Equitable, appropriate, safe, timely, patient-centered, accessible and continuous HS to entitled persons
A combat capable, multi-purpose HS force for operations

Intermediate Outcomes

Increase in health of population
Enhanced patient care
Resource stewardship
Operational readiness and excellence

Ultimate Outcome

Full spectrum, high quality health services are provided to entitled persons in Canada and abroad

Table D-1. Evaluation Matrix. This table lists the Evaluation Question and Indicators, and links them to the finding within the report.
Evaluation Issues/Questions	Indicators	Finding Number
Relevance
1.1 To what extent is there a continuing need for CAF Health Services?	1.1.1 Existing CAF operational requirements and DAOD that require medical, dental and health services	1
1.2 To what extent is there a federal role and responsibility for the delivery of CAF Health Services?	1.2.1 Existing Acts and legislations which outline the Federal Role and Responsibilities	2
1.3 To what extent are CAF Health Services aligned with governmental and departmental priorities?	1.3.1 Evidence of alignment with priorities from the Minister of Defence’s statements, mandate letter, and SSE	3
	1.3.2 Evidence of alignment with priorities from DND Departmental Plan and Departmental Results Report	3
Effectiveness
2.1 Does CFHS have the appropriate authorities and governance to achieve their mandate?	2.1.1 Extent that the SG and CDO have the required regulatory authorities within the CAF and DND to conduct their core roles	4
	2.1.2 Extent that the governance structure allows CFHS to achieve its objectives	5
	2.1.3 The health policies and directives have been developed and are up to date and communicated	6
	2.1.4 Extent that CF H Svcs Gp has an appropriate performance measurement system in place	7
2.2 To what extent is in-garrison care meeting the needs of the CAF?	2.2.1 CAF members are satisfied with the in-garrison care they receive	8, 9
	2.2.2 Commanders are satisfied with the in-garrison care provided to their units	8, 9
	2.2.3 Commanders are satisfied with the advice they receive from health services personnel	8, 10
	2.2.4 CF H Svcs Gp monitors the in-garrison care provided to the CAF	5, 7
Efficiency
3.1 To what extent is the CF H Svcs Gp providing military health care in a cost-efficient manner?	3.1.1 Best practices which can impact efficiency are implemented	12
	3.1.2 Extent CF H Svcs Gp has the required tools to achieve its objectives	5, 7
	3.1.3 Extent that CF Health Services is cost efficient	11, 12

Table D-1 Details - Evaluation Matrix

Table D-1 Summary: This table presents the evaluation questions and indicators for three sections of the evaluation: the relevance section, effectiveness section, and the efficiency section. The study used three evaluation questions to determine the Program’s relevance; these questions appear listed in the left-hand column, each under its own sub-heading. Select a question and read across that row to learn, first, the indicators used to assess the corresponding relevance question. Read across to the third column to learn the corresponding finding number.

The study used two evaluation questions to determine the Program’s effectiveness. These questions appear listed in the left-hand column, each under its own sub-heading. Select a question and read across that row to learn, first, the indicators used to assess the corresponding effectiveness question. Read across to the third column to learn the corresponding finding number.

The study used one evaluation question to determine the Program’s efficiency. The question appears in the left-hand column. Read across that row to learn, first, the indicators used to assess the corresponding effectiveness question. Read across to the third column to learn the corresponding finding number.

Back to Table of Contents

Footnotes

Footnote 1

As of April 2018, the PAA has been replaced by the DND/CAF Program Inventory. Within this new framework, the activities of Military Health Care fall under the Total Health Care program, which also includes civilian health. This report uses the term Military Health Care to clarify that civilian health programming is excluded from the evaluation scope. Prior to FY 2014/15, CAF health care and advice fell under PAA element 2.4.2.6, also called Military Health Care. To allow for accurate year-to-year comparisons, historical expenditure data covers PAA 4.1.8 activities exclusively, regardless of the framework used in a given year.

Return to footnote1 referrer

Footnote 2

“In-garrison health care” refers to all health care services provided to non-deployed CAF members, whether that care is provided on or off base.

Return to footnote2 referrer

Footnote 3

Spectrum of Care: Medical and Dental Benefits and Services. http://www.forces.gc.ca/en/caf-community-health-services-benefits-drug-coverage/index.page.

Return to footnote3 referrer

Footnote 4

2014 Surgeon General’s Report.

Return to footnote4 referrer

Footnote 5

DPR, 2015/16.

Return to footnote5 referrer

Footnote 6

2014 Surgeon General’s Report.

Return to footnote6 referrer

Footnote 7

DND/CAF Departmental Performance Reports, FY 2010/11 to FY 2015/16, and Departmental Results Report, FY 2016/17.

Return to footnote7 referrer

Footnote 8

The scope of the present evaluation is described below, in section 1.3. Military Health Care expenditures outside of CF H Svcs Gp include, for example, military personnel pay and benefits, some civilian pay and benefits, and expenditures by other CAF environments. Some CF H Svcs Gp expenditures are not included as Military Health Care expenditures, including some expenditures related to real property, infrastructure maintenance, and other areas.

Return to footnote8 referrer

Footnote 9

The DND/CAF Departmental Results Framework (DRF) and Program Inventory took effect April 2018. This evaluation refers to Military Health Care to clarify that civilian health and wellness is excluded from its scope.

Return to footnote9 referrer

Footnote 10

Evaluation of Medical Support to Deployed Operations, DND, 2014. http://www.forces.gc.ca/en/about-reports-pubs-audit-eval/235p0999.page.

Return to footnote10 referrer

Footnote 11

Medical Standards (CFP 154), Medical Fitness Chapter 2.

Return to footnote11 referrer

Footnote 12

Ibid.

Return to footnote12 referrer

Footnote 13

Canada Health Act, R.S.C., 1985, c. C-6, 2 – Definitions. http://laws-lois.justice.gc.ca/eng/acts/C-6/FullText.html.

Return to footnote13 referrer

Footnote 14

Canadian Life and Health Insurance Facts, 2016 Edition, Canadian Life and Health Insurance Association (CLHIA), 2016. https://www.clhia.ca/web/CLHIA_LP4W_LND_Webstation.nsf/resources/Factbook_2/$file/2016+CLHIA+Factbook+ENG.pdf.

Return to footnote14 referrer

Footnote 15

Canada’s defence policy: Strong, Secure, Engaged, 2017.

Return to footnote15 referrer

Footnote 16

1867 Constitution Act, s. 91(7). http://laws-lois.justice.gc.ca/eng/const/FullText.html.

Return to footnote16 referrer

Footnote 17

Queen’s Regulations and Orders, volume 1, chapters 34 & 35. http://www.forces.gc.ca/en/about-policies-standards-queens-regulations-orders-vol-01/index.page.

Return to footnote17 referrer

Footnote 18

Canada’s defence policy: Strong, Secure, Engaged, 2017.

Return to footnote18 referrer

Footnote 19

Ibid.

Return to footnote19 referrer

Footnote 20

2018/19 Departmental Plan, DND, 2018. http://forces.gc.ca/en/about-reports-pubs-report-plan-priorities/2018-index.page.

Return to footnote20 referrer

Footnote 21

QR&O chapters 34 and 35.

Return to footnote21 referrer

Footnote 22

Development of Canadian Forces Health Services Group Policy and Direction Documents: CF H Svcs Gp Instruction 2600-02, DND, 2015.

Return to footnote22 referrer

Footnote 23

RX2000: The Canadian Forces Medical Clinic, DND.

Return to footnote23 referrer

Footnote 24

QR&O 34: Rationale for Changes, DND (internal document), 2015.

Return to footnote24 referrer

Footnote 25

Terms of Reference. Surgeon General’s Clinical Council.

Return to footnote25 referrer

Footnote 26

Terms of Reference. Quality & Patient Safety Advisory Committee.

Return to footnote26 referrer

Footnote 27

Quality and Patient Safety Office. http://cmp-cpm.mil.ca/en/health/personnel-providers/quality-patient-safety.page. Last accessed: April 23, 2018.

Return to footnote27 referrer

Footnote 28

Accreditation Canada, Accreditation Report July 25, 2017.

Return to footnote28 referrer

Footnote 29

Accreditation Report - Canadian Forces Health Services Group. Accreditation Canada. November 26, 2013.

Return to footnote29 referrer

Footnote 30

Accreditation Canada, Accreditation Report July 25, 2017.

Return to footnote30 referrer

Footnote 31

Policy Framework for Military Personnel Management, CAF, DAOD 1000-7, 2017; Auditing Canadian Forces Health Services Group Policy and Direction Documents, CF H Svcs Gp, Instruction 2600-09, 2013; and DND/CAF intranet.

Return to footnote31 referrer

Footnote 32

DAOD under MPC authority with CF H Svcs Gp as point of contact, and CF Mil Pers Instr with CF H Svcs Gp listed as OPI. Source: DND/CAF intranet and websites.

Return to footnote32 referrer

Footnote 33

Based on a random sample of 160 out of 470 policies currently in force; 95 percent confidence and ±6 percent confidence interval. Source: DND/CAF intranet.

Return to footnote33 referrer

Footnote 34

Report of the Canadian Forces Expert Panel on Suicide Prevention, DND, 2010; and Report of the 2016 Mental Health Expert Panel on Suicide Prevention in the CAF, DND, 2017.

Return to footnote34 referrer

Footnote 35

Joint Suicide Prevention Strategy Annex: CAF and VAC Suicide Prevention Strategy Actions, CAF and VAC, 2017. (https://www.canada.ca/content/dam/dnd-mdn/documents/reports/2017/caf-vac-joint-suicide-prevention-strategy-annex.pdf).

Return to footnote35 referrer

Footnote 36

RX2000: The Canadian Forces Medical Clinic, DND.

Return to footnote36 referrer

Footnote 37

ADM(HR-MIL) Instruction 11/04: Medical Fitness for Duty, DND, 2004.

Return to footnote37 referrer

Footnote 38

Canadian Armed Forces Medical Standards (CFP 154), DND, chapter 4.

Return to footnote38 referrer

Footnote 39

2014 ADM(RS) Audit of CF H Svcs Gp. http://www.crs-csex.forces.gc.ca/reports-rapports/pdf/2014/p0988-eng.pdf. Date accessed: May 10, 2017.

Return to footnote39 referrer

Footnote 40

Digital Health Environment: Project Overview. Prepared by CF H Svcs Gp. September 9, 2016.

Return to footnote40 referrer

Footnote 41

HLIS: 82.2 percent of Regular Force personnel felt that the quality of CFHS in general was good, very good, or excellent. PEQ: 91.1 percent of patients reported being mostly or completely satisfied with the primary care, mental health, or dental services received in the previous three months.

Return to footnote41 referrer

Footnote 42

HLIS indicates that half of Regular Force personnel are satisfied with the wait time for a routine, ongoing, non-urgent health problem, and another 27 percent are neutral, while less than half (43 percent) are satisfied with the wait time for non-urgent dental appointments, and another 27 percent are neutral. HLIS, 2014.

Return to footnote42 referrer

Footnote 43

The PEQ shows 84 percent of CAF members report being able to book an appointment in a reasonable time.

Return to footnote43 referrer

Footnote 44

Survey of B/W Comds and COs: 76 percent agree that medical wait times are acceptable, with no significant differences between COs and B/W Comds. For dental and mental health, B/W Comds are significantly less likely to agree that wait times are acceptable (Chi Square test, α = 0.05): 54 percent agree that dental wait times are acceptable, and 46 percent agree that mental health wait times are acceptable.

Return to footnote44 referrer

Footnote 45

Measures of wait times and/or accessibility are currently tracked for mental health and physiotherapy services.

Return to footnote45 referrer

Footnote 46

HLIS.

Return to footnote46 referrer

Footnote 47

AMA Directive 100-01: Medical Standards for CF Aircrew, DND, 2014.

Return to footnote47 referrer

Footnote 48

Delayed Access to Health Care and Mortality, Prentice, JC, & Pizer, SD, Health Services Research, 2007; and The Effect of Wait Times on Mortality in Canada, Barua, Nadeem Esmail, & Jackson, Taylor, Fraser Institute, 2014.

Return to footnote48 referrer

Footnote 49

Canadian Forces Cancer and Mortality Study: Causes of Death. Statistics Canada, 2011.

Return to footnote49 referrer

Footnote 50

Accreditation Canada, Accreditation Report July 25, 2017.

Return to footnote50 referrer

Footnote 51

Evaluation of Medical Support to Deployed Operations, DND, 2014; Audit of Canadian Forces Health Services, DND, 2014; and Accreditation Report: Canadian Forces Health Services Group, Accreditation Canada, 2017.

Return to footnote51 referrer

Footnote 52

DND Departmental Performance Reports, 2013/14 to 2015/16.

Return to footnote52 referrer

Footnote 53

Canadian Forces Health Services Attraction and Retention. Presentation to CMP. March 22, 2016.

Return to footnote53 referrer

Footnote 54

Canada’s defence policy: Strong, Secure, Engaged, 2017.

Return to footnote54 referrer

Footnote 55

DND financial records for PAA sub-sub-program 4.1.8 – Military Health Care.

Return to footnote55 referrer

Footnote 56

National Health Expenditure Trends, 1975 to 2017: Data Tables, Series A, Canadian Institute for Health Information, 2017. Data from this source are based on calendar years, rather than fiscal years for DND expenditures. Expenditures for 2016 and 2017 are forecasts.

Return to footnote56 referrer

Footnote 57

Advisory Report on Information Management in the Federal Health Claims Processing Service Contract, DND, forthcoming.

Return to footnote57 referrer

Footnote 58

Chapter 4 Military Health Care – National Defence. Report of the Auditor General of Canada to the House of Commons. 2007.

Return to footnote58 referrer

Footnote 59

Comparative Cost Analysis of Health Care Services for the Canadian Forces. Government Consulting Services October 15, 2009.

Return to footnote59 referrer

Footnote 60

Activity categories in CF H Svcs Gp financial planning include direct support to operations, medical care, dental care, mental health, training and education, among others.

Return to footnote60 referrer

Footnote 61

Medical Coverage, Eligibility, Management and Access to Health Care, DND, 2017. http://www.forces.gc.ca/en/caf-community-health-services-benefits-drug-coverage/coverage-eligibility-management-access.page.

Return to footnote61 referrer

Footnote 62

Canadian Armed Forces Medical Standards (CFP 154), DND, chapter 4.

Return to footnote62 referrer

Footnote 63

Healthcare in Canada’s North: Are We Getting Value for Money? Young, TK, Chatwood, S, & Marchildon, GP. Healthcare Policy, 12(1):59-70, 2016; and National Health Expenditure Trends, 1975 to 2017. Canadian Institute for Health Information, 2017, (https://www.cihi.ca/sites/default/files/document/nhex2017-trends-report-en.pdf).

Return to footnote63 referrer

Footnote 64

Strengthening the Forces: The CAF's Health Promotion Program http://cmp-cpm.mil.ca/en/health/caf-members/health-promotion.page.

Return to footnote64 referrer

Footnote 65

Canadian Centre for Occupational health and Safety https://www.ccohs.ca/topics/wellness/promotion/; and Making the Economic Case for Investing in Public Health and the SDH http://www.cpha.ca/en/programs/social-determinants/frontlinehealth/economics.aspx.

Return to footnote65 referrer

Footnote 66

Making the Case for Investing in Mental Health. Mental Health Commission of Canada.

Return to footnote66 referrer

Footnote 67

Advisory Report on Information Management in the Federal Health Claims Processing Service Contract, DND, forthcoming.

Return to footnote67 referrer

Footnote 68

Bases were classified into rural, remote, semi-urban and urban as per APS 110.

Return to footnote68 referrer

Page details

Date modified:: 2019-05-17

Language selection

Search

Evaluation of Military Health Care

Table of Contents

Alternate Formats

Assistant Deputy Minister (Review Services)

Overall Assessment

ADM(RS) Recommendation

ADM(RS) Recommendation

ADM(RS) Recommendation

ADM(RS) Recommendation

2.2.2 Extent to which In-Garrison Care and Advice Meet the Needs of the CAF

ADM(RS) Recommendation

ADM(RS) Recommendation

ADM(RS) Recommendation

ADM(RS) Recommendation

ADM(RS) Recommendation

ADM(RS) Recommendation

ADM(RS) Recommendation

Page details