National Advisory Committee on Immunization (NACI): Guidelines for the economic evaluation of vaccination programs in Canada

Download in PDF format
(1.23 MB, 125 pages)

Download in Word format
(1.41 MB, 125 pages)

Organization: Public Health Agency of Canada

Published: 2022-04-14

Draft 1st Edition
For Public Consultation 2022

Acknowledgements

NACI economic guidelines task group members and liaisons

Task group members were responsible for identifying and discussing key issues related to each topic, drafting topic sections, reviewing all draft topic sections, reviewing the draft consolidated report, addressing peer review and stakeholder feedback, and reviewing and approving the final version of Guidelines.

PHAC members:

Man Wah Yeung, MSc
Senior health economist
PHAC
Toronto, Ontario
Austin Nam, PhD
Senior health economist
PHAC
Toronto, Ontario
Ashleigh Tuite, PhD
Manager
PHAC
Toronto, Ontario
Althea House, BScN
Manager
PHAC
Ottawa, Ontario
Matthew Tunis, PhD
Executive secretary
PHAC
Ottawa, Ontario

Contractor:

Nina Lathia, PhD
Health economist
No affiliation
Toronto, Ontario

Academic members:

Beate Sander, PhD, RN, MBA, MEcDev
Co-chair of Task Group and NACI member
Director, Toronto Health Economics and Technology Assessment (THETA) Collaborative
University of Toronto
Toronto, Ontario
Murray Krahn, MD, MSc, FRCPC
Co-chair of Task Group
Director, Toronto Health Economics and Technology Assessment (THETA) Collaborative
University of Toronto
Toronto, Ontario
Stirling Bryan, PhD, MSc
Professor
University of British Columbia
Vancouver, British Columbia
Werner Brouwer, PhD, MSc
Professor
Erasmus University Rotterdam
Rotterdam, Netherlands
Mark Jit, PhD, MPH
Professor
London School of Hygiene and Tropical Medicine
London, United Kingdom
Karen M. Lee, MA
CADTH Liaison
Director, Health Economics
Canadian Agency for Drugs & Technologies in Health (CADTH)
Ottawa, Ontario
Monika Naus, MD, MHSc, FRCPC, FACPM
Provincial/ Territorial Liaison and NACI Liaison
Medical Director, Communicable Diseases & Immunization Service
British Columbia Centre for Disease Control
Vancouver, British Columbia
Sachiko Ozawa, PhD, MHS
Associate Professor
University of North Carolina at Chapel Hill
Chapel Hill, North Carolina
Lisa Prosser, PhD
Professor
University of Michigan
Ann Arbor, Michigan

Reviewers

PHAC would like to acknowledge the academic peer-reviewers and NACI Economics Task Group (sister task group) for their peer-review of the full guidelines

Academic peer-reviewers:

Bohdan Nosyk, PhD
Simon Fraser University
Burnaby, British Columbia
Christopher McCabe, PhD
Ellen Rafferty, PhD
Jeff Round, PhD
Sasha van Katwyk, MSc
Kate Harback, PhD
Erin Kirwin, MA
on behalf of
Institute of Health Economics
Edmonton, Alberta
David Fisman, PhD
University of Toronto
Toronto, Ontario
Wendy Ungar, PhD
University of Toronto
Toronto, Ontario
Ava John-Baptiste, PhD
Western University
London, Ontario
Lauren Cipriano, PhD
Western University
London, Ontario
Kim Dalziel, PhD
University of Melbourne
Melbourne, Australia
Susan Griffin, PhD
University of York
York, England

Members of NACI economics task group:

Beate Sander, PhD, MBA, MEcDev
University of Toronto
Toronto, Ontario
Ellen Rafferty, PhD.
Institute of Health Economics
Edmonton, Alberta
David Fisman, PhD.
University of Toronto
Toronto, Ontario
Bernice Tsoi, PhD
Canadian Agency for Drugs & Technologies in Health (CADTH)
Ottawa, Ontario
Philippe De Wals, MD, PhD
Université Laval
Québec City, Québec
Joanne Langley, MD, FRCPC
Dalhousie University
Halifax, Nova Scotia
Monika Naus, MD, MHSc, FRCPC, FACPM
British Columbia Centre for Disease Control
Vancouver, British Columbia
Kristin Klein, MD, FRCPC
Alberta Health Services
Edmonton, Alberta

Contributors

PHAC would like to acknowledge the following individuals for their contributions:

Academic contributors:

Lisa Schwartz, PhD, MA – contributed to the development of the equity chapter
Richard Cookson, DPhil, MPhil – contributed to the development of the equity chapter
Pieter van Baal, PhD – contributed to the development of the resource use and costs chapter

Indigenous Services Canada contributors:

Kendra Hardy, MA – provided review of the foreword, equity chapter, and types of evaluation chapter
Melanie Knight, BScN, RN – provided review of the equity chapter and types of evaluation chapter
Andrea Monahan, BScN, RN, MPH – provided review of the foreword
Denise Hamilton, MPA – provided review of the foreword
Tom Wong, MD – provided review of the foreword, equity chapter, and types of evaluation chapter
Kim Daly, BN, RN, MN – provided review of the foreword
Pamela Wolfe-Roberge, BA – provided review of the equity chapter, and types of evaluation chapter

Public Health Ethics Consultative Group contributors:

Diego Silva, PhD – provided review of the equity chapter
Béatrice Godard, PhD – provided review of the equity chapter
Boluwaji Ogunyemi, MD – provided review of the equity chapter
Cassandra Opikokew Wajuntah, PhD (c) – provided review of the equity chapter
Maxwell J. Smith, PhD – provided review of the equity chapter
A.M. Viens, PhD – provided review of the equity chapter
Alice Virani, PhD – provided review of the equity chapter

NACI contributors:

Sheela Ramanathan, PhD – provided review of the effectiveness chapter
Kyla Hildebrand, MD, MScCH – provided review of the effectiveness chapter
Matthew Miller, PhD – provided review of the effectiveness chapter

PHAC contributors:

Shainoor Ismail, MD, MSc– provided review of the equity chapter
Angela Sinilaite, MPH – provided review of the equity chapter
Alexandra Cernat, MSc – provided support on referencing
Amanda Sumner, MA – provided technical support early in the project
Christine Mauviel, BA – provided project management support
Chantale Tremblay, BSc – provided project management support
Caroline Rodriguez-Charette, BJ – provided project management support
Siobhan Kelly, BA – provided project management support
Jennifer Daniel – provided project management support
PHAC would like to acknowledge the NACI meeting discussions with and/or review by NACI members, liaisons, and ex-officios.

NACI members:

Shelley Deeks, MD, Nova Scotia Health and Wellness
Robyn Harrison, MD, MSc, FRCPC, Alberta Health Services
Melissa Andrew, MD, MSc, Dalhousie University
Julie Bettinger, PhD, MPH, British Columbia Children’s Hospital Research Institute
Nicholas Brousseau, MD, MSc, Institut national de santé publique du Québec
Philippe De Wals, MD, PhD, Université Laval
Hélène Decaluwe, MD., PhD., University of Montréal
Eve Dubé, PhD, Université Laval
Vinita Dubey, MD, MPH, University of Toronto
Kyla Hildebrand, MD, MScCH, British Columbia Children’s Hospital
Kristin Klein, MD, Alberta Health Services
Anne Pham-Huy, MD, Children's Hospital of Eastern Ontario
Susan Smith, RN, Government of Alberta
Sarah Wilson, MD, MSc, Public Health Ontario

NACI liaisons:

Lucie Marisa Bucci, Canadian Public Health Association
Eliana Castillo, MD, Society of Obstetricians and Gynaecologist of Canada
Amanda Cohn, MD, Centers for Disease Control and Prevention
Lorette Dupuis, RN, Canadian Nurses Association
Jia Hu, MD, College of Family Physicians of Canada
Deshayne Fell, PhD, MSc, University of Ottawa
Martin Lavoie, MD, Vancouver Coastal Health
Dorothy Moore, MD, McGill University
Amanda Ung, Canadian Pharmacists Association
Lea Bill, RN, BScN, Canadian Indigenous Nurses Association
Marilee Nowgesic, Canadian Indigenous Nurses Association
Sarah Funnell, MD, Indigenous Physicians Association of Canada

NACI ex-officios

Erin E. Henry, BScN, Public Health Agency of Canada
Guillaume Poliquin, MD, PhD, Public Health Agency of Canada
Diane MacDonald, MPH, Public Health Agency of Canada
Susanna Ogunnaike-Cooke, MSc, Public Health Agency of Canada
Mireille Lacroix, LLM, Public Health Ethics Consultative Group
Kelly Robinson, MSc, Health Canada
Celia Lourenco, PhD, Health Canada
Vincent Beswick-Escanlar, MD, MPH, Canadian Armed Forces
Tom Wong, MD, MPH, Indigenous Services Canada

Conflicts of interest declaration

As part of standard procedures for identifying and addressing affiliations and interests, Economic Guidelines Task Group members completed individual disclosure forms which were assessed by PHAC to ensure no undue influence or perceived conflict of interest.

Beate Sander had no declarations of interests and affiliations to make.

Murray Krahn had no declarations of interests and affiliations to make.

Stirling Bryan had no declarations of interests and affiliations to make.

Werner Brouwer had no declarations of interests and affiliations to make.

Mark Jit declared receiving research funding from non-profit organizations (European Commission and the WHO).

Karen M. Lee had no declarations of interests and affiliations to make.

Monika Naus had no declarations of interests and affiliations to make.

Sachiko Ozawa declared research funding from Merck Sharp & Dohme Corporation, the Bill & Melinda Gates Foundation, the North Carolina Department of Health and Human Services and the National Cancer Institute.

Lisa Prosser had no declarations of interests and affiliations to make.

Abbreviations

AQoL: Assessment of Quality of Life
CADTH: Canadian Agency for Drugs and Technologies in Health
CBA: Cost-benefit analysis
CEAC: Cost-effectiveness acceptability curve
CEAF: Cost-effectiveness acceptability frontier
CHU9D: Child Health Utility 9-Dimensions
CoP: Correlate of protection
COVID-19: Coronavirus infection disease 2019
CUA: Cost-utility analysis
DSA: Deterministic Sensitivity Analysis
EEFA: Ethics, equity, feasibility, and acceptability
EQ-5D: EuroQol 5-Dimensions questionnaire
EQ-5D-Y: EuroQol 5-Dimensions questionnaire youth
Hib: Haemophilus influenzae type b
HIV: Human Immunodeficiency Virus
HPV: Human papilloma virus
HRQoL: Health-related quality of life
HUI: Health Utilities Index
ICER: Incremental cost-effectiveness ratio
NACI: National Advisory Committee on Immunization
PedsQL: Pediatric Quality of Life Inventory
PCV: pneumococcal conjugate vaccine
PSA: probabilistic sensitivity analysis
QALY: Quality-adjusted life year
RCT: Randomized controlled trial
SF-6D: Short Form 6-Dimensions
TB: Tuberculosis

Glossary

Age-shifting of infection: A phenomenon that can occur when a particular age cohort of the population is vaccinated, which reduces the force of infection in that cohort and shifts the average age at infection.

Agent-based model: A type of dynamic microsimulation model that allows individuals to act autonomously based on defined behavioural rules. See definition for individual-based model.

Basic reproduction number: The average number of secondary cases infected by an infectious person in a completely susceptible population.

Catch-up strategy: A strategy of vaccinating individuals who did not receive a particular vaccination at the recommended age. This strategy may be used in individuals who have not been previously eligible for vaccination, who have missed a scheduled vaccine dose, or who have not completed a vaccine series.

Coverage: The estimated percentage of eligible individuals who have received a particular vaccine.

Canadian Agency for Drugs and Technologies in Health (CADTH): Independent, not-for-profit organization tasked with providing Canada's health care decision-makers with evidence related to the optimal use of drugs and medical devices within the public health care system.

Closed (population) model: A model that follows a fixed cohort or cohorts of individuals. Individuals are not able to enter or exit the model through births, deaths or immigration over time.

Community Immunity: A state where a sufficient proportion of a population is immune to an infectious disease, either from vaccination or prior infection, thereby preventing outbreaks from occurring and making spread between individuals less likely. The term is commonly used to refer to the indirect protection unvaccinated individuals receive due to the presence of immune individuals in a population. This term is also referred to as herd immunity.

Confounding bias: A distortion in the estimate of the relationship between an exposure and an outcome in a study, resulting from a third variable, the confounder, which is related both to the exposure and the outcome.

Consumption: The value of goods and service bought by individuals.

Continuous (time) model: A model in which events can occur at any point in time.

Correlate of protection (CoP): An immune biomarker that predicts vaccine efficacy in vaccinated individuals and can be used as a surrogate endpoint in studies of vaccine efficacy or effectiveness.

Cost-effectiveness acceptability curve (CEAC): A graphic summary of the uncertainty in results of an economic evaluation, where a range of cost-effectiveness thresholds is plotted against the probability thatan intervention is cost-effective.

Cost-effectiveness acceptability frontier (CEAF): A graphical summary of the uncertainty in results of an economic evaluation, which indicates the strategy that is economically preferred at different threshold values for cost-effectiveness and the probability of that strategy being cost-effective. As the threshold increases the economically preferred treatment may change, the switch point being where the threshold value increases beyond the relevant ICER reported for the intervention of interest. CEAFs are most useful when three or more alternatives are being compared, in which case there may be two or more switch points at different threshold values.

Cost-benefit analysis (CBA): An economic evaluation in which both costs and outcomes are expressed in monetary terms.

Cost-effectiveness analysis (CEA): An economic evaluation in which health outcomes are expressed in natural units (e.g., infections avoided).

Cost-utility analysis (CUA): An economic evaluation in which health outcomes are expressed in quality-adjusted life years (or other generic measure of health-related utility). It is sometimes referred to as a cost-effectiveness analysis (CEA), or CEA with QALYs. This is the form of economic evaluation favoured by public health care decision-makers in Canada.

Decision problem: An explicit statement of the interventions, study populations, outcome measures, and perspective adopted in an economic evaluation, related specifically to the decision(s) that the evaluation is designed to inform.

Deterministic model: A model that describes what happens on average and in which events cannot occur randomly (by chance). For a defined set of parameters and starting conditions these models will always generate the same results each time they are run.

Deterministic sensitivity analysis (DSA): A method used to explore uncertainty in results of a model-based economic evaluation, where one or more parameters are changed across a pre-specified range while holding the remaining parameters fixed to determine the extent to which the parameter values impact the results of the analysis.

Discount rate: Costs and health outcomes occurring in the future are generally considered to be valued less than those occurring presently, and so they are discounted in an economic evaluation to ascertain their present day value. The factor by which costs and health outcomes are discounted is expressed as the discount rate.

Discrete (time) model: A model in which events can only occur at pre-specified points in time.

Disease control: The state in which incidence, prevalence, morbidity, or mortality of a particular disease has been reduced locally but continued efforts are required to maintain this reduction.

Disease elimination: Local incidence of a particular infection has been reduced to a level below that necessary to sustain ongoing transmission in a given geographic area, but continued efforts are required to maintain this reduction.

Disease eradication: The incidence of an infection has been permanently reduced to zero worldwide, the causative organism is no longer present in the wild and efforts are no longer required to maintain this reduction.

Distributional cost-effectiveness analysis: An extension to the conventional CEA framework that quantifies the distributional impacts of health interventions based on different equity criteria such as socioeconomic status or disease severity.

Dominance: Refers to a scenario in which a strategy results in greater benefits and fewer costs compared to its alternative.

Dose completion: The accumulation of the required number of doses of a vaccination regimen during a specified time period.

Dynamic (transmission) model: A model in which the force of infection can vary over time. Incidence is a function of the number (or proportion) of infected and susceptible individuals and the transmissibility of the virus. May also be referred to as a model with an endogenous force of infection.

Effectiveness: The extent to which an intervention provides the desired outcome(s) in the relevant study population in a real-world setting.

Effective reproduction number: The average number of secondary cases infected by an infectious person in a population where some of the individuals are immune due to vaccination or infection.

Efficacy: The benefit of an intervention produced in an experimental and controlled setting, such as in a randomized controlled trial (RCT).

Efficiency frontier: A graphical summary of cost-effectiveness results comparing multiple interventions. The plot compares the effect on the y-axis and the costs on the x-axis. The frontier links the interventions that are not dominated. An intervention located on or below the frontier can be considered reasonably efficient.

Epidemiologic equilibrium: A situation where the rate of new infections circulating in a population is equal to the rate of recovery from the infection, resulting in a stable or unchanging state.

Expected value of perfect information (EVPI): The maximum price that a decision-maker would be willing to pay to have perfect information regarding all parameter values that influence which intervention is preferred based on results of a CEA. This represents the value (in monetary terms) of removing all uncertainty about the parameters in the analysis. EVPI can also be expressed for the total population who stand to benefit over the expected lifetime of the intervention (known as population EVPI).

Expected value of partial perfect information (EVPPI): The maximum price that a decision-maker would be willing to spend in order to gain perfect information for one or more inputs to an economic model.

Extended dominance: A scenario where a strategy can be excluded when it costs more and provides fewer benefits than a combination of two other alternatives.

Extended cost-effectiveness analysis: An extension to the conventional CEA framework that quantifies the distributional impacts of health interventions based on equity criteria as well as financial risk protection.

Externalities: Costs and consequences of an intervention such as a vaccination program that fall on other members of the population beyond those producing, purchasing or consuming the intervention (e.g., community immunity, age-shifting of disease).

Equity: The absence of unfair and avoidable or remediable differences in health among population groups defined by any relevant characteristic (e.g., medical, social, economic, demographic, geographic). Horizontal equity refers to individuals with like characteristics (of ethical relevance) being treated the same way, while vertical equity allows for individuals with different characteristics (of ethical relevance) to be treated differently in order to achieve more equitable outcomes.

First-order uncertainty: Uncertainty related to random variability. This type of uncertainty is also referred to as stochastic uncertainty.

Force of infection: The rate at which susceptible individuals become infected per unit time. It is a function of the number of infectious individuals in the population at a given time and the transmissibility of the infection.

Health technology assessment (HTA): The multi-disciplinary evaluation of various domains of a health technology in order to inform its use, which may include clinical effectiveness, cost-effectiveness, social impacts, ethical impacts, among others.

Health equity: See definition for Equity.

Health-related quality of life (HRQoL): A combination of a person's physical, mental and social functioning.

Health utility: A measure of health-related quality of life that represents preference values that individuals attach to their overall health status. Conventionally the valuations are anchored by 0 (representing a health state equivalent to being dead) and 1 (representing a health state equivalent to perfect health). Health utilities are also referred to as preference-based measures of health-related quality of life.

Herd immunity: See definition for community immunity.

Heterogeneity: Differences between individuals that can, in part, be explained. This differs from the random chance that individuals with the same underlying characteristics will experience a different outcome.

Incremental costs: Difference in mean expected costs associated with the use of an intervention compared with the use of an alternative. This is a key output of an economic evaluation.

Incremental cost-effectiveness ratio (ICER): A ratio that is calculated by dividing the difference in mean expected costs by the difference in mean expected health outcomes or effects between two alternatives being compared in an economic evaluation. The comparator usually represents the current standard of care.

Incubation period: The time from infection to onset of clinical disease.

Indigenous Peoples: The earliest known people groups of any land around the world. For the purposes of these guidelines, the term ‘Indigenous Peoples’ refers to individuals who are First Nations, Inuit, and Métis.

Infectious period: The time from the end of latent or pre-infectious period until the host is no longer able to transmit the infection to other individuals.

Individual-based model (or Microsimulation): A model in which the individual, rather than the group, is the unit that is modelled. Microsimulation models that do not allow for interactions among individuals are classified as static microsimulation models. Microsimulation models that do allow for interactions among individuals or with the environment (such as the healthcare system) are classified as dynamic microsimulation models. An agent-based model is a type of dynamic simulation model.

Intergenerational equity: The concept that people of different generations should benefit equitably from policy decisions such as expenditures on vaccination programs.

Latent period: The time period from when a host acquires an infection until they are able to transmit it to another host. It is sometimes referred to as the pre-infectious period.

Methodologic uncertainty: Uncertainty related to the different methods that can be used to conduct an economic evaluation.

Microsimulation: See definition for individual-based model.

National Advisory Committee on Immunization (NACI): A national advisory committee of experts in the fields of pediatrics, infectious diseases, immunology, pharmacy, nursing, epidemiology, pharmacoeconomics, social science and public health. NACI makes recommendations for the use of vaccines currently or newly approved for use in humans in Canada, including the identification of groups at risk for vaccine-preventable diseases for whom vaccination should be targeted.

Net health benefit: A summary statistic, expressed in QALYs, that represents the impact on population health related to a given intervention, adjusted for the expected costs if purchasing care at the rate of a marginally cost-effective strategy. It is calculated by subtracting the ratio of the expected costs by the health opportunity cost.

Net monetary benefit: A summary statistic that represents the value of an intervention as the impact on population health, expressed in monetary terms, adjusted for the expected costs if purchasing care at the rate of a marginally cost-effective strategy. It is calculated by multiplying the expected QALYs by the health opportunity cost and subtracting the expected costs associated with the intervention.

Open (population) model: A model that allows new individuals to enter the model either through births or in-migration, or to exit the model through deaths or out-migration over time.

Parameters: Variables that determine the rates of movement between model states or probabilities of events within a model.

Parameter uncertainty: Uncertainty in parameter estimates that are used to populate a model. This type of uncertainty is also referred to as second-order uncertainty.

Pathogen variations: Differences between strains of a pathogen related to serotypes, serogroups, or genotypes.

Perspective: The viewpoint from which an economic evaluation will be conducted. The perspective determines the outcomes and costs that will be included in the analysis.

Population-based model: A model in which groups of individuals are assigned to compartments or health states based on their health status or other characteristics. Individuals in each compartment move according to parameter values defined at the aggregate level and the model records the number of individuals in each compartment over time. This type of model can also be referred to as an aggregate model.

Positive time preference: The preference for present benefits over benefits occurring in the future.

Probabilistic analysis: A method used to quantify parameter uncertainty in an economic analysis where a probability distribution is assigned to each uncertain parameter and values are randomly sampled from each distribution repeatedly to generate a distribution of outcomes that can be analyzed.

Probabilistic sensitivity analysis (PSA): See definition for probabilistic analysis.

Productivity: A measure of how efficiently production inputs, such as labour and capital, are being used in an economy to produce a given level of output.

Randomized controlled trial (RCT): A comparative study, designed to ascertain the efficacy of a health intervention, in which units such as individuals are randomly assigned to either the intervention or control group.

Real-world evidence: Evidence used for decision-making that is collected through non-experimental studies.

Reference case: A set of methods for conducting an economic evaluation specified by the decision maker. The purpose of the reference case analysis is to ensure consistency between methods underpinning analyses and the decision-making process that is employed.

Quality-adjusted life year (QALY): A summary outcome measure used to quantify the health outcomes associated with a particular intervention. QALYs combine the impact of benefits related to both survival and health-related quality of life expressed as health utilities, and allow comparisons between interventions across disease states.

Scenario analysis: An analysis that tests alternate model scenarios underpinned by different plausible structural assumptions.

Second-order uncertainty: See definition for parameter uncertainty.

Selection bias: Bias in a non-randomized study resulting from systematic differences in sampling of individuals who are in the exposed group compared to those who are not, resulting in a distribution of exposures and outcomes that is no longer representative of the source population. Selection bias can also occur in randomized studies due to attrition post-randomization.

Serotype replacement: The expansion in non-vaccine serotypes of a pathogen resulting from the removal of vaccine-specific serotypes from the population that compete with them for colonisation of hosts.

Spillover effects: The effects of conditions and treatments on different aspects of the welfare of other individuals such as family members, including caregiver health effects, informal care time costs, or both.

Static model: A model in which the force of infection is constant over time or dependent only on characteristics of each individual, and not on the number of other individuals who are infectious. May also be referred to as a model with an exogenous force of infection.

Stochastic model: For the purposes of these guidelines, a model that accounts for first-order uncertainty where events are programmed to occur randomly.

Structural uncertainty: Uncertainty related to the structure of a model and other unparameterized sources of uncertainty. Scenario analysis is one approach for assessing this type of uncertainty.

Time horizon: The time period over which outcomes and costs are quantified in an economic evaluation.

Value of information analysis: An analysis used to estimate of the value, in terms of cost and health outcomes, of collecting more data on key parameters influencing a funding decision. It is most useful where the output of an economic evaluation is uncertain, but close to a decision threshold and a key parameter on which the output is based is uncertain. See definitions for Expected value of perfect information (EVPI) and Expected value of partial perfect information (EVPPI).

Introduction

This is the first edition of the National Advisory Committee on Immunization (NACI) Guidelines on the Economic Evaluation of Vaccination Programs in Canada; in the text hereafter, they will be referred to as the Guidelines. These Guidelines have been established to articulate best practices for conducting and reporting economic evaluations of vaccination programs in Canada, be they regional, provincial, or national. Adherence to common best practices can allow decision-makers in Canada’s publicly funded health system to have access to consistent and credible information to inform funding decisions related to vaccination programs. These Guidelines focus on information specific to the vaccination programs. The Canadian Agency for Drugs and Technologies in Health’s Guidelines for the Economic Evaluation of Health Technologies: Canada^{Footnote 1} present more general information applicable to health technologies in Canada, and where CADTH’s Guideline Statements are applicable to economic evaluations of vaccination programs, they have been included in these Guidelines.

The main feature that distinguishes vaccination programs from other health technologies is their population-level effects, which result from their potential to affect both vaccinated and unvaccinated individuals. These Guidelines present detailed information on how to incorporate these population-level effects into economic evaluations of vaccination programs, including methods for addressing their non-health sector impacts.

The recommendations contained in these Guidelines were formulated by NACI’s Economic Guidelines Task Group, which consisted of Canadian and international experts in infectious diseases and health economics. This group engaged in a series of discussions that led to decisions made by consensus, which were supported by literature reviews for selected topics. A peer-review and public consultation process was undertaken after completing an initial draft of the Guidelines and informed the final recommendations and text contained in this document.

A social decision-making framework has been adopted for these Guidelines. The basis of this framework is that the health decision-maker acts on behalf of a socially legitimate higher authority (e.g., a democratically elected government) to achieve an explicit policy objective (e.g., improving overall population health). The function of an economic evaluation within this framework is to inform social decisions.^{Footnote 2} ^{Footnote 3} ^{Footnote 4}

Economic evaluations have traditionally focused on the trade-offs between incremental costs and incremental effects of different health interventions to enable decision-makers to make judgments related to efficiency. These Guidelines expand on the traditional approach by presenting recommendations on integrating equity considerations into economic evaluations of vaccination programs. The integration of equity in economic evaluations is consistent with NACI’s Ethics, Equity, Feasibility, and Acceptability (EEFA) Framework, which provides a mechanism for decision-makers to systematically consider important programmatic factors, alongside effectiveness and cost-effectiveness, when making recommendations about vaccination programs.^{Footnote 5}

These Guidelines recommend adoption of two reference case analyses for the economic evaluation of vaccination programs: one conducted from the publicly funded health system perspective and the other conducted from the societal perspective. The latter is to account for the full range of benefits associated with vaccination programs, including those that accrue to non-health sectors. The purpose of these reference cases is to encourage the use of a standard set of methods when conducting economic evaluations of vaccination programs and to ensure that decision-makers are able to compare results between different vaccination programs.

Recommendations are presented for the following aspects of economic evaluations of vaccination programs: decision problem, types of evaluations, study populations, comparators, perspectives, time horizon, discounting, modelling, effectiveness, measurement and valuation of health, resource use and costs, analysis, uncertainty, equity, and reporting. Guidance on each of these topics is contained in a separate chapter. Guidelines Statements are presented at the beginning of this document and at the beginning of each chapter for ease of use, followed by a detailed discussion of the recommendations. The Guidelines are written for end-users, including researchers and decision-makers, who are technically proficient in the methods of economic evaluation, and as such, background on these methods has been omitted. Similarly, the Guidelines omit detailed background information on scientific and technical subjects related to vaccines and immunization, as it is expected that researchers undertaking economic evaluations of vaccination programs will consult with subject matter experts in this area. The references contained in this document provide sources for researchers to obtain additional information when required.

The guidance presented in this document represents NACI’s current recommendations for conducting economic evaluations of vaccination programs. NACI and the Economic Guidelines Task Group have attempted to reflect current best practices, but the recommendations contained in these Guidelines will evolve alongside scientific and methodological advancements in this area. Topics for which there is no current consensus on best practices and require further research have been identified in the Guidelines. As such, the function of these Guidelines is not only to recommend current practices for the economic evaluation of vaccination programs, but also to suggest directions for future research and that will contribute to advancing methods used in this area.

The remit of these Guidelines is to specify methods for conducting economic evaluations of vaccination programs, and not to provide guidance or insights into the decision-making process. As such, considerations or factors related to making funding decisions about vaccination programs are not included in the Guidelines.

Guideline statements

1. Decision problem

1.1 “The decision problem addressed by the economic evaluation should be clearly stated.” [CADTH Guideline Statement]

1.2 “The decision problem statement should provide a comprehensive specification of the interventions to be compared, the setting(s) in which they are to be compared, the perspective of the evaluation, which costs and outcomes are to be considered, the time horizon, and the intended population for the evaluation.” [CADTH Guideline Statement]

1.3 A separate decision problem statement is required for each perspective and for each analysis related to a distinct population group for which the vaccination program may be intended.

1.4 In addition to specifying the intended population for the vaccination program, the decision problem must also identify other population groups that could be affected by the vaccination program, including the population at risk for the disease of interest, and any populations that may be indirectly affected by the vaccination program, either through externalities or spillover effects.

2. Types of evaluations

2.2 In the reference cases, the economic evaluation should be cost-utility analyses (CUA) with outcomes expressed as quality-adjusted life-years (QALYs). Any departure from this approach should be clearly justified. [CADTH Guideline Statement with amendment]

2.3 A cost-benefit analysis (CBA) may be used alongside the reference case CUAs in situations where the vaccination program may be compared to a non-health intervention.

3. Study populations

3.1 Researchers should identify the intended population(s) for the vaccination program, the population at risk for the disease of interest, and any populations that may be indirectly affected by the vaccination program, either through externalities or spillover effects.

3.2 Researchers should present an overall analysis that includes the costs and health outcomes for all of the affected populations. When relevant, researchers should also summarize the results separately for each affected group (e.g., intended population, population experiencing externalities or spillover effects) that was included in the overall analysis.

3.3 Where there are factors that could lead to differences in costs and outcomes related to the vaccine program across subgroups, researchers should conduct separate economic evaluations for each subgroup. These factors could include demographic factors, behavioural factors, disease-related factors, and effectiveness of the vaccine or comparator intervention(s).

4. Comparators

4.1 The choice of comparator(s) should be related to the scope of the decision problem. As such, the comparators should reflect the intended population for the vaccination program and the jurisdiction for which the decision is being made. [CADTH Guideline Statement with amendment]

4.2 Researchers should consider both preventive and treatment-based approaches when selecting comparators for economic evaluations of vaccination programs. Preventive interventions could include vaccine-based measures, screening programs, preventive medication-based interventions, and preventive non-medical interventions.

5. Perspectives

5.1 Two reference case analyses should be presented as part of the economic evaluation of vaccination programs: one conducted from the publicly funded health system perspective, and the other conducted from the societal perspective.

5.2 “Both costs and outcomes should be consistent with the stated perspective.” [CADTH Guideline Statement]

6. Time horizon

6.1 In the reference cases, the time horizon should be long enough to capture all relevant differences in the future costs and outcomes associated with the interventions being compared. Thus, the time horizon should be based on the condition and the likely impact of the intervention. [CADTH Guideline Statement with amendment]

6.2 Researchers should justify their choice of time horizon. Where it spans a long period of time (i.e., multiple decades), researchers should report ICER estimates from various time points throughout the time horizon.

7. Discounting

7.1 In the reference cases, costs and outcomes that occur beyond one year should be discounted to present values at a rate of 1.5% per year. [CADTH Guideline Statement with amendment]

7.2 “The impact of uncertainty in the discount rate should be assessed by comparing the results of the reference cases to those from non-reference case analyses, using discount rates of 0% and 3% per year.” [CADTH Guideline Statement]

8. Modelling

8.1 “Model conceptualization and development should address the decision problem.” [CADTH Guideline Statement]

8.2 “Researchers should consider any existing well-constructed and validated models that appropriately capture the clinical or care pathway for the condition of interest when conceptualizing their model.” [CADTH Guideline Statement]

8.3 The model structure should reflect the natural history of disease, the clinical or care pathway, and account for susceptibility, infectiousness, and immunity, related to the infection.

8.4 Relevant behavioural dynamics including contact patterns between individuals and behaviours related to infection prevention and control should be incorporated into the model where appropriate.

8.5 Dynamic models should be considered in economic evaluations of vaccines that are associated with externalities such as prevention of human-to-human transmission of infection and age-shifting of disease.

8.6 Other model attributes including whether the model is deterministic or stochastic, population-based or individual-based, and open or closed should be considered in the context of the decision problem.

8.7 Researchers should transparently report on model calibration and validation processes that were undertaken and on their results.

9. Effectiveness

9.1 “A comprehensive search of the available data sources should be conducted to inform the estimates of effectiveness and harms associated with the interventions. Report the included studies and methods used to select or combine the data.” [CADTH Guideline Statement]

9.2 “The data sources should be assessed based on their fitness for purpose, credibility, and consistency. Describe the trade-offs among these criteria and provide justification for the selected source(s).” [CADTH Guideline Statement with amendment]

9.3 The following criteria should be considered when assessing estimates of vaccine effectiveness: vaccine effectiveness by dose; expected vaccine coverage; pathogen variation-specific (i.e., serotypes, serogroups, strains) effectiveness; and geographic and host factors that may affect effectiveness.

9.4 Researchers should ensure that immune biomarkers used as surrogate outcomes in studies of vaccine efficacy or effectiveness meet the criteria for correlates of protection.

10. Measurement and valuation of health

10.1 In both reference cases, the quality-adjusted life year (QALY) should be used as the method for valuing health outcomes.

10.2 “Health preferences should reflect the general Canadian population.” [CADTH Guideline Statement]

10.3 In the reference cases, researchers should use health preferences obtained from an indirect method of measurement that is based on a generic classification system (e.g., EuroQol 5-Dimensions questionnaire [EQ-5D], Health Utilities Index [HUI], Short Form 6-Dimensions [SF-6D], Child Health Utility 9-Dimensions [CHU9D], Assessment of Quality of Life [AQoL]). Researchers must justify where an indirect method is not used. [CADTH Guideline Statement with amendment]

10.4 “The selection of data sources for health state utility values should be based on their fitness for purpose, credibility, and consistency. Describe the trade-offs among these criteria and provide justification for the selected sources.” [CADTH Guideline Statement]

11. Resource use and costs

11.1 For each reference case analysis, researchers should systematically identify, measure, value, and report all relevant resources consumed or saved as a result of the delivery or implementation of the vaccination program under consideration.

11.2 Where possible, researchers should value relevant resources identified for all sectors in monetary terms. In situations where this is not possible, researchers should present the relevant resources that have been identified in the Impact inventory table for economic evaluations of vaccination strategies for consideration by decision-makers.

11.3 “Resource use and costs should be based on Canadian sources and reflect the jurisdiction(s) of interest (as specified in the decision problem).” [CADTH Guideline Statement]

11.4 When valuing and monetizing resources, researchers should select data sources that most closely reflect the opportunity cost, given the perspective of the analysis. [CADTH Guideline Statement with amendment]

11.5 Researchers should assess sources used for cost data based on their fitness for purpose, credibility, and consistency. The selection of data sources should be based on trade-offs between these criteria.

12. Analysis

12.1 Incremental cost-effectiveness ratios (ICERs) and, where useful for interpretation, net monetary benefits or net health benefits, should be calculated for both reference case analyses.

12.2 “For analyses with more than two interventions, a sequential analysis of cost-effectiveness should be conducted following standard rules for estimating ICERs, including the exclusion of dominated interventions.” [CADTH Guideline Statement]

12.3 The expected values of costs and outcomes, where possible, should be generated probabilistically to reflect the overall uncertainty in the model parameters.

13. Uncertainty

13. Researchers should address parameter uncertainty using a probabilistic reference case analysis, where possible, as well as deterministic sensitivity analyses.

13.1 “Methodological uncertainty should be explored by comparing the reference case results to those from a non-reference case analysis that deviates from the recommended methods in order to examine the impact of methodological differences.” [CADTH Guideline Statement]

13.2 Cost-effectiveness acceptability curves (CEACs) and cost-effectiveness acceptability frontiers (CEAFs) should be used to represent the uncertainty in the estimates of costs and outcomes when these estimates have been generated probabilistically. [CADTH Guideline Statement with amendment]

13.3 When the decision problem includes the option of commissioning or conducting future research, value-of-information analysis may be helpful to characterize the value of these options and design future research and may be included in the reference case analysis. [CADTH Guideline Statement with amendment]

13.4 Scenario analyses should be used to assess structural uncertainty. [CADTH Guideline Statement with amendment]

14. Equity

14.1 Researchers and decision-makers should work together to establish which equity dimensions and goals should be included in the economic evaluation of the vaccination program being considered. Equity should be considered in the context of NACI’s Ethics, Equity, Feasibility, and Acceptability (EEFA) framework.

14.2 Analyses that incorporate relevant equity concerns should accompany the reference case analysis (e.g., distributional cost-effectiveness analysis, extended cost-effectiveness analysis, or other emerging methods), and presented alongside the reference case.

15. Reporting

15.1 “The economic evaluation should be reported in a transparent and detailed manner with enough information to enable the reader or user (e.g., decision-maker) to critically assess the evaluation. Use a well-structured reporting format.” [CADTH Guideline Statement]

15.2 “A summary of the evaluation written in non-technical language should be included.” [CADTH Guideline Statement]

15.3 “Results of the economic evaluation should be presented in graphical or visual form, in addition to tabular presentation.” [CADTH Guideline Statement]

15.4 “Details and/ or documents describing quality assurance processes and results for the economic evaluation should be provided. An electronic copy of the model should be made available for review with accompanying documentation in adequate detail to facilitate understanding of the model, what it does, and how it works.” [CADTH Guideline Statement]

15.5 “Funding and reporting relationships for the evaluation should be described, and any conflicts of interest disclosed.” [CADTH Guideline Statement]

15.6 Researchers should use NACI’s Guidelines for Reporting Economic Evaluations of Vaccination Programs in Canada, and complete the Impact inventory table for economic evaluations of vaccination strategies, which is found in Appendix 1.

Guidelines in detail

1. Decision problem

1.1 “The decision problem addressed by the economic evaluation should be clearly stated.” [CADTH Guideline Statement]

1.3 A separate decision problem statement is required for each perspective and for each analysis related to a distinct population group for which the vaccination program may be intended.

The decision problem being addressed by an economic evaluation of a vaccination program should address concerns relevant to decision-makers and be clearly articulated at the outset of the analysis, while ensuring consistency with other vaccine evaluations where possible. Decision-makers who assess economic evaluations of vaccination programs in Canada include NACI, provincial/ territorial immunization technical advisory groups, and provincial/ territorial Health Ministries. NACI develops non-binding, evidence-informed recommendations to facilitate timely decision-making for publicly funded vaccine programs at provincial and territorial levels. Some provinces and territories have formal immunization technical advisory groups while others do not. Formal advisory groups and Health Ministries make decisions on whether a vaccination program will be funded in a given jurisdiction, and how it will be implemented.

The decision problem should provide a detailed description, and justification, of the vaccination program being evaluated, including: 1) the perspectives from which the analysis is being carried out; 2) the type of economic evaluation being conducted; 3) which costs and outcomes will be quantified in the analysis; 4) the time horizon over which the analysis will be carried out; 5) the comparators that will be considered; and 6) the populations affected by the vaccination program. These populations include the intended population for the vaccination program, and where applicable, the population at risk for the disease of interest, and the population that may experience spillover effects (e.g., informal caregivers).

It should state all possible population subgroups that the decision-maker is considering vaccinating (e.g., age groups, clinical risk groups, people in certain professions, geographical areas, individuals who possess certain biomarkers or genetic profiles), as well as the potential vaccine delivery setting (e.g., physician clinics, pharmacies, schools, workplaces). All options of interest to the decision-maker should be evaluated together using the principles of full incremental analysis.

Researchers should seek out and engage with decision-makers to gain an understanding of the concerns they are intending to address with the introduction of the vaccination program. Some aspects of the decision problem that are particularly relevant to decision-makers include, but are not limited to: 1) the time horizon of the evaluation; 2) possible impacts of the vaccination program beyond the health sector; and 3) health inequities that could potentially be affected by introduction of the vaccination program.

Ensuring a time horizon that is relevant to decision-makers is particularly important when a vaccination program results in protection for unvaccinated individuals through community immunity (as known as herd immunity) with the potential for disease elimination. Often a very long time horizon (sometimes many decades) is required for the full costs and effects of a vaccination program to become apparent. Researchers should note that these long time horizons may not reflect present-day outcomes and costs that are relevant to decision-makers. In these cases, researchers should ensure that results of an economic evaluation are reported from several time points to allow decision-makers to determine when payoffs of the program become positive.

Given the broader, non-health-related outcomes that are associated with many vaccination strategies, a perspective broader than the health system perspective will usually be relevant. In these guidelines, health system refers to both healthcare treatment services and Public Health. Researchers should attempt to gain an understanding of the broader costs and benefits related to the vaccination program that may fall outside of the health sector, and that are relevant to the decision-maker. For example, because measles can lead to neurologic damage, preventing measles through childhood vaccination improves educational outcomes. Similarly, preventing influenza in the population through a universal vaccination program leads to productivity-related benefits. Further details on this topic are found in Chapter 5 on Perspectives.

Certain groups are vulnerable to infectious disease and the adverse impacts of infectious disease control policies due to historical harms and socially constructed barriers. Vaccines have been identified as a strategy to potentially reduce specific inequities relating to risk of infection or burden of the disease in question. Researchers, in collaboration with decision-makers, should identify specific groups that may especially benefit from the vaccination program. For example, individuals of lower socioeconomic status and those belonging to minorities experience a higher incidence of cervical cancer and greater mortality related to the disease, which could be prevented through a human papilloma virus (HPV) vaccination program.^{Footnote 6} Conversely, researchers, in collaboration with decision-makers, should also consider whether some groups may not benefit from the vaccination program, thus potentially increasing health inequities. A further discussion on this topic is found in Chapter 14 on Equity.

The type of economic evaluation should be specified and justified. The type of economic evaluation, along with the perspective, will determine which costs and outcomes should be included (and how). The included outcome measures, which should be the same for each comparator, should be explicitly stated in the decision problem and listed by sector (e.g., health outcomes, educational achievement). Similarly, the included costs should be explicitly stated and listed by cost category (e.g., healthcare costs, education-related costs, productivity-related costs).

A clear description of the vaccine being evaluated including the dosage of vaccine, the number of doses required, dose schedule, whether any booster doses are required, expected dose completion, handling of vaccine wastage, assumptions on waning, coverage estimates, and setting of vaccine delivery should be provided along with detailed descriptions of comparators. Comparators could include other existing preventive vaccines, non-vaccine-based preventive approaches, and current treatment approaches including best supportive care.

2. Types of evaluations

2.1 In the reference cases, the economic evaluation should be cost-utility analyses (CUA) with outcomes expressed as quality-adjusted life-years (QALYs). Any departure from this approach should be clearly justified. [CADTH Guideline Statement with amendment]

2.2 A cost-benefit analysis (CBA) may be used alongside the reference case CUAs in situations where the vaccination program may be compared to a non-health intervention.

In the reference cases, the economic evaluation should be a cost-utility analysis (CUA) with outcomes expressed as quality-adjusted life-years (QALYs). There is recognition, however, that there are populations in whom CUAs cannot be robustly conducted because valid instruments for direct utility elicitation do not exist, such as children under 8 years of age. In these cases, alternative analytic approaches such as cost-effectiveness analysis (CEA) with a relevant outcome measure in natural health units should be justified.

In addition to the reference case CUAs, a cost-benefit analysis (CBA) may be presented in cases where broader impacts beyond health are important factors for decision-makers. CBA has been proposed as a method to evaluate vaccination programs associated with consequences that fall outside of the health sector.^{Footnote 7} ^{Footnote 8} ^{Footnote 9} NACI’s recommendation to conduct a reference case CUA from the societal perspective should enable researchers to account for non-health sector benefits by monetizing them, and including them in the incremental costs and subsequently the numerator of the incremental cost-effectiveness ratio (ICER) estimate. This approach, however, does not enable decision-makers to compare non-health benefits of alternative programs, or to compare vaccination programs to non-health programs since the denominator of the ICER estimate is reported in quality-adjusted life years (QALYs). In cases where a decision-maker may be interested in comparing the economic attractiveness of a vaccination program to a non-health intervention (e.g., school lunch program), researchers could present a CBA alongside the societal perspective reference case analysis to enable such a comparison.^{Footnote 7} Researchers should be aware that different approaches can be used to monetize benefits in a CBA, and that this could lead to wide variations in the results of a CBA.^{Footnote 8} The choice of a particular approach needs to be specified and justified.

3. Study populations

The results of any economic evaluation of a vaccination program depend on the impact of the vaccination program on three populations: 1) the intended population(s) for the vaccination program; 2) the population at risk for the disease of interest; and 3) population(s) that may experience externalities or spillover effects. In cases where a vaccination program is associated with externalities, both the intended population for the vaccination program and the population expected to experience externalities should be identified in the decision problem. Researchers should identify any externalities associated with vaccination programs (e.g., community immunity, age-shifting of disease), and the population(s) they are expected to affect. For example, a measles vaccination program intended for infants and children may result in community immunity that could potentially lead to population-wide disease elimination. Another example is varicella vaccine intended for young children for the prevention of chickenpox, which could increase the incidence of herpes zoster in the general population. Further details on incorporating externalities into economic evaluations are provided in Chapter 8 on Modelling. Additionally, researchers should identify any population(s) that may experience spillover effects (e.g., caregivers) related to the vaccination program.

Researchers should provide a detailed description of each population being considered in the analysis that includes age, gender, and geographic location. Researchers should also describe any other factors that determine eligibility for the vaccination program being evaluated, and factors that may affect the magnitude of the externalities experienced.

Researchers should present an overall analysis that includes the costs and outcomes for all of the affected populations, including the group(s) identified for the vaccination program, and any groups that may experience externalities or spillover effects. Such an analysis should be presented for each implementation strategy that is being considered (e.g., universal vaccination, vaccination of high-risk groups only, vaccination of children only). When relevant, researchers should also summarize the results separately for each affected group (e.g., intended population, population experiencing externalities or spillover effects) that was included in the overall analysis.

However, in situations where heterogeneities may affect the results of an economic evaluation have been identified between groups of individuals, economic evaluations for different strategies that improve coverage in each of the subgroups and present outcomes stratified by subgroup should be undertaken. This should ideally based on an underlying mathematical model that considers all the subgroups and interactions between them. Important heterogeneities with respect to vaccination programs could include demographic factors (e.g., age, gender, geographic location), behavioural factors (e.g., expected uptake of the vaccination program, risk-taking behaviours), disease-related factors (e.g., natural history of the disease, risk of disease transmission), effectiveness of the vaccine or comparator intervention(s), and health utilities or costs associated with the health states or interventions included in the analysis.

4. Comparators

When selecting comparators for economic evaluations of vaccination programs, researchers should consider all current interventions, those that may become available in the near future, and those that may be displaced by the vaccination program being evaluated. Interventions used for both prevention or treatment of the disease of interest should be considered. Preventive interventions could include vaccine-based measures, screening programs, preventive medication-based interventions, and preventive non-medical interventions. Often, more than one comparator will be relevant for the economic evaluation, and therefore, all relevant comparators should be included.

Vaccine-based measures could include alternative vaccines against the same pathogen (e.g., parenteral trivalent inactivated vaccine versus intranasal live attenuated vaccine for influenza, whole-cell versus acellular vaccines for pertussis, mRNA versus viral vector vaccines for coronavirus infection disease 2019 (COVID-19)), or vaccines with additional valents (e.g., 10-valent and 13-valent pneumococcal conjugate vaccines (PCV10 and PCV13)). Vaccine-based measures could also include different implementation or delivery with the same vaccine product. Examples include universal vaccination versus vaccination of high-risk groups only; vaccination of the intended age group with no catch-up strategy versus vaccination of the intended age group with a catch-up strategy for other age groups; vaccination strategy with no booster doses versus strategy with booster doses; vaccination strategies based on a differing number of doses or differing administration schedules; or different settings for delivery such as a school-based strategy versus a public health clinic-based strategy versus mass vaccination strategy in hot spots or warehouses.^{Footnote 10} ^{Footnote 11} ^{Footnote 12} ^{Footnote 13}

Screening programs (also known as secondary prevention) could include regular exams and tests to detect disease in its earliest stage. They may be changed by the introduction of vaccination programs. For example, HPV vaccination may change the value and necessity for routine cytological smears for cervical cancer screening.

Preventive medication-based interventions may involve pre- or post-exposure administration of medications. Examples include anti-malarial medications for Canadian travellers to malaria-endemic regions, and pre- and post-exposure prophylaxis for human immunodeficiency virus (HIV).

Preventive non-medical interventions could include physical measures such as condoms to prevent sexually transmitted infections, face masks to prevent transmission of respiratory infections, or behavioural modifications such as physical distancing and hand washing to prevent infections that are transmitted through close personal contact between individuals.

When treatment-based comparators are being considered, researchers should be aware that best supportive care should be considered the relevant comparator in cases where no curative treatments exist for the disease of interest.

5. Perspectives

5.2 “Both costs and outcomes should be consistent with the stated perspective.” [CADTH Guideline Statement]

Two reference case analyses should be conducted as part of the economic evaluation of vaccination programs: one conducted from the publicly funded health system perspective, and the other conducted from the societal perspective. In these guidelines, health system refers to both healthcare treatment services and Public Health.

Publicly funded health system perspective

For the reference case analysis conducted from the publicly funded health system, the scope of the perspective should be defined to include a single provincial/territorial publicly funded health system, multiple regional publicly funded health systems, or a national system. Researchers should include: 1) health outcomes experienced by vaccinated individuals and their informal caregivers; and 2) costs incurred by the health system. It must be recognized that when the reference case analysis includes multiple publicly funded health systems, the publicly funded cost items may vary from jurisdiction to jurisdiction (e.g., prescription medications), or even within a jurisdiction (long-term care). Variations in what items are included across systems should be made transparent.

In cases where vaccines are associated with externalities, the health outcomes and costs considered in the analysis also include those experienced by unvaccinated individuals since vaccine plays a critical role in population health.^{Footnote 14} Population-level health outcomes that should be considered include: 1) incidence of infection and disease in vaccinated and unvaccinated individuals; 2) changes in the age distribution of individuals who are infected as a result of age-shifting related to the vaccination program (when this has consequences on the overall disease burden as a result of age-dependency in severity of disease); 3) emergence of new diseases related to variations of the pathogen (i.e., serotypes, serogroups, strains) or unrelated pathogens that may replace the one(s) targeted by the vaccine; and 4) disease eradication.

Population-level costs that should be considered from this perspective include: vaccination program implementation, delivery and sustainment costs including public health campaigns; transaction costs related to introduction of new vaccines or switching between vaccines; costs related to screening, diagnosis, and treatment of disease; and epidemiological surveillance, contact tracing, case investigations, and outbreak investigations. Guidance on quantifying the costs associated with these outcomes is found in Chapter 11 on Resource Use and Costs.

Societal perspective

A societal perspective reference case analysis is also recommended because many vaccines prevent diseases that have impacts in areas beyond health. For example, the Haemophilus influenzae type b (Hib) vaccine administered to infants prevents neurological sequelae (e.g., deafness, blindness, developmental delays), all of which would affect a child’s school attendance, future productivity and consumption, as well as broader well-being.^{Footnote 15} ^{Footnote 16} Even relatively mild diseases such as childhood diarrhea resulting from rotavirus infection can lead to impacts outside of health. In many cases, medical attention is not required to treat these infections; nonetheless, a parent is required to take time off work to care for the sick child.^{Footnote 17} ^{Footnote 18} Finally, diseases such as COVID-19 have tremendous health and economic impacts that extend to every area of the economy,^{Footnote 19} and their impacts could be mitigated through vaccination programs.^{Footnote 20} ^{Footnote 21} Failing to consider the full range of benefits associated with vaccines underestimates the role of health as a driver of economic activity and well-being, and could lead to undervaluation of vaccination programs.^{Footnote 14}

The societal perspective analysis captures all the health outcomes and health system costs from the health system perspective. In addition, it captures impacts that fall outside of the publicly funded health system, including: healthcare costs not publicly funded by the health system, direct out-of-pocket costs, productivity, consumption, education, social services, and environment. Longer term impacts such as the effect of childhood illness on their neurodevelopmental impairment, educational attainment and subsequent long-term productivity (and consumption) should also be considered where relevant and feasible. These potential impacts are listed along with examples in Table 1: Impact inventory table for economic evaluations of vaccination strategies. This table was adapted from the impact inventory published by the 2^nd Panel on Cost-Effectiveness^{Footnote 22} to also include broader impacts associated with vaccines described in the literature.^{Footnote 14} ^{Footnote 22} ^{Footnote 23} ^{Footnote 24} ^{Footnote 25} The table provides a comprehensive list of health and non-health impacts that could result from vaccination programs. The intent is to allow researchers to consider the impacts systematically when planning for, and conducting economic evaluations of vaccination programs. Specific guidance on quantifying these impacts and their associated costs is found in Chapter 11 on Resource Use and Costs.

Researchers should complete and present Table 1 as part of their analysis to explicitly indicate which impacts are included and excluded in the economic evaluation for each of the two reference case analyses. The comments column could be used to provide justification for including or excluding certain impacts or to provide additional information.

6. Time horizon

6.2 Researchers should justify their choice of time horizon. Where it spans a long period of time (i.e., multiple decades), researchers should report incremental costs, incremental effects, and ICER estimates from various time points throughout the time horizon.

Models used to estimate the cost-effectiveness of vaccination programs can be closed or open models. Closed models follow a cohort of individuals over a length of time and do not allow for the entry of new individuals into the model. Most Markov (state-transition) models are closed models. Closed models are usually static, meaning that they do not account for disease transmission dynamics between individuals. Open models, on the other hand, do allow for entry of new individuals into the model over time (e.g., via new births, immigration), specifically to account for disease transmission dynamics within a population over time.^{Footnote 26}

Since closed models of vaccination programs follow a single group of individuals, these models should follow the group for a long enough time horizon to capture all important differences in future costs and outcomes related to the vaccination strategies being compared.

Open models may have time horizons that extend beyond the life of any individual alive at the start of the simulation, and so may require a time horizon that spans multiple birth cohorts. This is particularly true for vaccines that provide population-level protection through community immunity over multiple birth cohorts. For example, a cohort of individuals vaccinated against measles today may prevent transmission of this infection to another cohort years later. Individuals who are not vaccinated would benefit from this protection for the rest of their lives, in turn not infecting future cohorts with measles who would also benefit for the rest of their lives.^{Footnote 27} ^{Footnote 28}

There are typically three phases in open models: 1) the run-in phase; 2) the evaluation phase; and 3) the steady-state phase. The run-in phase must account for epidemiologic characteristics of the disease prior to introduction of the vaccine in order to realistically and accurately predict uptake of the vaccine. The epidemiologic estimates used during the run-in phase should be validated based on historical data pertaining to the disease of interest. More information on validation can be found in Chapter 8 on Modelling. The evaluation phase begins when the vaccination program is implemented in the intended population, and should be long enough to account for externalities associated with the program. The steady-state phase begins once epidemiologic variation terminates.^{Footnote 29}

For the measles example above, and for similar vaccines, the model time horizon should continue until the undiscounted ICER reach a steady-state. This is when the ratio between cumulative incremental costs and cumulative incremental health outcomes (QALYs) between the interventions being compared stabilizes. In these cases, the appropriate duration of the model time horizon should be ascertained during, rather than prior, to the analysis.^{Footnote 9}

Stability of the undiscounted incremental estimates and ICER as a criterion should mean that the time horizon of the model will be long enough to capture the full costs and benefits of community immunity as well as any other externalities (e.g., age-shifting of disease) associated with a vaccination program. Researchers should note that models might achieve epidemiologic steady-state prior to the incremental estimates and ICER estimates stabilizing. For example, one program may continue accruing costs or QALYs relative to the other even after epidemiologic equilibrium has been realized. In these cases, the model time horizon needs to continue until the ICER estimate has stabilized.

For some vaccinations programs, modelling a very large number of birth cohorts may be required to achieve stable ICER estimates, but this approach may not be practical or appropriate for the decision-making process.^{Footnote 28} For instance, researchers should note that modelling a large number of birth cohorts is not required in situations where the vaccination program is not expected to result in disease elimination or to take many years to deliver its full impact, such with some seasonal illnesses (e.g., current vaccines against influenza), or with infections whose source is non-human and transmission between individuals is not possible (e.g., tetanus). If the model is not run until the incremental estimates and ICER have stabilized, researchers should justify why this is the case, and define the run time in terms of time horizon or number of cohorts, and provide justification for this choice.^{Footnote 30}

In justifying the time horizon and number of cohorts, researchers should be aware of the trade-offs between bias and uncertainty. Shorter time horizons may introduce bias into cost-effectiveness estimates since they do not allow for enough time to account for epidemiologic changes resulting from the vaccination program. Shorter time horizons place a greater weight on upfront costs related to initiation of the vaccination program relative to later or annual costs, and reduce the consequences of discounting on measured outcomes. Shorter time horizons also may not quantify all of the benefits accrued to the final cohorts vaccinated. This may not be an issue for large-scale vaccine programs where the payer borrows to fund the program, and costs are annuitized.

For models with long time horizons, researchers should consider the potential for future changes that might alter the costs and benefits of the vaccine (e.g., technological change, long-term estimates of vaccine effectiveness, demographic projections).^{Footnote 10} ^{Footnote 31} ^{Footnote 32} While some of this uncertainty may be accounted for in the discount rate (in particular, by the “catastrophic risk”— the risk of an unanticipated event removing much of the value of the intervention), researchers may wish to consider context-specific, long-term uncertainties such as the emergence of treatment-resistant disease. Where the time horizon spans a long period (i.e., multiple decades), researchers should report ICER estimates from various time points throughout the time horizon.

In some jurisdictions, the effects of high, upfront implementation costs for vaccination programs can be tempered by annuitization, reflecting the borrowing practices of government agencies to fund wide-scale programs. Regardless of the time horizon length, analyses should indicate the rates used for annuitization and amortization (if applicable). It may be appropriate to conduct the analysis with and without annuitization of upfront implementation costs.

7. Discounting

7. 1 In the reference cases, costs and outcomes that occur beyond one year should be discounted to present values at a rate of 1.5% per year. [CADTH Guideline Statement with amendment]

Discounting costs, health outcomes, and non-health outcomes in economic evaluations reflects the societal preference for present consumption over future consumption. This is because discounting reduces the value of future costs and outcomes compared to their present value. The discount rate accounts for the social rate of time preference, growth rates in healthcare resources and the consumption value of health, and uncertainty about whether future health outcomes will be realized.^{Footnote 33} ^{Footnote 34}

Discounting in economic evaluations of vaccination programs can have a profound effect on the cost-effectiveness of programs, particularly in situations where the expected benefits of the vaccination program may not become apparent for years or even generations (e.g., prevention of cervical cancer through an HPV vaccination program). Pediatric populations are particularly susceptible to these effects. Discounting of health and non-health outcomes that accrue in the distant future may lead to a considerably reduced present value of outcomes. For example, in a cost-effectiveness analysis of an HPV vaccination program, the authors report an undiscounted ICER of €7,600/QALY, which increased to €59,100/QALY when a 4% discount rate was applied to both costs and benefits (3,462 undiscounted QALYs versus 438 discounted QALYs).^{Footnote 35}

The two most common approaches to discounting in economic evaluations of vaccination programs are: 1) constant discounting, where the same fixed discount rate is applied to both outcomes and costs; and 2) differential discounting, where a lower discount rate is applied to outcomes compared to costs.^{Footnote 33} ^{Footnote 34} ^{Footnote 35} ^{Footnote 36}

The approach most commonly employed in economic evaluations of vaccination programs is constant discounting, which is also the approach most commonly used for non-vaccine health interventions.^{Footnote 34} Some national immunization technical advisory group guidelines and health technology assessment (HTA) guidelines, recommend differential discounting approaches in uncertainty analyses or in special circumstances.^{Footnote 37} ^{Footnote 38} Arguments in favour of constant discounting of outcomes and costs include consistency and horizontal equity.^{Footnote 34} The consistency argument posits that health technologies associated with the same outcomes and costs over the same analytic time horizon receive equal priority by decision-makers, regardless of the time at which they are initiated.^{Footnote 39} This is because of the constant value of health over time. The horizontal equity argument posits that all individuals who potentially benefit from a vaccination program are treated equally, regardless of when they experience the benefits relative to when the program was initiated. Constant discounting prevents vaccination programs that span multiple generations from being given preference over programs that span a shorter time.^{Footnote 33}

One argument for differential discounting with a lower discount rate for health and non-health outcomes compared to costs is to normatively place more weight on future benefits. Another main argument for differential discounting is the increasing value that future health expectedly represents, or the changing thresholds for judging cost-effectiveness.^{Footnote 40} Discount rates could be adjusted to reflect these changes, although they could also be dealt with more explicitly in an analysis. With respect to vaccination programs, long time horizons— often generations, are required to achieve outcomes related to indirect effects of community immunity, which benefit not only the vaccinated individuals but also future cohorts through disease elimination or eradication.^{Footnote 34} Using constant discounting, particularly with higher discount rates, may render the present value of these programs close to zero. Differential discounting, on the other hand, increases the present value of outcomes occurring in the distant future compared to a constant discounting strategy.

A downside of differential discounting is that strategic use of time horizons and the number of included cohorts could alter cost-effectiveness estimates. O’Mahony et al., for instance, provide an example comparing constant and differential discounting approaches in a cost-effectiveness analysis of an HPV vaccination program in 12-year-old girls. The authors considered 1, 10, 20, and 30 birth cohorts. They discounted health outcomes and costs with an equal rate of 4%, and with differential rates of 1.5% and 4% respectively. As expected, they demonstrated that the ICER decreased as the number of cohorts increased with the differential discounting strategy, but not with the constant strategy.^{Footnote 41} Although normative and analytical solutions to this problem have been formulated,^{Footnote 41} ^{Footnote 42} it does raise potential concerns that unjustified analytic choices in economic analyses could lead to variations in results. This underscores the need for appropriate guidance on the use of differential discounting.

Researchers should discount health outcomes, non-health outcomes, and costs occurring beyond one year at a rate of 1.5% per year in the reference case analyses. This value represents the real cost of long-term borrowing for Canadian provinces, who are the authorities responsible for funding the majority of the Canadian healthcare system,^{Footnote 4} and approximates the rate at which society is willing to trade-off consumption today for consumption in the future.^{Footnote 34}

Sensitivity analyses should be conducted using rates of 0% and 3% per year applied to both outcomes and costs to account for any uncertainty in the discount rate. The low discount rate in the reference case analyses mitigates some of the concerns regarding present values of expected outcomes in the distant future. In situations where effects of a vaccination program span multiple generations and may be affected by the discounting strategy or rate used in the analysis, presenting undiscounted results will help decision-makers to assess potential intergenerational effects. A discussion of intergenerational equity is found in Chapter 14 on Equity.

8. Modelling

8.1 “Model conceptualization and development should address the decision problem.” [CADTH Guideline Statement]

8.3 The model structure should reflect the natural history of disease, the clinical or care pathway, and account for susceptibility, infectiousness, and immunity related to the infection.

8.4 Relevant behavioural dynamics including contact patterns between individuals and behaviours related to infection prevention and control should be incorporated into the model where appropriate.

8.7 Researchers should transparently report on model calibration and validation processes that were undertaken and on their results.

This chapter presents considerations related to constructing models used to estimate the cost-effectiveness of vaccination programs. A look at dynamic versus static models in the context of infectious disease modelling is presented, followed by an overview of other model attributes. Finally, recommendations related to model calibration and validation are presented.

Model structure and attributes

The model’s structure and attributes should reflect the natural history of disease, and include all relevant health states and transitions between these states. There are two primary considerations when conceptualizing a model used to estimate the cost-effectiveness of a vaccination program: whether transmission of infection between individuals is important in estimating the effects of a vaccination program; and whether individual behaviours and characteristics are important in understanding outcomes related to a vaccination programs. Researchers should refer to more detailed model taxonomies by Brennan et al.,^{Footnote 43} Kim and Goldie,^{Footnote 26} Stahl^{Footnote 44} and Mac et al.^{Footnote 45} for additional details if required.

Further guidance on constructing models for economic evaluation of vaccination programs can be found in Chapter 13 on Uncertainty (e.g., ensuring that the model structure accounts for factors related to transmission of infection between individuals, the natural history of the disease being modeled, as well as the direct and indirect effects of the vaccination program).^{Footnote 28} ^{Footnote 46}

Endogenous vs. exogenous infection rate

Models need to capture externalities related to vaccination programs such as community immunity and age-shifting of disease. In these guidelines, the terms “dynamic model” and “static model” refer to the nature of the incidence rate being dynamic or not (i.e. changing over time based on the proportion of the population that is infectious). They may also be referred to as having an “endogenous” or “exogenous” rate of infection, respectively.

Static models, which typically use a constant risk of exposure, do not explicitly represent dynamic infection transmission. These models are acceptable for use in economic evaluations of vaccination programs where there is no human-to-human transmission (e.g., tetanus or rabies).^{Footnote 9} They are also acceptable in situations where the intended group for vaccination is not epidemiologically influential with respect to transmission (e.g., hepatitis A vaccination of healthcare workers, influenza or pneumococcal vaccination in the elderly).^{Footnote 9} ^{Footnote 28} Static models may also be acceptable for infections where the individual is already a “host” (e.g., some pneumococcal strains; varicella-zoster virus where incidence of infection is more a random event in a person’s life after long-standing colonization). Finally, a static model is acceptable when: 1) a vaccination program is demonstrated to be cost-effective, and a dynamic model would only serve to reinforce this conclusion by accounting for infections prevented through indirect protection or secondary transmission; or 2) a vaccination program is not demonstrated to be cost-effective, but there are epidemiological or modelling data available that will allow estimation of the magnitude of community immunity or secondary transmission in the same or very similar setting.^{Footnote 28} ^{Footnote 47}

Although the scenarios above outline situations where static models may be acceptable for estimating the cost-effectiveness of vaccination programs, researchers should be aware of their limitations. First, when a static model has demonstrated the cost-effectiveness of a vaccine despite not accounting for the effects of community immunity or secondary transmission, the true cost-effectiveness of the intervention may be underestimated, and could result in biased resource allocation decisions.^{Footnote 28} Second, when epidemiological or modelling data are used to estimate the magnitude of community immunity or secondary transmission in the context of static models, there may be biased cost-effectiveness estimates when the data used are from a different population than the one considered in the model and when there are other important differences. Also, if the data used are from epidemiologic equilibrium, the fluctuation in prevalence in the initial post-vaccination period will not be accounted for. This limitation is particularly important for vaccination programs with a positive time preference given that the initial time period is when most of the costs and benefits related to the vaccine are accrued.^{Footnote 47} Examples of static models include decision trees, cohort-based Markov models, and discrete event simulations.

Dynamic models, which explicitly represent infection transmission, should be considered in economic evaluations of vaccination programs where human-to-human transmission is an important factor. For example, dynamic models should be employed when a large-scale vaccination program is expected to change the force of infection leading to control, elimination, or eradication of a disease by preventing its transmission.

Dynamic models should also be employed when serotype replacement and age-shifting of disease could potentially result from a vaccination program. Vaccines that are specific for certain pathogen variation (i.e., serotype, serogroup, or strain) may reduce one variation of the disease, but in the presence of multiple variations, the prevalence of infection from non-vaccine variations may still increase.^{Footnote 48} ^{Footnote 49} For diseases caused by multiple variations of a pathogen, researchers should include each variation separately within the model so that infection and disease related to the emergence of new variations can be accounted for. Situations where a vaccination program leads to an increase or decrease in the average age of individuals affected by an infection may lead to a corresponding increase in disease severity, treatment costs, and mortality, which should also be accounted for in a cost-effectiveness analysis.^{Footnote 50} ^{Footnote 51} Examples of dynamic models include dynamic cohort models and individual-based simulation models.

When choosing between a dynamic or static model, researchers should consider the trade-offs between the need to represent transmission, and the additional complexities associated with dynamic models. In some situations, the decision to select one model type over another may not always be straightforward. Dynamic models are conceptually and computationally more complex than static models. Decision-makers who are the end-users of the results generated must be able to understand and interpret the structure of the model. They also need to trust that the results are a reasonable representation of what would be expected to transpire in the real-world setting after the implementation of the vaccination program. There is also a trade-off between the complexity (and realism) of a model, and the ease by which it can be understood, communicated and validated. In some cases, transmissibility between individuals may result in spread of an infection, but the nature of the vaccination program may negate the need to represent transmission in an economic model. For example, for a universal vaccination program that is expected to achieve a high level of coverage in the population, a static model may be adequate in predicting its effects. For further guidance on whether to use a static or dynamic model when estimating the cost-effectiveness of a vaccination program, researchers could consult published schematic diagrams that delineate considerations related to this choice by Jit and Brisson and the World Health Organization (Figure 4, Table 8).^{Footnote 9} ^{Footnote 47}

It should be noted that there are “hybrid” models between dynamic and static models, in which researchers do not fully account for infection transmission. Rather, they estimate the average number of secondary infections averted through the prevention of a case and incorporate the costs and benefits of preventing those cases into the analysis.

Other attributes

Although the fundamental choice facing researchers who are modelling the cost-effectiveness of vaccines is between selecting static versus dynamic modelling techniques, they must also consider other attributes related to the model structure. Considerations related to these attributes are discussed below.

Deterministic versus stochastic

In deterministic models, events depend on pre-specified parameters and model structure; in other words, first-order uncertainty is not accounted for since events cannot occur randomly (by chance). In stochastic models, on the other hand, events are programmed to occur randomly, accounting for first-order uncertainty.^{Footnote 26} ^{Footnote 52} For a discussion of second-order (parameter) uncertainty, researchers should refer to Chapter 14 on Uncertainty.

Average parameter values used in deterministic models may realistically approximate the processes being modeled if the population at risk is large, and the infection is not close to elimination or global eradication (e.g., HPV). For small populations, (e.g., college outbreak of meningococcal B infection), or when modelling the rise of an emerging infection or a rare infection that is on the verge of elimination (e.g., measles and polio in some countries) models that incorporate individual variability and first-order uncertainty (e.g. individual-based models) are more appropriate since they are able to account for random transmission events that are important in these situations.^{Footnote 9} ^{Footnote 28}

Aggregate versus individual-based

In aggregate models (also referred to as population-based or cohort models) such as Markov cohort models and dynamic compartmental models, groups of individuals are aggregated into compartments representing health states based on their characteristics. Changes over time represent shifts in the proportion of the population in each health state based on average parameter values.^{Footnote 26} ^{Footnote 53}

In individual-based models (also called micro-simulations or agent-based models), the individual, rather than the group, is the unit that is modelled. Models that simulate transmission between infected and susceptible individuals are dynamic, in that they have a changing risk of infection over the simulation, whereas those that assume an exogeneous risk of infection independent of the whether there are infected people in the population are static.^{Footnote 26} This type of model is generally more complex and requires more data than a population-based model, and can be programmed stochastically so that an individual’s probability of future events accounts for uncertainty related to randomness.^{Footnote 45}

Individual-based models are also appropriate when there are significant heterogeneities between individuals in a population. These heterogeneities may be related to genetic factors, socioeconomic status, age, access to healthcare services, occupational risk, and behaviour changes in response to disease outbreaks, just to name a few. See Chapter 14 on Equity for more equity-relevant differences. These models may be programmed such that the individuals are able to alter their behaviours over time based on their previous interactions.^{Footnote 52}

Individual-based models are also appropriate when there are significant heterogeneities between individuals in a population. These heterogeneities may be related to genetic factors, socioeconomic status, age, access to healthcare services, and behaviour changes in response to disease outbreaks, just to name a few. See Chapter 14 on Equity for more equity-relevant differences. These types of models account for these characteristics and the effect that they could have on outcomes related to the introduction of a vaccination program.^{Footnote 54}

Population-based models, on the other hand, are appropriate for vaccination programs for relatively homogeneous groups of individuals (e.g., a pneumococcal vaccination program for elderly individuals in one geographic area)^{Footnote 55} since they have similar characteristics that could be reasonably represented by average values as they transition through different health states. Note that population-based models can nonetheless incorporate some heterogeneity through stratifying by risk, and/ or incorporating assortative mixing by age groups and on other risk factors.

When modelling heterogeneities between groups or individuals (including equity-relevant differences), researchers should consider how much detail is required to adequately model the cost-effectiveness of a vaccination program, and consider the trade-offs between different model types that could be used to account for these heterogeneities.

Open versus closed

Models can represent open or closed populations. Open models allow new susceptible individuals, through births and immigration, to enter and exit the model over time, while closed models do not. Although open models may be computationally more complex, they allow researchers to estimate the evolution of the population intended for vaccination and account for its characteristics such as risk exposure, age, and disease severity.^{Footnote 26} ^{Footnote 56}

Open models are generally useful for projecting changes in healthcare costs and treatment outcomes for infectious diseases at different time points over the model time horizon,^{Footnote 57} and should be used when the effects of vaccination programs in one cohort will affect other population cohorts (e.g., childhood immunization programs for diseases such as measles or polio). Closed models are appropriate when examining vaccination programs in small groups of individuals that are unlikely to be epidemiologically influential in the wider population (e.g., hepatitis A vaccination program for healthcare workers), or where the effects of the vaccine are short-lived (e.g., seasonal influenza vaccination program). Note that closed models with long time horizons may undercount potential costs and health benefits.

Discrete versus continuous time

Continuous time models are recommended when multiple events need to be modelled simultaneously. One case may be in disease outbreaks where, for example, transmission of infection between individuals may depend on multiple factors such as contact patterns between individuals, as well as the number of infectious individuals in a given population.^{Footnote 43} Although continuous time models may provide more accurate results in such situations, these models are computationally more complex. They require use of ordinary differential equations for which solutions may be difficult to obtain. Results of continuous models may be approximated by employing discrete time models with a small time steps and appropriately rescaling parameters.^{Footnote 26} ^{Footnote 43}

Model calibration

Model calibration is the process used to infer unknown model parameters by adjusting them to ensure that model outputs represent a good fit with observed data (calibration targets).^{Footnote 58} In infectious disease modelling, many parameters may be unknown or cannot be directly estimated based on available data. These could include parameters related to the natural progression of the infection or disease, details related to sexual behaviours in the case of sexually transmitted infections, and data related to uptake and distribution of results of screening interventions.^{Footnote 59} Calibration targets that are selected should be independent data that are accurately reported with a high degree of both internal and external validity. When appropriate, these data should be stratified by relevant subgroups to ensure adequate model performance across key population strata.^{Footnote 60} Researchers could also consider eliciting expert opinion when selecting calibration targets.

Researchers should be aware that because subjective decisions are required during the calibration process, such as selecting calibration targets, goodness-of-fit measures, and calibration method, there is uncertainty related to the calibration methods that are employed. These uncertainties can lead to considerable differences in results of economic evaluations. Although calibration is often computationally intensive, when possible, researchers should consider using more than one approach for model calibration, and multiple goodness-of-fit statistics.^{Footnote 61} Researchers should retain uncertainty in the calibration estimated parameters, which then can be used in probabilistic analysis.

Difficulty calibrating multiple model parameters may indicate that the model structure or its underlying assumptions are incorrect. It may also suggest a limited understanding of the natural history of the disease being modelled, or of the behaviours that affect its transmissibility, detection, or treatment. Alternatively, it may reveal biases, inconsistency, or imprecision in the data being used as calibration targets. As such, it should not be minimized or ignored, but rather used to help establish future research priorities.^{Footnote 28}

Model validation

Validation is the process that is used to ensure the accuracy of results generated from models used in economic evaluations. The validity of a model should be examined within a relevant decision-making context so that decision-makers are able to determine whether the model under consideration addresses the decision problem at hand.^{Footnote 62} Researchers should assess various aspects of model validity using different methods.

Face validity concerns whether a model reflects the current understanding and evidence related to the disease and vaccination program being considered. It involves the subjective assessment of a model’s structure, assumptions, data sources, and results. This is best conducted by clinical experts in the field, and can also be done by comparing the model structure to accepted clinical disease algorithms. Internal validity is often referred to as verification, and refers to whether the model behaves as it should. It involves verifying that the mathematical equations used in the model have been programmed correctly. It ensures that there are no computational errors in the model. Cross-validation involves comparing the results generated from one model, and determining the extent to which they correspond to results of other models.^{Footnote 63} External validity involves comparing results generated from a model with existing data from independent sources such as clinical trials, epidemiologic studies, routinely available population statistics such as mortality data, or electronic health records. External validation is not possible in situations where the model makes use of all relevant known data. It may be difficult in situations where these types of data do not exist, or when they are not sufficiently detailed to allow appropriate comparison.^{Footnote 64} Predictive validity refers to whether a model is performing its intended purpose, which is to predict outcomes related to a vaccination program. It is also the most difficult type of validation to perform since results must relate to events or studies conducted in the future. This type of validation is usually not applicable to decision-making related to a new vaccination program.^{Footnote 64} However, it may be relevant when developing a model based on older models. Researchers can assess the older models prior to re-use. As with model calibration, researchers could consider eliciting expert opinion when undertaking model validation processes.

9. Effectiveness

9. 1“A comprehensive search of the available data sources should be conducted to inform the estimates of effectiveness and harms associated with the interventions. Report the included studies and methods used to select or combine the data.” [CADTH Guideline Statement]

9.4 Researchers should ensure that immune biomarkers used as surrogate outcomes in studies of vaccine efficacy or effectiveness meet the criteria for correlates of protection.

This chapter details factors that should be considered when assessing the effectiveness of vaccines, and considerations related to data synthesis, interpretation and use of surrogate outcomes, and extrapolation of effectiveness estimates.

Assessing estimates of vaccine effectiveness

There are several factors specific to vaccines that should be considered when interpreting effectiveness data. These factors are discussed below.

Researchers should be aware of differences between efficacy and effectiveness related to vaccines. Efficacy is established through randomized controlled trials (RCTs), which evaluate changes in immune markers, reductions in disease severity, and improvements in health outcomes in vaccinated individuals. Effectiveness of vaccines in individuals is often different from efficacy. For example, there are often higher rates of vaccine series completion in RCTs compared to the real-world setting; there are limitations of the RCT design in capturing community immunity; and there are other differences between RCT populations and the real-world populations in which the vaccine is used.

Vaccine series completion is an important consideration for the many vaccines that require administration of multiple doses at defined time intervals. For example, the HPV vaccine was initially administered on a 3-dose schedule, although a 2-dose schedule is now recommended for some. For the 3-dose schedule, the second dose is given 1–2 months after the first dose, and the third dose 6 months after the first dose.^{Footnote 65} Researchers should be mindful that individuals who do not receive all doses of a recommended vaccine series might experience lower rates of vaccine effectiveness than those who receive the full series. Researchers should assess both clinical trial data and expected real-world dose completion estimates, as both have strengths and limitations. Real-world data may be obtained from acceptability studies on vaccine series completion or from data on completion of other vaccine series used in similar populations with a similar number of doses. Researchers should keep in mind that residual confounding may affect results of observational studies that examine the relationship between dose completion rates and vaccine effectiveness. Specifically, factors that predict for lower probability of dose completion may also increase the underlying risk of infection (e.g., earlier sexual exposure in girls who receive fewer than three doses of HPV vaccine).^{Footnote 13} Researchers should use expected real-world dose completion estimates based on the relevant jurisdiction(s) and intended population for the vaccination program for the reference case analyses.

In terms of community immunity, RCTs may underestimate a vaccine’s population-level effects. That is, community immunity is not observed in RCT participants since they represent a very small proportion of the population. Community immunity is dependent on the distribution of immunity conferred by the vaccine and natural infection within the population, the transmissibility of the infection, and contact patterns of individuals in the population.^{Footnote 66} Population-level effectiveness is usually established through observational studies, which would normally capture the indirect effects of a vaccine. Researchers should be aware, however, that studies using surveillance data are subject to the same limitations as other observational studies, and may not be appropriate to extrapolate to different settings.^{Footnote 28} In such cases, dynamic models parameterised using local epidemiological data can be used to estimate indirect effects of vaccines. When assessing whether to include estimates of vaccine efficacy or effectiveness from RCTs or from observational studies in the reference case analyses, researchers should justify which data sources best represent results in populations most similar to the population(s) affected by the vaccination program to be implemented.

Geographic variation should be considered with respect to vaccine efficacy and effectiveness. Several studies have found that vaccine efficacy and effectiveness can vary by country. Several factors have been postulated to account for these differences, including: 1) differences in serotype or strain prevalence; 2) the role of climate and daily mean temperatures; 3) population heterogeneities with respect to social and demographic factors that influence vaccine efficacy and effectiveness; 4) co-administration of other vaccines (e.g., oral rotavirus and polio vaccines co-administration); and 5) differences in prevalence of other endemic infections.^{Footnote 67} ^{Footnote 68}

Host factors should also be considered when evaluating fitness for purpose of vaccine efficacy and effectiveness data. Examples of such factors include age, genetic susceptibility to infection, inborn errors of immunity, the effect of nutrition on host responses, previous sensitization to organisms antigenically related to the pathogen, comorbidities, particularly those that can affect immune response, secondary immunodeficiencies due to medications, and possible genetic differences in response to a particular vaccine.^{Footnote 69} RCTs tend to include only healthy adults, whereas real-world studies include at-risk populations that would otherwise be excluded from RCTs including pregnant persons, children and those who are immunocompromised.

Vaccination coverage may differ between groups of individuals or by geographic area. For example, diphtheria, pertussis and tetanus (DTaP) vaccine coverage of four or more doses in two-year-old children differs between Canadian provinces, with Newfoundland and Labrador achieving the highest coverage (89%) and Manitoba the lowest (66%), based on the 2017 Canadian Immunization Coverage Survey.^{Footnote 70} Coverage is an important factor in determining effectiveness at the population-level through community immunity. Achieving high levels of vaccination coverage depends on the implementation strategy undertaken when a new vaccination program is introduced, and the ongoing strategies employed to scale up and sustain the program. For instance, health promotion, information campaigns and other efforts to build community trust may counteract vaccine hesitancy. The success of these strategies will depend on the capacity of the resources deployed, ease of access to vaccines doses in the intended population, preparedness of healthcare providers, and attitudes of both healthcare providers and the public. These are all distinct elements related to vaccine coverage, and different levers can be pulled to achieve better outcomes. Researchers should incorporate these factors into economic evaluations to better align these evaluations with decision-makers’ practical needs. Importantly, including these factors focuses decision-makers’ attention on specific implementation strategies, the relative time and effort needed to execute each one, the inherent trade-offs posed by these alternative courses of action, and their independent and joint effects on population coverage.^{Footnote 71}

Some vaccines provide protection only against some variations of a pathogen. For example PCV13 is active against 13 out of over 90 known pneumococcal serotypes,^{Footnote 9} ^{Footnote 72} and 23-valent pneumococcal polysaccharide vaccine (PPSV23) is active against 23 pneumococcal serotypes.^{Footnote 73} HPV vaccines are available in bivalent and quadrivalent forms, although there are over 100 HPV serotypes.^{Footnote 74} For these types of vaccines, researchers should ensure that efficacy and effectiveness data being considered are specific to the diseases caused by the variations of the pathogen targeted by the vaccines. Researchers should also be aware that in some cases vaccines that are specific to certain pathogen variations may also confer some degree of protection against variations of the pathogen not covered by the vaccine. An example of this cross-protection has been shown with both bivalent and quadrivalent HPV vaccines demonstrating some protection against infections and lesions associated with HPV 31, 33, and 45, which are non-vaccine serotypes.^{Footnote 74}

Data synthesis

Researchers should be mindful of vaccine-specific considerations when combining data from different sources. These include potential geographic and host factors outlined above that may be different between study populations and the population being considered in an economic analysis conducted in the Canadian setting.

Surrogate outcomes

Whenever possible, the efficacy or effectiveness of vaccines should be determined with comparative studies (either RCTs or observational studies) that report the incidence of the infectious disease targeted by the vaccine, in the vaccinated group versus relevant comparator(s).

The primary endpoint of these studies should be defined as clinically apparent infection that meets clinical and laboratory diagnostic criteria. In some situations, it may not be possible to measure cases of clinically apparent infection. One example of such a situation arises when the incidence of the infection is too low to measure in a study, typically constrained by its study time period and study population size. This situation occurs with rare infectious diseases (e.g., meningitis due to meningococcal group B infection), or those that rarely afflict the population because current vaccines provide effective prevention.^{Footnote 75} ^{Footnote 76} Another example arises with seasonal influenza vaccines, many of which receive provisional approval based on immunogenicity alone.^{Footnote 77}

In these situations, correlates of protection (CoPs), which are immune biomarkers (antibodies or T-cells) that predict vaccine efficacy in vaccinated individuals, can be used as surrogate endpoints.^{Footnote 75} ^{Footnote 78} ^{Footnote 79} Researchers should be aware that multiple CoPs can exist for a single vaccine,^{Footnote 80} ^{Footnote 81} and that different vaccine types and formulations indicated same disease may be associated with different CoPs.^{Footnote 82} ^{Footnote 83} For multivalent vaccines that provide protection against multiple variations of a pathogen, higher titers of the CoP may be required for protection against some variations compared to others.^{Footnote 84} Finally, it is important for researchers to identify which dimension of prevention (e.g., preventing infection, preventing disease, reducing severity of disease) is linked to a CoP since correlates may differ quantitatively and qualitatively based on the preventive outcome being considered.^{Footnote 85}

Extrapolation

The duration of clinical trials is often not long enough to ascertain the duration of protection provided by a vaccine, and researchers have to extrapolate estimates of duration of protection from clinical trial data.^{Footnote 86} ^{Footnote 87} A number of different modelling techniques (e.g., logarithmic waning, exponential waning) can be used to generate duration of protection estimates, which can vary widely based on the technique chosen. Consequently, cost-effectiveness estimates can be sensitive to assumptions on duration of protection.^{Footnote 86} This has been demonstrated with cost-effectiveness analyses of herpes zoster vaccine (Zostavax®) in Belgium, where the authors found that cost-effectiveness estimates varied considerably based on the choice of model used estimate to vaccine efficacy.^{Footnote 31} Specific guidance on addressing uncertainty of the estimates of duration of protection is provided in Chapter 13 on Uncertainty.

10. Measurement and valuation of health

10.1 In both reference cases, the quality-adjusted life year (QALY) should be used as the method for valuing health outcomes.

10.2 “Health preferences should reflect the general Canadian population.” [CADTH Guideline Statement]

10. 3 In the reference cases, researchers should use health preferences obtained from an indirect method of measurement that is based on a generic classification system (e.g., EuroQol 5-Dimensions questionnaire [EQ-5D], Health Utilities Index [HUI], Short Form 6-Dimensions [SF-6D], Child Health Utility 9-Dimensions [CHU9D], Assessment of Quality of Life [AQoL]). Researchers must justify where an indirect method is not used. [CADTH Guideline Statement with amendment]

QALYs are the metric used to quantify health outcomes in a CUA. QALY estimates are generated by combining data on survival and health-related quality of life (HRQoL). In order to estimate QALYs, HRQoL data in the form of a summary measure, often referred to as a health utility, is required. As the CUA implicitly espouses an extra-welfarist foundation, decision-makers are concerned with HRQoL because the key output of health interventions is health outcomes.

Health utility data

The utilities obtained from HRQoL instruments should represent the preferences of the general Canadian population, consistent with the social decision-making standpoint adopted by these guidelines. Population preferences for health states defined in an HRQoL instrument are normally elicited from a sample of the general population using methods such as standard gamble or time trade-off.

Although it is possible to elicit health utilities directly from respondents, instruments designed to capture health utilities indirectly provide a more efficient and consistent method of obtaining this information. Both disease-specific and generic HRQoL instruments are available to obtain indirect health utility measurements. The most commonly used generic instruments are the EuroQol 5-Dimensions questionnaire (EQ-5D), Health Utilities Index (HUI), the Short Form 6-Dimensions (SF-6D), and the Assessment of Quality of Life (AQoL). Instruments for children’s HRQoL include Child Health Utility 9-Dimensions (CHU9D), KIDSCREEN Qality of Life Questionnaire, Pediatric Quality of Life Inventory (PedsQL) Generic Cores Scales, and EQ-5D-Youth (EQ-5D-Y). Researchers should use HRQoL data obtained from a generic instrument to estimate QALYs to ensure comparability between vaccination programs being considered by decision-makers. Where multiple estimates of utilities are available, source studies should be subjected to formal quality appraisal using a suitable quality appraisal tool.^{Footnote 88}

Health utility data used to populate an economic model are often derived from published literature. To ensure consistency within a model, health utility valuations for all health states included in the model should be obtained from the same instrument and use preference weights obtained from the same population, whenever possible.^{Footnote 89} When this is not possible, researchers should consider trade-offs between the fitness for purpose, credibility, and consistency for the available data. In these cases, researchers may also consider pooling health utility data using techniques such as meta-analysis or meta-regression, although the usefulness of these methods may be limited by the considerable heterogeneity in the valuation methods and study populations.^{Footnote 90} Researchers should explore uncertainty in health utilities in sensitivity analyses.

It must be recognized that there are no valid instruments for directly measuring utility in neonates, newborns, infants or young children, although this is an active area of current research.^{Footnote 91} Moreover, the construct of HRQOL for children differs by age group and is conceptually different than adults.^{Footnote 92} While several pediatric-specific preference-based measures of health-related quality of life have been developed recently (e.g., EQ-5D-Y, CHU-9D, A-QOL), all have lower age limits and typically rely on tariff sets derived from adult populations. The convergent validity of pediatric-specific and adult preference-based HRQOL measures requires study. Despite the limitations, researchers should ideally use utilities for child health states sourced from a pediatric-specific generic instrument, as opposed to using adult utilities. If a pediatric-specific generic instrument is not used for a child health state, this should be justified and its impact tested in sensitivity analysis. The use of generic instruments is encouraged in pediatrics, despite direct elicitation methods being frequently used. Utilities generated from direct elicitation for health states are sensitive to framing. In cases where utilities may be missing due to a child’s young age (e.g., under 5 years), assumptions used should be explicit and justified. Preferences should be from a general population, supplemented with child valuations if available. Proxy respondents (e.g., by parents or healthcare providers) are often required in pediatrics because valuation methods can be cognitively difficult or require reading comprehension. However, proxy responses can systematically differ from child self-report where the directionality of the discrepancy is difficult to predict.^{Footnote 93} Researchers should use child utilities from instruments that are self-reported where possible, and specify if proxies are used. Further, many vaccines are given in infancy or childhood, some of which prevent diseases in childhood and others in diseases that emerge in adulthood. Researchers should explicitly state which health states in a model are related to child health states and which relate to future adult health states. In economic evaluations where adult and children are modelled, consistency in the use of instrument across ages is encouraged.

In addition to including health utility data for the population intended for the vaccination program and any population(s) that may experience externalities related to the program, researchers should include health utility data for informal caregivers in cases where potential spillover effects have been identified that could affect the health states of this population.A more detailed discussion on HRQoL measurement and data can be found in Chapter 10 Measurement of Valuation of Health of CADTH’s Guidelines for the Economic Evaluation of Health Technologies: Canada 4th Edition.^{Footnote 1}

Quality-adjusted life years in societal perspective economic evaluations

There is uncertainty about whether QALYs capture only health benefits, or whether they also, implicitly or explicitly, capture non-health-related effects. This uncertainty is particularly germane to CUAs conducted from the societal perspective since these analyses are concerned with not only costs and outcomes borne by the health system, but also with costs and outcomes that fall onto non-health sectors. Specifically, uncertainties exist around how to include the impacts of productivity and consumption in the ICER estimate.

As noted above, QALYs are estimated using survival and HRQoL data. The HRQoL data are elicited based often on health domains included in common HRQoL instruments. However, it is unclear whether, or to what extent, respondents implicitly consider non-health-related factors when valuing these health states. In particular, the degree to which respondents consider how changes in productivity and consumption may affect their HRQoL, and the extent to which these considerations are implicitly incorporated into respondents’ valuation of their health states have received attention. The available evidence suggests their influence is limited.^{Footnote 94} ^{Footnote 95} ^{Footnote 96} ^{Footnote 97}

If individuals were to account for the economic effects of productivity when valuing their health states, then including monetary estimates of productivity in the numerator of the ICER estimate, along with costs of other resources, double counts these impacts.^{Footnote 96} The current consensus is that productivity and income changes are not likely to be captured in QALY estimates.^{Footnote 98} ^{Footnote 99} This supports the inclusion of productivity costs in the numerator of the ICER estimate.

Similarly, questions have been posed about whether respondents in health state valuations consider and value non-medical consumption such as clothing and housing. One argument posits if the utility value of this consumption is not (implicitly) captured in QALYs, it would be inconsistent to include changes in such consumption on the costs side of the evaluation and therefore, these costs should be excluded.^{Footnote 97} Another argument posits that non-medical consumption (e.g., daily food intake) is necessary to stay alive. Even if respondents would not consider this in their valuations of health states, it should nonetheless be included as a cost. This would be even more evident if respondents would assume usual levels of consumption in answering health state valuation questions. The same argument applies to other non-medical consumption, which to some extent may also contribute to an individual’s HRQoL.^{Footnote 95}

In contrast to findings related to respondents’ consideration of productivity changes when valuing health states, evidence suggests that respondents do consider utility of consumption when valuing health states.^{Footnote 100} This suggests that health improvements may lead to increases in the marginal utility of non-health-related consumption. Although further research is required to corroborate these findings, they do provide justification for including consumption costs in the numerator of the ICER estimate.

In summary, for the societal perspective reference case analysis, changes related to productivity and costs of non-health-related consumption should be included in the numerator of the ICER estimate. Further details on quantifying the costs of productivity and consumption are found in Chapter 11 on Resource Use and Costs.

11. Resource use and costs

11.3 “Resource use and costs should be based on Canadian sources and reflect the jurisdiction(s) of interest (as specified in the decision problem).” [CADTH Guideline Statement]

11.4 When valuing and monetizing resources, researchers should select cost data sources that most closely reflect the opportunity cost, given the perspective of the analysis. [CADTH Guideline Statement with amendment]

Both increases and decreases in consumption of resources and services may result from vaccination programs. They are related to both the implementation of the vaccination program and its ongoing delivery, as well as to downstream effects of the program. Resource consumption may fall upon vaccinated individuals, the population at risk for the disease of interest when the vaccination program is associated with externalities, and the population that experiences spillover effects (e.g., informal caregivers). Furthermore, resources consumed as a result of vaccination programs may fall within the health system sector or outside the health system. Researchers should use the Impact inventory table for economic evaluations of vaccination strategies to systematically identify all potential resources and services associated with the vaccination program under consideration. Once the range of resources and services occurring as a result of a vaccination program has been identified, researchers should determine which of the resources consumed can be measured and valued in monetary terms.^{Footnote 101} ^{Footnote 102}

Health system sector

For the health system perspective reference case analysis, researchers should identify and include all resources within the publicly funded health system that are consumed through the delivery of the vaccination program, and resources that are consumed or saved as a result of its implementation. All health system costs incurred over the model time horizon should be included.

Healthcare costs

When assigning local unit costs to resources that have been identified as relevant to the decision problem being modelled, researchers should consult the Guidance Document for the Costing of Healthcare Resources in the Canadian Setting: Second Edition,^{Footnote 103} which provides key recommendations and data sources for identifying, valuing, and measuring costs within the Canadian healthcare system. For analyses that are conducted within or across multiple jurisdictions, variations in the public financing of specific resources and services should be indicated. Researchers should report whether a single price set is used or multiple jurisdictional price sets set are applied and methods used for assigning price sets to multi-jurisdictional data.

Formal caregiving

Individuals requiring a caregiver may receive this care from either a professional caregiver such as a nurse or a nursing assistant who is hired to perform these services, or an informal caregiver, usually a family member. Professional caregiver time should be valued at the hourly wage rate that would be paid to an individual who performs this service. Formal caregiving costs may be incurred or may not be incurred by the publicly funded health system, depending on the precise nature of these costs and the relevant jurisdiction. Informal caregiving is discussed later in the chapter under Productivity.

Future healthcare costs

For vaccination programs that confer a survival benefit in relation to the comparator(s) being considered in the economic evaluation, researchers should include in the reference case analyses future healthcare costs, both related to the infection and disease of interest and unrelated. This recommendation is underpinned by the following considerations: 1) there is an opportunity cost associated with life-prolonging interventions since they increase future health expenditures in those individuals— expenditures which could have been used towards other individuals’ healthcare needs; 2) it is often difficult to distinguish related costs from unrelated costs, such as in the case of different diseases that share overlapping physiologic pathways (e.g., diabetes and coronary heart disease), potentially leading to arbitrary decisions about which costs are related versus unrelated; and 3) internal consistency: the benefits related to future medical spending are already included in CUAs through estimates of survival and quality of life, and are based on the assumption that the individual will receive future medical care, both related and unrelated to the condition of interest.^{Footnote 95} ^{Footnote 101} ^{Footnote 104} ^{Footnote 105}

Excluding future costs leads to lower incremental cost estimates and ICER estimates for life-prolonging interventions, and may make them seem more economically attractive than those that improve quality-of-life. Including future costs, however, increases incremental cost estimates and ICERs for life-prolonging interventions, leading in some cases to a do-nothing option (i.e., where patients do not survive) being more cost-effective than providing treatment for a disease.^{Footnote 106} In some cases even relatively inexpensive life-prolonging interventions in patients with high costs of ongoing care may not be cost-effective when future costs are considered in an economic evaluation.^{Footnote 107} Researchers should present outcomes and costs in a disaggregated manner so that decision-makers are aware of how individual components included in the analysis contribute to the overall cost-effectiveness of the vaccination program. Researchers should present: 1) the expected health outcomes of the vaccination program and the comparator(s); 2) the direct health system costs resulting from the vaccination program and the comparator(s) but excluding costs of future care; and 3) the expected increase in costs of ongoing care resulting from improved survival for the vaccination program and comparator(s).^{Footnote 107}

Estimates of future healthcare costs may be obtained from data published by the Canadian Institute for Health Information’s National Health Expenditure Trends.^{Footnote 108} In situations where cost estimates are required for populations with high costs of ongoing care (e.g., dialysis patients, solid organ transplant recipients).^{Footnote 109} ^{Footnote 110} researchers may have to consult the published medical literature to obtain these estimates.

Public health costs

Public Health costs may represent a large share of the costs associated with vaccination programs, and management of infectious diseases. Accurately quantifying these costs is necessary to ensure that results generated from economic evaluations of vaccination programs are valid, and lead to optimal funding decisions. Public Health costs can be categorized as either program-related costs or intervention-related costs. Program-related costs are the costs of program implementation, delivery and sustainment costs. They include costs of public health campaigns and health promotion activities; transaction costs related to introduction of new vaccines or switching between vaccines; and costs related to population-based screening, epidemiological surveillance, contact tracing, case investigations, and outbreak investigations. Specific components that should be considered when quantifying these costs include personnel costs, overhead costs, travel costs, and other service-related and administrative costs.^{Footnote 103} ^{Footnote 111} Specific components that should be considered when quantifying costs of disease outbreaks include laboratory serologic testing; personnel time related to contact tracing, symptom screening, travel, monitoring, and follow-up; post-exposure prophylactic vaccines or immune globulin doses and associated administration costs.^{Footnote 112} ^{Footnote 113} ^{Footnote 114} ^{Footnote 115} Intervention-related costs include costs of vaccine doses, distribution (e.g., transportation of vaccines and cold storage), and administration of the vaccine, including any wastage and ancillary supplies required. Researchers should present costs related to different aspects of implementation and ongoing delivery of the vaccination program in a disaggregated manner. Further, researchers should elaborate on the different levels of intensity of the implementation strategy, which is especially relevant for public health campaigns and health promotion activities for instance, as they can produce different levels of benefit.

Given the paucity of published data on program-related Public Health resource use and prices in the Canadian setting, researchers may have to rely on data obtained from local Public Health authorities or provincial ministries of health through personal communication. Although costs from local Public Health authorities and provincial ministries are jurisdiction-specific, they may be generalizable to other areas. When determining the applicability of data from one jurisdiction to another, researchers should consider factors such as geographic similarities, population characteristics, and epidemiologic patterns.

There are limited Canadian data available on intervention-based Public Health resource use and prices. Some provincial Public Health agencies, such as Quebec’s Institut national de santé publique du Quebec,^{Footnote 116} publish findings of their work online, which may include epidemiologic surveillance and cost data relevant to the economic evaluation of a vaccination program. If the required data are not available through publications from provincial Public Health agencies, researchers may have to obtain these data from provincial ministries of health or local Public Health authorities. The actual price paid by governments for vaccine doses is confidential. Researchers should use the manufacturer’s list price in the reference case analyses and conduct deterministic sensitivity analyses using plausible discounted prices. Researchers should also consider cost items related to the administration of vaccine doses, as they may vary considerably based on the setting of delivery. For example in Alberta, community-based delivery of HPV vaccine is considerably more expensive than school-based delivery.^{Footnote 117} Resources and services related to providing culturally safe access to healthcare and vaccine program communication materials should also be considered in situations where they are applicable.

Healthcare costs not funded by the health system

Some services associated with vaccination programs may not be reimbursed or publicly funded by the healthcare system. Services that are excluded from the publicly funded healthcare system may vary by jurisdiction or region. Examples of such costs include long-term care services, private nursing, drug treatments for individuals who do not have coverage through a publicly funded drug insurance program, non-prescription drugs, as well as ancillary costs related to items such as private insurance copayments, dental and vision care, assistive devices, physiotherapy and others. These costs may be funded through private insurance plans, by the individual(s), or a combination of both. Regardless of how these costs are funded, they should be quantified and included in the incremental costs and ICER (where applicable) for the societal perspective reference case analysis.

Non-healthcare areas

Researchers should also identify all resources consumed as a result of the implementation or ongoing delivery of the vaccination program that fall outside of the publicly funded health system and quantify their corresponding costs. For example, relevant non-health sectors for the societal perspective reference case analysis could include: direct out-of-pocket costs (e.g., co-payments, transportation costs, private caregivers), paid and unpaid labour time losses, non-medical consumption, and services not funded by other sectors including education, social services, and environment. Guidance on identifying resources and quantifying costs for non-healthcare sectors is presented below.

Direct out-of-pocket costs

Estimates of direct out-of-pocket costs (e.g., transportation costs, accommodation costs) should be included in economic evaluations of vaccination programs. Transportation costs include costs related to public transit, including fully accessible barrier-free transportation when necessary, taxis, personal vehicle use, and parking fees.^{Footnote 118}

Productivity

Researchers should consider the effects of vaccination programs on the productivity of vaccinated individuals and caregivers, and where applicable, on macroeconomic consequences. For the former, vaccine-related productivity improvements may occur through: 1) increased paid and unpaid labour productivity related to either prevention of illness, or decreased severity of illness in vaccinated individuals; and 2) increased productivity of caregivers related to decreased care needs for sick individuals.^{Footnote 7} ^{Footnote 25} ^{Footnote 119} When productivity gains for life-prolonging interventions are included in an analysis from the societal perspective, they may attenuate or offset increased incremental costs due to increased future healthcare consumption in survivors.

Individual productivity

Productivity costs are output losses associated with productive time spent in paid labour, or unpaid labour (e.g., volunteering, helping, mentoring) including household production (e.g., cooking, cleaning, shopping, raising children).^{Footnote 101}

There are two primary methods for quantifying lost productivity related to paid work: the human capital approach, and the friction cost approach. The human capital approach is based on the cost of forgone productive time, whereas the friction cost approach attempts to estimate overall societal production losses, assuming replacement of ill workers in the formal labour market.^{Footnote 101} ^{Footnote 120} ^{Footnote 121} ^{Footnote 122}

The human capital approach is commonly used to value lost production. It typically requires estimates of time lost from paid work, and averages wage rates of the involved individuals. As such, it may be seen as estimating the lost production (or income) from an individual perspective, due to illness, disability, or death. As the human capital method does not account for societal replacement mechanisms, especially for longer periods of absence (e.g., in case of disability or premature death), it has been suggested that it likely overestimates the true cost of lost production from a societal perspective.^{Footnote 123} This is a particularly important consideration in situations where childhood death or lifelong disability may be avoided as a result of a vaccination program.

The friction cost approach, on the other hand, attempts to quantify lost productivity on a societal level based on the assumption that production levels can be restored by substituting labour for labour (e.g., in case of unemployment) or for capital.^{Footnote 120} This implies that after some ‘friction period’ production losses cease to occur from a societal perspective. Macro-economic consequences of changes in labour supply and unemployment benefits have been estimated to be small for typical health care programs. Applying this method requires more detailed information on periods of absence, the available labour pool, and the relevant friction period in a country or province.^{Footnote 122}

While both of these methods primarily focus on valuing lost production in the context of paid work, changes in productivity related to unpaid work should also be captured. Lost productivity in the context of unpaid work can be captured by valuing lost hours with an appropriate value. Estimations of (changes in) productive time in unpaid work for the relevant population may be difficult to obtain in some cases.^{Footnote 124} Other than using general estimates from existing sources, questionnaires may be used to estimate these changes.^{Footnote 125}

Researchers should calculate total change in productive time, related to both paid and unpaid work, attributable to the vaccination program. Researchers should account for losses of an individual’s productive time related to obtaining a vaccine, seeking treatment, illness, disability, and death of vaccinated or otherwise affected individuals. Changes in productivity associated with vaccination programs should be quantified using the human capital approach. Given that it is the most commonly recommended approach in pharmacoeconomic guidelines across different countries,^{Footnote 126} it allows increased comparability between economic evaluations of vaccination programs undertaken in different jurisdictions.

For the societal reference case analysis, researchers should include the full-time period over which affected individuals are expected to incur paid production losses. These losses should be valued based on age-specific average income and number of hours worked based on Statistics Canada data^{Footnote 127} ^{Footnote 128} combined with the disease-specific likelihood of an individual participating in the labour force. Using the same wage rate for both genders is a correction for measurement bias because females are on average paid less than male for the same work.^{Footnote 129}

In most cases, there will be equity considerations related to whether and how productive time is valued. If it is differentially valued based on attributes such as age, gender, or health status, results of an economic evaluation could favour groups with the greatest income-earning potential and disadvantage other groups such as children who do not work or individuals with disabilities or severe health conditions that prevent them from holding high-income jobs.^{Footnote 101} In these situations, researchers should conduct an additional sensitivity analysis using the average income and the average number of full-time hours worked for all Canadians based on Statistics Canada data.^{Footnote 127} ^{Footnote 128} Although the measurement of these losses is imperfect and biased towards high-wage earners, this approach reveals the efficiency losses that decision makers need to be prepared to accept each time they choose an option that is neutral to individual characteristics with respect to production.

To account for the likely overestimation of production losses associated with the human capital approach, researchers should include an additional sensitivity analysis that accounts for production losses for a single year using the average number of full-time hours worked for all Canadians based on Statistics Canada data.^{Footnote 127} ^{Footnote 128} Average yearly income and average yearly number of hours worked for all Canadians should be used for this analysis. This approach represents a naïve friction cost approximation.

Although productivity losses may result from both absenteeism (time off work) and presenteeism (continuing to work but with reduced productivity), researchers are not required to account for the effects of presenteeism in their estimates of lost productivity in the reference case analysis. It is often difficult to collect this information given that it requires survey data from affected individuals and recall can be subjective in many cases.^{Footnote 125} ^{Footnote 130}

Lost unpaid production should be valued by estimating lost hours of unpaid work, and valuing this using the replacement cost method. Although unpaid work may differ in terms of tasks performed and required skills, for the reference case analysis, lost hours should be valued using the wage-rate of a professional. Researchers should exclude costs of leisure time from the economic evaluation of vaccination programs.

Informal caregiver productivity

As described above, individuals requiring a caregiver may receive this care from either a professional caregiver, or an informal caregiver, usually a family member. Two approaches have been proposed for valuing informal caregiver time: 1) the replacement cost approach; and 2) the opportunity cost approach. The replacement cost approach is based on the estimated cost of hiring a paid caregiver should informal care not be available. The opportunity cost approach is based on the cost of displaced productive time that results from time spent providing informal care.^{Footnote 131} Since individuals may receive a mix of formal and informal care, researchers should use the replacement cost approach to value caregiver time for the societal perspective reference case analysis. Such estimates can be used alongside estimates of potential health spillover effects due to informal care, captured in terms of caregiver QALYs.^{Footnote 132}

Macroeconomic consequences

Although most vaccination programs are unlikely to have large macroeconomic impacts, those that are designed to prevent widespread disease pandemics, such as the 2020 COVID-19 pandemic caused by the SARS-CoV-2 virus, could attenuate important consequences. Macroeconomic impacts include labour supply shocks and widespread business closures, which may affect labour pools and workforce participation rates, and changes in household consumption preferences.^{Footnote 133}

Non-Medical consumption

Non-medical consumption represents expenditure on non-health-related items that contribute to overall welfare. These items include individual financial expenditures, and consumption of public goods and services such as clean water and safe roads.^{Footnote 95} ^{Footnote 101} Researchers should include consumption costs whenever they will be altered by the vaccination program.

Researchers should use Statistics Canada data on household spending as the information source for non-medical consumption (Table: 11-10-0222-01, formerly CANSIM 203-0021, “Household spending, Canada, regions and provinces”).^{Footnote 134} In order to obtain an estimate of non-medical consumption, researchers should exclude health consumption from total consumption. Estimates of individual consumption should be obtained by adjusting household consumption estimates using an equivalence scale, to account for consumption by household size, reflecting the fact that one-person households would have higher per-person consumption compared to multi-person households.^{Footnote 135} For vaccination programs that result in changes to consumption, researchers should subtract individual estimates of consumption from individual estimates of productivity during the relevant time period. To ensure consistency between estimates of productivity and consumption, estimates of consumption should not be stratified by gender for the reference case analysis.

Education

Vaccination programs may affect educational outcomes by preventing diseases that lead to serious morbidities that, in turn, could affect an individual’s level of educational achievement. For example, a Danish study found that children who suffered from bacterial meningitis experienced lower levels of educational achievement and economic self-sufficiency in adulthood.^{Footnote 136}

Higher levels of educational achievement are associated with a greater likelihood of labour market participation and higher labour market earnings.^{Footnote 137} ^{Footnote 138} In Canada, it is estimated that each additional year of schooling increases lifetime earnings by approximately 11 to 12%. Assuming that decreasing an individual’s education level similarly decreases lifetime earnings, it is estimated that each month of education loss will result in an approximately 1% drop in lifetime earnings.^{Footnote 139} Changes in earnings related to education achievement should be accounted for in estimates of lost (or gained) productivity, and researchers should ensure that these costs are not double counted when considering educational impacts of vaccination programs.

In addition to effects on educational achievement and labour market productivity, vaccination programs may result in direct effects on the education sector. For example, children who have suffered from bacterial meningitis may experience cognitive impairment, hearing loss, seizures, and learning disabilities,^{Footnote 140} and may require in-school special education resources. Boards of education and schools may also invest in vaccination delivery programs, as well as ancillary programs to improve the learning environment during a pandemic (e.g., upgraded heating, ventilation, and air conditioning, reduced classroom size, virtual learning).

Researchers should consider potential education-related outcomes and direct effects on the education sector that could result from the vaccination program and the comparator(s) being considered. Where possible, these effects should be monetized for inclusion in the ICER estimate. For outcomes that may be difficult to monetize (such as disruptions to learning outcomes as a result of school-based vaccine delivery, paediatric disease and disability, or death/disability of a close family member), researchers should nonetheless identify them and include in them in the Impact inventory table for economic evaluations of vaccination strategies for consideration by decision-makers.

Social services

Vaccination programs may affect social services, community services, and child and youth services by preventing diseases that lead to serious morbidities. Examples include disability support, awareness programs, family respite, and programs to improve access to vaccination programs. Researchers should identify (and where feasible monetize) consequences of social services.

Environment

Vaccination programs and the comparator(s) included in the analysis may have environmental impacts related to the manufacture or distribution of vaccine doses, as well as to implementation the program. For example, vaccines have been shown to reduce antibiotic use,^{Footnote 141} ^{Footnote 142} which may lead to decreased residual antibiotics from sources such as households, the pharmaceutical industry, and hospitals in wastewater, which has been identified as a reservoir of antibiotic resistant organisms.^{Footnote 143}

Environmental impacts may should be identified and included in the Impact inventory table for economic evaluations of vaccination strategies for consideration by decision-makers. They should be monetized where possible, although this is sometimes difficult to do.

Other areas

Researchers should consider other sectors that may offer services or programs relevant to specific vaccination programs. Examples of such sectors could include the legal/criminal justice sector (e.g., the cost to the state of handling potential lawsuits against vaccine manufacturers resulting from adverse effects of vaccines, or the cost to the healthcare system of lawsuits from patients if a vaccine is not introduced), or the housing sector (e.g., changes in type of housing or adaptations to housing required because of functional disabilities resulting from infection, or to improve ventilation/reduce crowding to reduce infection transmission).

12. Analysis

12.1 Incremental cost-effectiveness ratios (ICERs) and, where useful for interpretation, net monetary benefits or net health benefits, should be calculated for both reference case analyses.

12.3 The expected values of costs and outcomes, where possible, should be generated probabilistically to reflect the overall uncertainty in the model parameters.

Researchers should generate two sets of estimates of expected values for costs related to each intervention considered in the economic evaluation: one for the publicly funded health system perspective reference case analysis, and the other for the societal perspective reference case analysis. One estimate of expected values for outcomes (i.e., QALYs) should be generated for use in both reference case analyses. These estimates, where possible, should be generated probabilistically so that the expected values reflect the overall uncertainty in the model parameters. In most cases, the probabilistic analysis will take the form of a Monte Carlo simulation, where an appropriate point estimate, range, and probability distribution are applied to each parameter. Each simulation should produce estimates for mean costs and mean effectiveness for each comparator, as well as estimates of incremental costs and incremental effectiveness. All values, including incremental estimates, must be reported with 95% confidence or credible intervals as indicators of precision. These intervals can be obtained from the 2.5% and 97.5% bounds from the generated simulations. Additional indicators of precision may also be appropriate if the distribution of uncertain outcomes is not approximately Gaussian. In cases where probabilistic analyses are not possible, estimates of these values should be generated deterministically. This scenario is most likely to occur when the computational power required for a probabilistic analysis is a limiting factor, especially for agent-based models.

For the publicly funded health system reference case analysis, the following costs and outcomes should be incorporated: all costs directly borne by the publicly funded health system in Canada, and QALYs that accrue to individuals who are vaccinated, individuals who experience externalities related to the vaccination program, as well as QALYs that accrue to informal caregivers. For the societal perspective reference case analysis, costs and outcomes from the publicly funded health system perspective should be included, along with the following, at minimum: patient-borne costs, caregiver costs, and productivity costs. Non-health impacts, such as consumption, social services, education, and environment, should also be included when relevant.

Depending on the position of scenarios in the cost-effectiveness plane, it may not be sensible to calculate ICERs, such as in the case of dominance of the vaccination strategy or the alternative care comparator. In all cases, however, mean values for costs, effectiveness, incremental costs and incremental effectiveness should be reported with 95% confidence or credible intervals. When the mean incremental values for costs and effectiveness are both positive, ICERs should be presented (i.e., the ratio of the difference in expected costs to the difference in expected outcomes for the two interventions being compared). Specifically, when two interventions are being compared, there should be an ICER for each reference case perspective. Where there are more than two interventions being considered in the analysis, sequential ICERs should be presented. This approach involves comparing each intervention to the next most costly intervention, and excluding all interventions that are either dominated or subject to extended dominance. Graphically, results should be presented as health production functions or cost-effectiveness efficacy frontiers.

In cases where subgroup analyses have been conducted, expected values for costs and outcomes as well as ICERs should be generated for each relevant subgroup in accordance with the guidance presented in this chapter. In cases where multiple regional or provincial/ territorial publicly funded health system perspectives have been analyzed, findings should be reported for each one.

13. Uncertainty

13.1 Researchers should address parameter uncertainty using a probabilistic reference case analysis, where possible, as well as deterministic sensitivity analyses.

13.2 “Methodological uncertainty should be explored by comparing the reference case results to those from a non-reference case analysis that deviates from the recommended methods in order to examine the impact of methodological differences.” [CADTH Guideline Statement]

13.3 Cost-effectiveness acceptability curves (CEACs) and cost-effectiveness acceptability frontiers (CEAFs) should be used to represent the uncertainty in the estimates of costs and outcomes when these estimates have been generated probabilistically. [CADTH Guideline Statement with amendment]

13.4 When the decision problem includes the option of commissioning or conducting future research, value-of-information analysis may be helpful to characterize the value of these options and design future research and may be included in the reference case analysis. [CADTH Guideline Statement with amendment]

13.5 Scenario analyses should be used to assess structural uncertainty. [CADTH Guideline Statement with amendment]

Decision-makers need information about uncertainty related to the results of economic evaluations of vaccination programs in order to avoid making suboptimal funding decisions. Specifically, three types of uncertainty should be examined and reported: parameter, structural, and methodological.

Parameter uncertainty

Parameter uncertainty, also called second-order uncertainty, refers to uncertainty in parameter estimates that are used to populate a model.^{Footnote 46} ^{Footnote 144} ^{Footnote 145} This differs from random variability, also called first-order uncertainty or stochastic uncertainty, as well as from heterogeneity. Most guidelines on conducting economic evaluations of healthcare interventions recommend using probabilistic reference case analysis, and/ or probabilistic sensitivity analysis (PSA) to explore parameter uncertainty, but in rare situations, this technique may not be feasible with dynamic models. Such situations arise when models are particularly complex (e.g., agent-based simulations), or when only limited computing power is available. In dynamic models, many parameters related to transmission, such as contact patterns between individuals and prevention-related behaviours, may be correlated and these correlations must be preserved in the models to generate sensible results that fit to existing data (e.g., epidemiologic surveillance data). In some cases, correlations between parameters may be unknown,^{Footnote 28} ^{Footnote 52} although they can sometimes be established using Bayesian parameter inference methods.^{Footnote 146} ^{Footnote 147} In these instances, researchers may be required to choose between a complex model structure that does not allow for probabilistic analysis, and a simpler structure that allows exploration of the impact of parameter uncertainty.

Where feasible, parameter uncertainty should be addressed probabilistically through probabilistic reference case analyses. Parameter ranges chosen to assess uncertainty should, where possible, be based on estimates from observational studies or surveillance data. Results of these analyses should be presented as cost-effectiveness acceptability curves (CEACs) or cost-effectiveness acceptability frontiers (CEAFs). Scatter plots on the cost-effectiveness plane may be provided alongside CEACs and CEAFs. Scatter plots are useful to observe the density and spread of the iterations, and to assess inflection points and the shape of the ellipses produced.

In addition to quantifying uncertainty probabilistically in the reference cases, researchers should conduct deterministic sensitivity analyses (DSA) on individual model parameters to gain insight into the isolated effects of variations in these parameters that is provided by deterministic methods. In particular, researchers should conduct a DSA on vaccine price using a number of plausible values since the actual unit price of vaccine doses in Canada is often confidential. DSAs should also be conducted on estimates of vaccine effectiveness as there is often a high degree of uncertainty in these parameters. Ideally, the DSA should be based on the output of the probabilistic analysis rather than assuming base case values (e.g., using partial rank correlation coefficients or linear regression).

Researchers should consider conducting threshold analyses on the most uncertain parameters that may not be based upon empirical evidence (e.g., implementation parameters such as population coverage), so that decision makers are able to ascertain ranges of parameter values that result in a cost-effective vaccination program.

Researchers can present results of one-way (or univariate) DSAs using a tornado diagram, and of two-way DSAs using two-way threshold graphs.^{Footnote 145}

When conducting DSAs researchers should identify parameter regions associated with distinct model behaviours such as epidemic spread or extinction of disease, and state whether the sensitivity analysis has been confined to a single region. If the sensitivity analysis spans more than one region, researchers should indicate the probability of achieving different disease equilibrium states as the parameter values vary.^{Footnote 28}

When probabilistic analyses are not undertaken in the context of non-linear dynamic models, researchers should conduct comprehensive DSAs on uncertain parameters. In these situations, researchers may consider using novel DSA methods such as stepwise DSA and distributional DSA.^{Footnote 148}

The effects of parameter uncertainty may be particularly pronounced in dynamic models compared to static models because of non-linearity in these models that can lead to more variable population outcomes model behaviour in different parameter regions. For example, a small change in parameter values may alter the model state from a disease-free state to a state of endemic equilibrium when the basic reproduction number (R₀) is near a value of 1. These model behaviours have implications for the effectiveness of vaccination programs. If the program is introduced near a threshold state (e.g., beginning of an epidemic), its indirect effects may be substantial compared to a program introduced at disease equilibrium where its effectiveness may exhibit a linear relationship between the number of individuals vaccinated and prevention of the disease of interest.^{Footnote 28}

It can often be difficult to obtain accurate parameter estimates for infectious disease models since researchers frequently must rely on observational studies or surveillance data.^{Footnote 28} Parameter values derived from surveillance data may be biased because the proportion of cases detected is often low and varies considerably between different diseases, even for infectious diseases that are reportable as part of Public Health surveillance requirements.^{Footnote 149} Severity of the infectious disease impacts detection. For example, infection with pertussis may be asymptomatic, associated with mild symptoms, or severe coughing or even death.^{Footnote 150} Thus, surveillance systems that rely upon passive reporting often overestimate disease severity, morbidity, and mortality, while underestimating the true incidence of infection in the population.^{Footnote 28} ^{Footnote 150}

Uncertainty in estimates of vaccine effectiveness may arise from differences between data obtained from RCTs compared to that obtained from large observational studies. In RCTs, the force of infection does not change and leads to an underestimate of the true population-based effectiveness of the vaccine because these studies do not account for indirect effects of vaccination (i.e., impact of community immunity). On the other hand, large population-based observational studies of vaccine effectiveness do account for indirect effects, but they are limited by the potential for selection bias and unmeasured confounding.^{Footnote 28} ^{Footnote 151} Selection bias may result from systematic differences in sampling of individuals who are vaccinated compared to those who are not, resulting in a distribution of exposures and outcomes that is no longer representative of the source population. Confounding occurs when all or part of the apparent association between the exposure (the vaccination program) and outcome (e.g., hospitalizations averted, deaths averted) is, in fact, accounted for by other variables that affect the outcome and are not themselves affected by exposure. Examples of these factors could include level of access to healthcare services, socioeconomic status, and prevalence of natural immunity.^{Footnote 151} Researchers should note that observational studies of vaccine effectiveness are difficult to conduct, and usually cannot be undertaken prior to a vaccine being licensed. Also, effectiveness of vaccination programs for preventing disease in both vaccinated and unvaccinated individuals at the population-level (community immunity) is dependent upon vaccine coverage and dose completion rates. Uncertainty in these parameters should be accounted for. When considering observational studies of vaccine effectiveness for inclusion in economic evaluations, researchers may consider referring to published guidelines for assessing evidence from comparative effectiveness studies.^{Footnote 152} ^{Footnote 153}

Uncertainty in parameters related to transmission of infection between individuals should be reflected in an uncertainty analysis. These parameters include contact patterns between individuals, as well as other behaviours that may influence disease prevention and control. Researchers should account for any differences in these parameters between groups. For example, in diseases where asymptomatic or mildly symptomatic individuals can transmit infection to others, these individuals are less likely to modify their behaviours to reduce transmission compared to individuals whose symptoms are more severe.^{Footnote 28}

In some cases parameter values are estimated using models, which could be considered sub-models of the primary decision-analytic model.^{Footnote 144} For example, a predictive model may be required to establish the relationship between immune biomarkers that are vaccine CoPs, and the incidence of clinically apparent disease. In this case, uncertainty in the values related to CoPs as well as uncertainty in the methods used to model the relationship between CoPs and clinical disease need to be accounted for.

When calibration is used to estimate model parameters, uncertainty in the estimates derived from the calibration process should be explored.^{Footnote 145} As Taylor et al. have demonstrated in their cost-effectiveness analysis of HPV vaccine, failing to account for uncertainty related to calibrated parameters in the model underestimates the true extent of uncertainty in the cost-effectiveness estimates.^{Footnote 61}

Structural uncertainty

Structural uncertainty relates to the choice of model structure. When constructing models for economic evaluation of vaccination programs, researchers need to ensure that model structure accounts for factors related to transmission of infection between individuals, including the role of population subgroups that may be at high risk for transmitting or acquiring infection, the natural history of the disease being modeled, as well as the direct and indirect effects of the vaccination program.^{Footnote 28} ^{Footnote 46}

Structural uncertainty related to transmission of infection can be related to any of the following factors: 1) mode of transmission; 2) the relationship between severity of symptoms and transmissibility (i.e., whether asymptomatic or minimally symptomatic individuals can transmit infection); 3) mixing and contact patterns of individuals within populations; and 4) behavioural changes in response to disease outbreaks.^{Footnote 28} ^{Footnote 154} ^{Footnote 155} Researchers should test alternate assumptions related to these factors in all applicable situations to ensure that uncertainty related to transmission has been adequately examined.

For certain infectious diseases, population subgroups may be epidemiologically important to disease transmission, or to risk of infection in the overall population. These subgroups may intersect with equity-relevant subgroups, which are discussed in Chapter 14 on Equity. Examples include persons with hepatitis A, men who have sex with men and injection and non-injection drug users— all of whom are at high risk for transmitting and acquiring infection.^{Footnote 156} Model assumptions of the role of high-risk groups with respect to transmission should be tested to understand the degree of uncertainty they contribute.

Uncertainty about the natural history of an infectious disease often centers on whether it leads to latent infection or natural immunity. This is the case with certain high-risk strains of HPV. Decision analytic models, where different assumptions on latency and natural immunity to these HPV strains were employed, demonstrated that results are highly sensitive to these assumptions.^{Footnote 157} Accordingly, researchers should account for such uncertainties in the structure of a model. In this HPV example, both susceptible-infectious-susceptible (SIS) and susceptible-infectious-removed (immune) (SIR) frameworks could be examined to assess how results vary with alternate model structures.^{Footnote 28}

Important aspects of structural uncertainty related to vaccination programs include the timing of vaccine doses, the duration of protection provided by the vaccination, and any indirect effects that may result from the vaccine.^{Footnote 28} ^{Footnote 86} When applicable, the use of vaccine booster doses should also be evaluated.

In situations where there is uncertainty about whether the protection provided by a vaccine wanes, researchers should test different plausible assumptions related to duration of protection. These assumptions, where possible, should be based on immunologic evidence on the relationship between immune correlates of long-term protection and occurrence of clinical disease in the post-vaccination period.^{Footnote 86} ^{Footnote 158} Epidemiologic data related to disease outbreaks, when available, might also be useful in modelling duration of protection conferred by vaccines as has been demonstrated with the examples of childhood mumps vaccination,^{Footnote 159} and whole-cell and acellular pertussis vaccination.^{Footnote 12} Examples of some methods used to predict duration of effect include linear functions, logarithmic functions, and exponential functions. Constant functions are used in models that assume no waning of protection.^{Footnote 31} ^{Footnote 86}

One of the critical decisions faced by researchers conducting economic evaluations of vaccination programs is whether to model the indirect effects of vaccines using dynamic models. It has been argued that if the only indirect effect of a vaccination program is community immunity, then the only uncertainty in results based on a static model (which does not account for these effects) is on how much more favourable the results of the economic evaluation would be towards the vaccination program being evaluated.^{Footnote 9} This uncertainty, however, is only acceptable in situations where a static model has demonstrated that a vaccination program is cost-effective; in cases where the vaccination program has not been found to be cost-effective, it is problematic since a dynamic model could either confirm the lack of cost-effectiveness or produce a result that demonstrates that the vaccination program is indeed cost-effective.

In cases where there are indirect effects associated with a vaccine such as age-shifting of disease or serotype replacement, the decision to use a static model instead of a dynamic model could result in a greater degree of uncertainty. Dynamic models are required to account for uncertainties on age-shifting of the disease in economic evaluations of vaccination programs. Researchers should consider various scenarios related to changing epidemiology of disease after the introduction of a vaccination program to assess the effects of age-shifting on the results of an economic evaluation. The effects of serotype replacement should also be explored using dynamic models.

Decisions on how to address uncertainty related to the different dimensions of vaccine effectiveness (prevention of clinical disease, severity of clinical disease, infection, and infectiousness) may be complicated in cases where these effects are not well understood. For example, it is thought that meningococcal B vaccines do not provide community immunity by preventing transmission of the bacteria between individuals, but data on the true extent of the effectiveness of this vaccine is limited.^{Footnote 160} Researchers should use different model structures to explore this type of uncertainty where relevant.

Structural uncertainty influences results of economic evaluations at least to the same extent as parameter uncertainty, and often to a greater extent.^{Footnote 46} ^{Footnote 145} It is particularly important to explore structural uncertainty in dynamic models with uncertainty analysis because of their non-linear effects, which can lead to variable model behaviour.^{Footnote 28} Scenario analysis should be used to explore structural uncertainties in models. This technique involves testing alternate model scenarios that are underpinned by different plausible structural assumptions. Results from each scenario analysis should be presented individually. Model averaging can then be used to summarize results from all of the alternate model scenarios that were tested. When averaging results from the scenarios analyses, weightings for each model based on the model’s predictive ability according to available data (e.g., measures of fit) should be used.^{Footnote 161} ^{Footnote 162} When weightings cannot be derived from data, researchers’ judgment and expert opinion may be required.

Parameterization is an emerging method for addressing structural uncertainty. It involves adding parameters to a model that are assumed to be the sources of structural uncertainty and assigning them a single, often extreme value so that in some cases they may be completely excluded from the model whereas in other cases they are an important component of the model. This method allows structural uncertainty to be internalized in the model, and could be used to inform any decision about future research to resolve these uncertainties.^{Footnote 145} ^{Footnote 161} Justification should be provided for any structural uncertainties that have not been addressed.^{Footnote 145}

Methodological uncertainty

Methodological uncertainty relates to the different methods that can be used to conduct an economic evaluation. When conducting economic evaluations of vaccination programs, important methodological choices that researchers must consider include the type of analysis, perspective, discounting approach and rate, and time horizon.^{Footnote 46} ^{Footnote 144}

Because vaccination programs often prevent diseases that could result in catastrophic consequences (e.g., meningococcal B vaccination could prevent death or permanent neurological sequelae resulting from meningitis due to meningococcal type B bacteria), they produce health-related benefits as well as non-health-related benefits such as improvements in education or lifetime productivity. As such, some authors have argued that CBAs should be considered in the economic evaluation of vaccination programs in order to account for their full range of benefits.^{Footnote 8} In practice, however, uncertainty related to the type of evaluation conducted is rarely examined.^{Footnote 46} In principle, the non-health costs of vaccination programs could be captured in a CUA if a broader perspective (e.g., societal perspective) for the analysis is adopted, but capturing non-health benefits may be more challenging.^{Footnote 8} ^{Footnote 144} Discrete choice experiments are an increasingly popular option for capturing relevant trade-offs for non-health benefits of interventions for either CBA or CUA. Accordingly, these guidelines recommend conducting two reference case analyses: one from the publicly funded health system perspective and another from the societal perspective.

Economic evaluations of vaccination programs are particularly sensitive to the discounting strategy, use of annuitization, and time horizon chosen for the analysis since costs related to the initiation of a vaccination program are incurred when the program is introduced while the full benefits of the program often takes a much longer period of time, sometimes many years or decades, to realize.^{Footnote 9} ^{Footnote 46} As such, researchers should examine how varying the discounting approach, as well as time horizon of the analysis affects the results. When the time horizon of the analysis is very long (e.g., multiple decades), results of the economic evaluation should be reported for a range of time horizons to ensure that decision-makers are considering costs and outcomes that are relevant to the decision problem they are addressing.

To thoroughly explore many aspects of methodological uncertainty, multiple different models are ideally required, which is often practically difficult. As such, researchers should transparently collaborate with other groups addressing similar decision problems, whenever possible, so that the extent of uncertainty related to methodological choices can be explored.

Value-of-information

If a value-of-information analysis is undertaken, summarize the value of additional information using the expected value of perfect parameter information and the population expected value of perfect parameter information. See CADTH Guidelines for further guidance.

14. Equity

The traditional emphasis of economic evaluations of healthcare interventions has been the assessment of efficiency. This exercise sits within the larger decision-making framework of HTA, which synthesizes and appraises primarily clinical and economic evidence related to a new health intervention or technology. However, there is growing recognition that ethical and moral questions related to how a technology is appraised and used should be addressed as part of decisions on the adoption of new health technologies.^{Footnote 163} ^{Footnote 164}

NACI has established the Ethics, Equity, Feasibility and Acceptability (EEFA) Framework to systematically consider these factors as part of a multi-criteria approach to vaccine recommendations. In this framework, ethics and equity are considered with the feasibility and acceptability of a recommendation, alongside a vaccine’s clinical effectiveness, immunogenicity, safety, and cost-effectiveness.^{Footnote 5} Public health ethics is the domain of applied ethics relevant to vaccination. It is primarily concerned with the following core ethical dimensions: 1) respect for persons and communities; 2) non-maleficence and beneficence; 3) trust; and 4) justice.^{Footnote 165}

Equity is considered within the core ethical dimension of justice, and is defined as “the absence of avoidable, unfair, or remedial differences among groups of people, whether those groups are defined socially, economically, demographically or geographically or by means of stratification.”^{Footnote 5} ^{Footnote 166} Equity in economic evaluations is an approach to distributive justice that concerns judgments about the fairness in distribution of health outcomes and experiences in a population, and it relates to the fair allocation of resources and achievement of health improvements between individuals or groups.^{Footnote 167} There has been considerable recent activity and methodological development related to equity in the economic evaluation of health technologies.^{Footnote 164} ^{Footnote 168} ^{Footnote 169} ^{Footnote 170}

The distributional consequences related to adoption of a new health technology become particularly important in situations where decision-makers must make trade-offs between attributes of health technologies. For example, health economists frequently flag trade-offs between cost-effectiveness and health equity. These situations arise when a technology is cost-effective but increases inequity between groups in a population because some segments of society may benefit from the technology more than others. Alternatively, they arise when a technology is not cost-effective but its adoption would improve equity between groups by reducing disparities in health gains, or when a technology would increase equity between some groups (e.g., income strata) but decrease equity between others (e.g., geographical).^{Footnote 171}

Researchers and decision-makers should work collaboratively to establish which of the following equity goals the vaccination program is aiming to address: 1) improving equity in access to the vaccination program for eligible individuals; 2) improving equity in uptake of the vaccination program in eligible individuals; 3) improving equity in health benefit related to health conditions addressed by the vaccination program; 4) reducing lifetime health inequities between groups by means of the vaccination program; or 5) reducing overall (i.e., health and non-health-related) inequities between groups by means of the vaccination program. Different approaches may be used to conceptualize equity goals (e.g., proportionate universalism, egalitarianism). Researchers may find it helpful to refer to published literature on this topic when conceptualizing equity objectives for inclusion in an economic evaluation.^{Footnote 172} ^{Footnote 173} ^{Footnote 174}

When establishing equity goals researchers should consider whether there are key groups of individuals experiencing health inequities and barriers to health that could be reduced or addressed by the vaccination program. Examples of groups that may experience health inequity in Canada include Indigenous Peoples (specifically, First Nations, Inuit, and Métis Peoples for the purposes of these guidelines), individuals of low socioeconomic status, people who are part of ethnic, sexual, or gender minority groups, populations living in certain geographic locations (urban vs. rural vs. remote and isolated), individuals with disabilities, and vulnerable groups such as children, seniors or institutionalized persons.^{Footnote 5} ^{Footnote 117} ^{Footnote 170} ^{Footnote 175} ^{Footnote 176}

Researchers should also consider factors that could lead to differences in health benefits resulting from the vaccination program between groups experiencing health inequities. Factors include underlying health conditions, potential for lifetime benefit, health-seeking behaviours, uptake of the vaccine and the role of community immunity in reducing or increasing inequities between groups, risk-taking behaviours, different mixing or contact patterns within groups, and access to culturally safe healthcare.^{Footnote 6} ^{Footnote 177}

Researchers should be aware that some groups may benefit from the vaccination program, and some groups may not, thus potentially increasing inequities. For example, differential access to an HPV vaccination program can worsen inequity by reducing the rate of cervical cancer in a population who was already at lower risk but have greater access, thereby increasing the difference in outcomes between groups. Interventions that appear to reduce inequities should be examined to investigate how relevant barriers to access faced by the population would be overcome.

Once researchers have established the equity-relevant outcomes of interest, features of the vaccination program intended to achieve these outcomes should be considered. For instance, if the goal of the program is to improve equity in access to the vaccine for all eligible individuals, then a program that decreases barriers to access should be considered. An example of such a program would be a school-based HPV vaccination program that eliminates barriers for individuals such as the cost of the vaccine doses, and the need for transportation to a clinic or physician’s office.^{Footnote 117} If the goal of the program, however, is to improve equity in uptake of the vaccine, researchers could consider scenarios in which vaccines are mandatory or that address misinformation about the vaccine. An example of such a program would be a legally mandated school-based program for HPV vaccination, with a provision for active opt-out.^{Footnote 178} If the goal is to reduce lifetime health inequity between groups with the vaccination program, a program that is consistent with the principle of vertical equity, which entails treating individuals with different ethically-relevant characteristics differently, should be considered.^{Footnote 179} An example would be a vaccination program aimed at achieving high levels of vaccination coverage among Indigenous Peoples. Indigenous Peoples experience a greater burden of vaccine-preventable diseases than non-Indigenous People in Canada (e.g., cervical cancer, hepatitis A) due to systematic inequities such as poverty, crowded housing conditions, lack of running water, and poor underlying health status, which increase the risk for acquiring these infections.^{Footnote 180} ^{Footnote 181} Additionally, Indigenous Peoples living on reserves and in remote communities may also experience inequities in access to treatment when they become ill, increasing their risk of infection-related morbidity and mortality.^{Footnote 182} Researchers must be aware, however, that vaccination programs restricted to certain high-risk groups that are vulnerable or marginalized may serve to further stigmatize those groups. Alternative approaches, such as more universal programs, should be considered. Finally, if the goal is to improve health and non-health equity between groups, researchers could consider vaccination programs that contribute towards improving health as well as economic productivity. Examples of such programs are childhood vaccination programs, which enable children to participate in education, in turn allowing them to become healthy and economically productive adults.^{Footnote 183} When consideration of equity-relevant outcomes relates to selection and definition of comparator(s) to be included in the analysis, researchers should refer to Chapter 3 on Comparators of these guidelines.

When presenting results of economic evaluations by equity-relevant subgroups, researchers should ensure that the criteria for establishing these subgroups has been transparently delineated and justified. A recent review of equity-informative CEAs identified eleven different criteria that have been used to explicitly incorporate equity in a cost-effectiveness framework, with socioeconomic status and race/ ethnicity used most frequently.^{Footnote 184} Distributed (DCEA) and extended CEA (ECEA) frameworks provide guidance and methods for conducting equity-informative CEAs.^{Footnote 168} ^{Footnote 185}

In addition to considering equity-related outcomes associated with vaccination programs, researchers should also consider the distribution of opportunity costs related to the implementation of these programs.^{Footnote 171} This redistribution of resources could, for example, result in decreased expenditures on screening programs or non-vaccine preventive measures related to the infection being targeted by vaccination program. Opportunity costs could also fall outside of the health sector, for example, through decreased funding of educational or social programs.^{Footnote 171} Although in many cases, it may be difficult to explicitly identify opportunity costs related to implementing vaccination programs, where possible, researchers should quantify opportunity costs in a manner that is relevant to decision-makers. In some cases, interventions to improve equity may not carry a net opportunity cost, since it may be efficient to allocate resources to groups with higher health burden.

When relevant, researchers should consider the implications of vaccination programs on intergenerational equity. Vaccination programs that result in externalities have effects on cohorts of individuals other than the cohort that is vaccinated.^{Footnote 34} For example, a childhood varicella vaccination program may result in increased cases of herpes zoster in older adults;^{Footnote 186} conversely, an HPV vaccination program may lead to disease eradication for future generations.^{Footnote 187} In both of these examples, the indirect effects on cohorts of individuals not intended for the vaccination program should be accounted for with dynamic models used to generate estimates of cost-effectiveness. Researchers should then explicitly consider the equity implications of these results. In the first example, researchers need to qualitatively identify the trade-offs between improved child health and negative health outcomes that may be experienced by older individuals. Quantitatively, the summary costa and outcomes estimated in the analysis capture the trade-offs. In the second example, researchers need to consider how health outcomes that accrue to cohorts far into the future should be valued in present day terms.

Such valuation of health outcomes in cohorts far into the future is contingent on the discounting strategy employed in the economic evaluation. An equal discount rate results in greater value placed on health outcomes for the present cohort and cohorts close in time to the present, while lesser value is placed on health outcomes for cohorts in the distant future, which some authors argue is an unfair feature of this strategy.^{Footnote 33} Use of lower discount rates, however, could result in giving greater weight to health outcomes in cohorts that are likely to have greater income, and access to more health interventions, and as such, more potential to improve health. Given the variable outcomes related to intergenerational effects of vaccination programs that can result from different discounting strategies, researchers should consider and report the intergenerational equity implications of vaccination programs that result in health benefits for cohorts in the distant future.^{Footnote 33} ^{Footnote 34} ^{Footnote 188}

15. Reporting

15.2 “A summary of the evaluation written in non-technical language should be included.” [CADTH Guideline Statement]

15.3 “Results of the economic evaluation should be presented in graphical or visual form, in addition to tabular presentation.” [CADTH Guideline Statement]

15.5 “Funding and reporting relationships for the evaluation should be described, and any conflicts of interest disclosed.” [CADTH Guideline Statement]

Reporting results of economic evaluations should provide decision-makers with transparent and credible information that enables them to address the decision problem of interest, and make an optimal funding decision related to the vaccination program being considered.

Vaccination-specific reporting considerations should be addressed including the time horizon of the evaluation, and the mechanisms through which vaccines exert their effects. In cases where the model time horizon of an economic evaluation spans a long period of time, results from various time points over the model time horizon should be reported to ensure that findings of the analysis are relevant to the time horizon being considered by decision-makers. Since vaccines may exert their effects through various mechanisms (e.g., preventing transmission of infection, preventing infection, preventing disease or decreasing its severity), researchers should report outcomes of vaccination programs not only in terms of QALYs, but also in terms of the number of cases prevented, the number of relevant healthcare utilization units (e.g., hospitalizations) averted, the number of deaths averted, and the number of individuals needed to vaccinate, where applicable. Reporting these metrics in addition to QALYs increases the credibility and transparency of the analysis for decision-makers.

Resource use, costs, and outcomes should be reported in a disaggregated fashion for each comparator considered in the analysis and for both reference case analyses. If analyses are conducted from multiple public payer perspectives (e.g., for multiple provinces/ territories), each should be reported separately.

All assumptions and decision rules used in the analyses should be transparently reported.

Researchers should provide details related to quality assurance processes, and results undertaken as part of the model verification process. A fully executable electronic copy of the model should be made available along with details related to the model’s functionality to enable the decision-maker to verify results of the analysis or conduct additional analyses if required.

Researchers should disclose all sources of funding for the economic evaluation and state the role of the funder(s) in the identification, design, conduct, and reporting of the analysis. Non-monetary (e.g., in-kind) sources of support should also be disclosed.^{Footnote 189}

Researchers should disclose all potential conflicts of interest, both financial and non-financial. Types of affiliations and interests to disclose include: participation in research, equity ownership, intellectual property, and any other interest that readers may perceive as a competing interest (e.g., public statements about the topic).^{Footnote 190}

The “NACI Guidelines for Reporting Economic Evaluations of Vaccination Programs in Canada” on the NACI website provides a standard format for reporting the results of economic evaluations of vaccination programs.^{Footnote 191} Researchers should follow the structure outlined in this document when presenting their results.

Appendix 1: Impact inventory table

Table 1: Impact inventory table for economic evaluations of vaccination strategies
Area of Impact	Definitions/Examples	Included in Reference Case?		Comments
Area of Impact	Definitions/Examples	Publicly funded health system perspective	Societal perspective	Comments
Health
Health outcomes	Individual health outcomes for persons intended for vaccination
	Mortality			N/A
	Health-related quality of life			N/A
	Safety (i.e., adverse events)			N/A
	Irreversible health impacts not captured by QALYs (e.g., infertility associated with sexually transmitted infections			N/A
	Individual health outcomes for informal caregivers
	Health-related quality of life			N/A
	Population health outcomes
	Incidence of disease in vaccinated and unvaccinated individuals			N/A
	Changes in age distribution of individuals who develop infection and disease			N/A
	Changes in infection and disease incidence related to variations of pathogen or other pathogens that replace ones targeted by vaccine			N/A
	Disease eradication			N/A
Health system costs	Healthcare treatment costs
	Publicly funded healthcare services (e.g., physician visits, diagnostic tests, drug treatment where applicable, hospitalization, formal caregiving,^{Footnote a} rehabilitation in a facility or at home,^{Footnote a} home care,^{Footnote a} long-term care in nursing homes^{Footnote a})			N/A
	Future related and unrelated healthcare costs			N/A
	Public health costs
	Program-related costs (e.g., implementation, delivery and recurrent costs, public health campaigns, health promotion activities, transaction costs, population-based screening, epidemiologic surveillance, contact tracing, investigation and management of outbreaks)			N/A
	Intervention-related costs (e.g., cost of vaccine doses, distribution such as transportation and cold storage, administration including personnel, wastage and ancillary supplies)			N/A
Healthcare costs NOT funded by the health system	Drug treatments (in some cases)	N/A		N/A
	Formal caregiver services,^{Footnote a} rehabilitation in a facility or at home,^{Footnote a} home care,^{Footnote a} long-term care in nursing homes^{Footnote a} (in some cases)	N/A		N/A
	Miscellaneous out-of-pocket costs (e.g., non-prescription medications)	N/A		N/A
	Ancillary costs (e.g., private insurance copayments, dental care, vision care, assistive devices, physiotherapy, etc.)	N/A		N/A
Non-Health Areas
Direct out-of-pocket costs	Transportation costs	N/A		N/A
Direct out-of-pocket costs	Accommodation costs	N/A		N/A
Productivity loss	Paid work
	Time off work resulting from treatment, illness, disability, or death			N/A
	Presenteeism	N/A		N/A
	Lifetime productivity consequences of childhood disease	N/A		N/A
	Unpaid work
	Time off work in informal labour market resulting from treatment, illness, disability, or death	N/A		N/A
	Uncompensated household production (e.g., Cooking, cleaning, shopping, raising children, other tasks related to household management)	N/A		N/A
	Informal caregiver productivity
	Time off work resulting from caring for sick individuals	N/A		N/A
	Caregiver presenteeism	N/A		N/A
	Macroeconomic consequences
	Labour supply shocks, widespread business closures	N/A		N/A
Consumption	Future individual non-medical consumption	N/A		N/A
	Changes in household consumption	N/A		N/A
	Health impacts of consumption (e.g., associated with job loss)	N/A		N/A
Education	Level of educational achievement as a result of physical health, mental health, and cognition	N/A		N/A
Education	Costs of special education needs as a result of illness/disability	N/A		N/A
Social services and community services	Social services and community services (e.g., disability support, programs to improve access to vaccination programs for adults)	N/A		N/A
Social services and community services	Child and Youth Services (e.g. awareness programs, family respite, programs to improve access to vaccination programs for children and youth)	N/A		N/A
Environment	Environmental impact of vaccination programs and comparators (e.g., manufacturing, distribution, and implementation)	N/A		N/A
Other Areas	Consider areas such as legal/criminal or housing when applicable	N/A		N/A
Footnote a Some of these costs may or may not be incurred by the publicly funded health system, depending on the precise nature of these costs and the relevant jurisdiction Return to footnote a referrer

Appendix 2: Reference case

Specifications

Table 2 presents recommendations for the reference case analyses. In situations where the analyses do not follow the recommendations presented below, researchers should identify any deviations and provide justification based on the decision problem.

Table 2: Recommendations for reference case analyses
Section	Guidance
Decision Problem	Specify the details of the vaccination program, setting, perspective, costs, outcomes, time horizon and intended population for the evaluation.
Types of Evaluations	Conduct a cost-utility analysis (CUA) capturing health outcomes in terms of quality-adjusted life-years (QALYs).
Study Populations	Identify the population(s) in which the vaccination program will be used, and, when applicable, any populations that might experience externalities resulting from the vaccination program. Conduct stratified analysis where distinct subgroups are identified.
Comparators	Compare all relevant interventions, including other vaccination programs, screening interventions, medical and non-medical preventive interventions, and treatment-based approaches presently used in a Canadian context.
Perspective	Conduct two reference case analyses, one from the publicly funded health system perspective and one from the societal perspective.
Time Horizon	Select a time horizon that is long enough to capture all relevant differences in the future costs and outcomes associated with the interventions being compared.
Discounting	Discount costs and outcomes at a rate of 1.5% per year.
Measurement and Valuation of Health	Identify, measure, and value all relevant health outcomes based on the perspectives of the publicly funded health system and society. Use health preferences that reflect the general Canadian population. Obtain health preferences from an indirect method of measurement that is based on a generic classification system.
Resource Use and Costs	Identify, measure, and value all relevant resources and costs based on the perspective of the i) publicly funded health system, and ii) society. Estimate Canadian resources and costs using data that reflect the jurisdiction(s) of interest.
Analysis	Derive expected values of costs and outcomes for both the publicly funded health system perspective analysis and the societal perspective analysis for each intervention through probabilistic analysis, incorporating potential correlation among parameters, whenever possible. Where distinct subgroups are identified within the intended population, conduct a stratified analysis and present results for each subgroup. Calculate incremental costs, incremental effectiveness, and incremental cost-effectiveness ratios (ICERs) for both the publicly funded health system and societal perspective analyses. For evaluations with more than two comparators, calculate ICERs sequentially.
Uncertainty	Address methodological uncertainty by comparing the reference case results to those from a non- reference case analysis. Summarize decision uncertainty, using cost-effectiveness acceptability curves (CEACs) and cost-effectiveness acceptability frontiers (CEAFs), where possible. Use scenario analysis to address structural uncertainty. If a value-of-information analysis is undertaken, summarize the value of additional information using the expected value of perfect parameter information and the population expected value of perfect parameter information.
Equity	Consider whether there are inequities experienced by specific groups that could be improved by the vaccination program. Equity should be explored using methods such as distributional cost-effectiveness analysis and extended cost-effectiveness analysis. Any additional analyses should accompany the references case analyses when applicable.

References

Footnote 1

Canadian Agency for Drugs and Technologies in Health (CADTH). Guidelines for the Economic Evaluation of Health Technologies: Canada 4th edition. 2017.

Return to footnote 1 referrer

Footnote 2

Baron J. A theory of social decisions. J Theory Soc Behav. 1995; 25.

Return to footnote 2 referrer

Footnote 3

Claxton K., Paulden M., Gravelle H., Brouwer W., Culyer A.J. Discounting and decision making in the economic evaluation of health-care technologies. Health Econ. 2011; 20(1): 2-15. doi:10.1002/hec.1612

Return to footnote 3 referrer

Footnote 4

Paulden M., Galvanni V., Ghakraborty S., Kudinga B., McCabe C. Discounting and the evaluation of health care programs. n.d.

Return to footnote 4 referrer

Footnote 5

Ismail S.J., Hardy K., Tunis M.C., Young K., Sicard N., Quach C. A framework for the systematic consideration of ethics, equity, feasibility, and acceptability in vaccine program recommendations. Vaccine. 2020; 38(36): 5861-76. doi:10.1016/j.vaccine.2020.05.051

Return to footnote 5 referrer

Footnote 6

Brisson M., Drolet M., Malagón T. Inequalities in Human Papillomavirus (HPV)-associated cancers: implications for the success of HPV vaccination. J Natl Cancer Inst. 2013; 105(3): 158-61. doi:10.1093/jnci/djs638

Return to footnote 6 referrer

Footnote 7

Bloom D.E., Brenzel L., Cadarette D., Sullivan J. Moving beyond traditional valuation of vaccination: Needs and opportunities. Vaccine. 2017; 35 Suppl 1: A29-a35. doi:10.1016/j.vaccine.2016.12.001

Return to footnote 7 referrer

Footnote 8

Park M., Jit M., Wu J.T. Cost-benefit analysis of vaccination: a comparative analysis of eight approaches for valuing changes to mortality and morbidity risks. BMC Med. 2018; 16(1): 139. doi:10.1186/s12916-018-1130-7

Return to footnote 8 referrer

Footnote 9

Walker D.G., Hutubessy R., Beutels P. WHO Guide for standardisation of economic evaluations of immunization programmes. Vaccine. 2019; 28(11): 2356-9. doi:10.1016/j.vaccine.2009.06.035

Return to footnote 9 referrer

Footnote 10

Brisson M., Edmunds W.J. The cost-effectiveness of varicella vaccination in Canada. Vaccine. 2002; 20(7-8): 1113-25. doi:10.1016/s0264-410x(01)00437-6

Return to footnote 10 referrer

Footnote 11

Christensen H., Hickman M., Edmunds W.J., Trotter C.L. Introducing vaccination against serogroup B meningococcal disease: an economic and mathematical modelling study of potential impact. Vaccine. 2013; 31(23): 2638-46. doi:10.1016/j.vaccine.2013.03.034

Return to footnote 11 referrer

Footnote 12

Gambhir M., Clark T.A., Cauchemez S., Tartof S.Y., Swerdlow D.L., Ferguson N.M. A change in vaccine efficacy and duration of protection explains recent rises in pertussis incidence in the United States. PLoS Comput Biol. 2015; 11(4): e1004138. doi:10.1371/journal.pcbi.1004138

Return to footnote 12 referrer

Footnote 13

Markowitz L.E., Drolet M., Perez N., Jit M., Brisson M. Human papillomavirus vaccine effectiveness by number of doses: Systematic review of data from national immunization programs. Vaccine. 2018; 36(32 Pt A): 4806-15. doi:10.1016/j.vaccine.2018.01.057

Return to footnote 13 referrer

Footnote 14

Bärnighausen T., Bloom D.E., Cafiero-Fonseca E.T., O'Brien J.C. Valuing vaccination. Proc Natl Acad Sci U S A. 2014; 111(34): 12313-9. doi:10.1073/pnas.1400475111

Return to footnote 14 referrer

Footnote 15

Chandran A., Herbert H., Misurski D., Santosham M. Long-term sequelae of childhood bacterial meningitis: an underappreciated problem. Pediatr Infect Dis J. 2011; 30(1): 3-6. doi:10.1097/INF.0b013e3181ef25f7

Return to footnote 15 referrer

Footnote 16

Grimwood K., Anderson P., Anderson V., Tan L., Nolan T. Twelve year outcomes following bacterial meningitis: further evidence for persisting effects. Arch Dis Child. 2000; 83(2): 111-6. doi:10.1136/adc.83.2.111

Return to footnote 16 referrer

Footnote 17

Soriano-Gabarró M., Mrukowicz J., Vesikari T., Verstraeten T. Burden of rotavirus disease in European Union countries. Pediatr Infect Dis J. 2006; 25(1 Suppl): S7-s11. doi:10.1097/01.inf.0000197622.98559.01

Return to footnote 17 referrer

Footnote 18

Standaert B., Van de Mieroop E., Nelen V. Exploring the potential impact of rotavirus vaccination on work absenteeism among female administrative personnel of the City of Antwerp through a retrospective database analysis. BMJ Open. 2015; 5(6): e007453. doi:10.1136/bmjopen-2014-007453

Return to footnote 18 referrer

Footnote 19

Kim D.D., Neumann P.J. Analyzing the Cost Effectiveness of Policy Responses for COVID-19: The Importance of Capturing Social Consequences. Med Decis Making. 2020; 40(3): 251-3. doi:10.1177/0272989x20922987

Return to footnote 19 referrer

Footnote 20

Folegatti P.M., Ewer K.J., Aley P.K., Angus B., Becker S., Belij-Rammerstorfer S., et al. Safety and immunogenicity of the ChAdOx1 nCoV-19 vaccine against SARS-CoV-2: a preliminary report of a phase 1/2, single-blind, randomised controlled trial. Lancet. 2020; 396(10249): 467-78. doi:10.1016/s0140-6736(20)31604-4

Return to footnote 20 referrer

Footnote 21

Zhu F.-C., Guan X.-H., Li Y.-H., Huang J.-Y., Jiang T., Hou L.-H., et al. Immunogenicity and safety of a recombinant adenovirus type-5-vectored COVID-19 vaccine in healthy adults aged 18 years or older: a randomised, double-blind, placebo-controlled, phase 2 trial. The Lancet. 2020; 396(10249): 479-88. doi:10.1016/S0140-6736(20)31605-6

Return to footnote 21 referrer

Footnote 22

Sanders G.D., Neumann P.J., Basu A., Brock D.W., Feeny D., Krahn M., et al. Recommendations for Conduct, Methodological Practices, and Reporting of Cost-effectiveness Analyses: Second Panel on Cost-Effectiveness in Health and Medicine. Jama. 2016; 316(10): 1093-103. doi:10.1001/jama.2016.12195

Return to footnote 22 referrer

Footnote 23

Bärnighausen T., Bloom D.E., Canning D., O'Brien J. Accounting for the full benefits of childhood vaccination in South Africa. S Afr Med J. 2008; 98(11): 842, 4-6.

Return to footnote 23 referrer

Footnote 24

Bloom B.S., Bruno D.J., Maman D.Y., Jayadevappa R. Usefulness of US cost-of-illness studies in healthcare decision making. Pharmacoeconomics. 2001; 19(2): 207-13. doi:10.2165/00019053-200119020-00007

Return to footnote 24 referrer

Footnote 25

Jit M., Hutubessy R., Png M.E., Sundaram N., Audimulam J., Salim S., et al. The broader economic impact of vaccination: reviewing and appraising the strength of evidence. BMC Med. 2015; 13: 209. doi:10.1186/s12916-015-0446-9

Return to footnote 25 referrer

Footnote 26

Kim S.Y., Goldie S.J. Cost-effectiveness analyses of vaccination programmes : a focused review of modelling approaches. Pharmacoeconomics. 2008; 26(3): 191-215. doi:10.2165/00019053-200826030-00004

Return to footnote 26 referrer

Footnote 27

Centres for Disease Control and Prevention (CDC). Measles (Rubeola). 2020. Available from: https://www.cdc.gov/measles/hcp/index.html.

Return to footnote 27 referrer

Footnote 28

Pitman R., Fisman D., Zaric G.S., Postma M., Kretzschmar M., Edmunds J., et al. Dynamic transmission modeling: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force--5. Value Health. 2012; 15(6): 828-34. doi:10.1016/j.jval.2012.06.011

Return to footnote 28 referrer

Footnote 29

Ultsch B., Damm O., Beutels P., Bilcke J., Brüggenjürgen B., Gerber-Grote A., et al. Methods for Health Economic Evaluation of Vaccines and Immunization Decision Frameworks: A Consensus Framework from a European Vaccine Economics Community. Pharmacoeconomics. 2016; 34(3): 227-44. doi:10.1007/s40273-015-0335-2

Return to footnote 29 referrer

Footnote 30

Mauskopf J., Talbird S., Standaert B. Categorization of methods used in cost-effectiveness analyses of vaccination programs based on outcomes from dynamic transmission models. Expert Rev Pharmacoecon Outcomes Res. 2012; 12(3): 357-71. doi:10.1586/erp.12.11

Return to footnote 30 referrer

Footnote 31

Bilcke J., Marais C., Ogunjimi B., Willem L., Hens N., Beutels P. Cost-effectiveness of vaccination against herpes zoster in adults aged over 60 years in Belgium. Vaccine. 2012; 30(3): 675-84. doi:10.1016/j.vaccine.2011.10.036

Return to footnote 31 referrer

Footnote 32

Zechmeister I., Blasio B.F., Garnett G., Neilson A.R., Siebert U. Cost-effectiveness analysis of human papillomavirus-vaccination programs to prevent cervical cancer in Austria. Vaccine. 2009; 27(37): 5133-41. doi:10.1016/j.vaccine.2009.06.039

Return to footnote 32 referrer

Footnote 33

Attema A.E., Brouwer W.B.F., Claxton K. Discounting in Economic Evaluations. Pharmacoeconomics. 2018; 36(7): 745-58. doi:10.1007/s40273-018-0672-z

Return to footnote 33 referrer

Footnote 34

Jit M., Mibei W. Discounting in the evaluation of the cost-effectiveness of a vaccination programme: A critical review. Vaccine. 2015; 33(32): 3788-94. doi:10.1016/j.vaccine.2015.06.084

Return to footnote 34 referrer

Footnote 35

Westra T.A., Parouty M., Brouwer W.B., Beutels P.H., Rogoza R.M., Rozenbaum M.H., et al. On discounting of health gains from human papillomavirus vaccination: effects of different approaches. Value Health. 2012; 15(3): 562-7. doi:10.1016/j.jval.2012.01.005

Return to footnote 35 referrer

Footnote 36

OECD. Economic Valuation of Environmental Health Risks to Children: OECD Publishing; 2006.

Return to footnote 36 referrer

Footnote 37

National Institute for Health and Care Excellence (NICE). Discounting of health benefits in special circumstances. 2011.

Return to footnote 37 referrer

Footnote 38

STIKO. Modelling methods for predicting epidemiological and health economic effects of vaccinations - guidance for analyses to be presented to the German Standing Committee on Vaccination (STIKO). 2016.

Return to footnote 38 referrer

Footnote 39

Weinstein M.C., Stason W.B. Foundations of cost-effectiveness analysis for health and medical practices. N Engl J Med. 1977; 296(13): 716-21. doi:10.1056/nejm197703312961304

Return to footnote 39 referrer

Footnote 40

Gravelle H., Smith D. Discounting for health effects in cost-benefit and cost-effectiveness analysis. Health Econ. 2001; 10(7): 587-99. doi:10.1002/hec.618

Return to footnote 40 referrer

Footnote 41

O'Mahony J.F., de Kok I.M., van Rosmalen J., Habbema J.D., Brouwer W., van Ballegooijen M. Practical implications of differential discounting in cost-effectiveness analyses with varying numbers of cohorts. Value Health. 2011; 14(4): 438-42. doi:10.1016/j.jval.2010.09.009

Return to footnote 41 referrer

Footnote 42

O’Mahoney J.F., Postma M.J., Boersma C., Van Rosmalen J., Jit M., editors. Correcting for Multiple Future Cohorts When Applying Differential Discounting of Costs and Health Effects. Society for Medical Decision Making 14th Bienniel European Meeting; 2012.

Return to footnote 42 referrer

Footnote 43

Brennan A., Chick S.E., Davies R. A taxonomy of model structures for economic evaluation of health technologies. Health Econ. 2006; 15(12): 1295-310. doi:10.1002/hec.1148

Return to footnote 43 referrer

Footnote 44

Stahl J.E. Modelling methods for pharmacoeconomics and health technology assessment: an overview and guide. Pharmacoeconomics. 2008; 26(2): 131-48. doi:10.2165/00019053-200826020-00004

Return to footnote 44 referrer

Footnote 45

Mac S., Mishra S., Ximenes R., Barrett K., Khan Y.A., Naimark D.M.J., et al. Modeling the coronavirus disease 2019 pandemic: A comprehensive guide of infectious disease and decision-analytic models. J Clin Epidemiol. 2021; 132: 133-41. doi:10.1016/j.jclinepi.2020.12.002

Return to footnote 45 referrer

Footnote 46

Brisson M., Edmunds W.J. Impact of model, methodological, and parameter uncertainty in the economic analysis of vaccination programs. Med Decis Making. 2006; 26(5): 434-46. doi:10.1177/0272989x06290485

Return to footnote 46 referrer

Footnote 47

Jit M., Brisson M. Modelling the epidemiology of infectious diseases for decision analysis: a primer. Pharmacoeconomics. 2011; 29(5): 371-86. doi:10.2165/11539960-000000000-00000

Return to footnote 47 referrer

Footnote 48

Lipsitch M. Bacterial vaccines and serotype replacement: lessons from Haemophilus influenzae and prospects for Streptococcus pneumoniae. Emerg Infect Dis. 1999; 5(3): 336-45. doi:10.3201/eid0503.990304

Return to footnote 48 referrer

Footnote 49

Weinberger D.M., Malley R., Lipsitch M. Serotype replacement in disease after pneumococcal vaccination. Lancet (London, England). 2011; 378(9807): 1962-73. doi:10.1016/S0140-6736(10)62225-8

Return to footnote 49 referrer

Footnote 50

Berge J.J., Drennan D.P., Jacobs R.J., Jakins A., Meyerhoff A.S., Stubblefield W., et al. The cost of hepatitis A infections in American adolescents and adults in 1997. Hepatology. 2000; 31(2): 469-73. doi:10.1002/hep.510310229

Return to footnote 50 referrer

Footnote 51

Edmunds W.J., Brisson M. The effect of vaccination on the epidemiology of varicella zoster virus. J Infect. 2002; 44(4): 211-9. doi:10.1053/jinf.2002.0988

Return to footnote 51 referrer

Footnote 52

Chhatwal J., He T. Economic evaluations with agent-based modelling: an introduction. Pharmacoeconomics. 2015; 33(5): 423-33. doi:10.1007/s40273-015-0254-2

Return to footnote 52 referrer

Footnote 53

Marshall D.A., Burgos-Liz L., MJ I.J., Crown W., Padula W.V., Wong P.K., et al. Selecting a dynamic simulation modeling method for health care delivery research-part 2: report of the ISPOR Dynamic Simulation Modeling Emerging Good Practices Task Force. Value Health. 2015; 18(2): 147-60. doi:10.1016/j.jval.2015.01.006

Return to footnote 53 referrer

Footnote 54

Grefenstette J.J., Brown S.T., Rosenfeld R., DePasse J., Stone N.T.B., Cooley P.C., et al. FRED (A Framework for Reconstructing Epidemic Dynamics): an open-source software system for modeling infectious diseases and control strategies using census-based populations. BMC Public Health. 2013; 13(1): 940. doi:10.1186/1471-2458-13-940

Return to footnote 54 referrer

Footnote 55

Wolff E., Storsaeter J., Örtqvist Å., Naucler P., Larsson S., Lepp T., et al. Cost-effectiveness of pneumococcal vaccination for elderly in Sweden. Vaccine. 2020; 38(32): 4988-95. doi:10.1016/j.vaccine.2020.05.072

Return to footnote 55 referrer

Footnote 56

Ethgen O., Standaert B. Population- versus cohort-based modelling approaches. Pharmacoeconomics. 2012; 30(3): 171-81. doi:10.2165/11593050-000000000-00000

Return to footnote 56 referrer

Footnote 57

Mauskopf J. Meeting the NICE requirements: a Markov model approach. Value Health. 2000; 3(4): 287-93. doi:10.1046/j.1524-4733.2000.34006.x

Return to footnote 57 referrer

Footnote 58

Vanni T., Karnon J., Madan J., White R.G., Edmunds W.J., Foss A.M., et al. Calibrating models in economic evaluation: a seven-step approach. Pharmacoeconomics. 2011; 29(1): 35-49. doi:10.2165/11584600-000000000-00000

Return to footnote 58 referrer

Footnote 59

Van de Velde N., Boily M.C., Drolet M., Franco E.L., Mayrand M.H., Kliewer E.V., et al. Population-level impact of the bivalent, quadrivalent, and nonavalent human papillomavirus vaccines: a model-based analysis. J Natl Cancer Inst. 2012; 104(22): 1712-23. doi:10.1093/jnci/djs395

Return to footnote 59 referrer

Footnote 60

Zang X., Krebs E., Min J.E., Pandya A., Marshall B.D.L., Schackman B.R., et al. Development and Calibration of a Dynamic HIV Transmission Model for 6 US Cities. Med Decis Making. 2020; 40(1): 3-16. doi:10.1177/0272989x19889356

Return to footnote 60 referrer

Footnote 61

Taylor D.C., Pawar V., Kruzikas D.T., Gilmore K.E., Sanon M., Weinstein M.C. Incorporating calibrated model parameters into sensitivity analyses: deterministic and probabilistic approaches. Pharmacoeconomics. 2012; 30(2): 119-26. doi:10.2165/11593360-000000000-00000

Return to footnote 61 referrer

Footnote 62

Eddy D.M., Hollingworth W., Caro J.J., Tsevat J., McDonald K.M., Wong J.B. Model transparency and validation: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force--7. Value Health. 2012; 15(6): 843-50. doi:10.1016/j.jval.2012.04.012

Return to footnote 62 referrer

Footnote 63

Van de Velde N., Brisson M., Boily M.C. Modeling human papillomavirus vaccine effectiveness: quantifying the impact of parameter uncertainty. Am J Epidemiol. 2007; 165(7): 762-75. doi:10.1093/aje/kwk059

Return to footnote 63 referrer

Footnote 64

Ghabri S., Stevenson M., Möller J., Caro J.J. Trusting the Results of Model-Based Economic Analyses: Is there a Pragmatic Validation Solution? Pharmacoeconomics. 2019; 37(1): 1-6. doi:10.1007/s40273-018-0711-9

Return to footnote 64 referrer

Footnote 65

Government of Canada. Recommended immunization schedules: Canadian Immunization Guide. 2020. Available from: https://www.canada.ca/en/public-health/services/publications/healthy-living/canadian-immunization-guide-part-1-key-immunization-information/page-13-recommended-immunization-schedules.html.

Return to footnote 65 referrer

Footnote 66

Fine P., Eames K., Heymann D.L. "Herd immunity": a rough guide. Clin Infect Dis. 2011; 52(7): 911-6. doi:10.1093/cid/cir007

Return to footnote 66 referrer

Footnote 67

Czerkinsky C., Holmgren J. Vaccines against enteric infections for the developing world. Philos Trans R Soc Lond B Biol Sci. 2015; 370(1671). doi:10.1098/rstb.2015.0142

Return to footnote 67 referrer

Footnote 68

Gjini E. Geographic variation in pneumococcal vaccine efficacy estimated from dynamic modeling of epidemiological data post-PCV7. Sci Rep. 2017; 7(1): 3049. doi:10.1038/s41598-017-02955-y

Return to footnote 68 referrer

Footnote 69

Plotkin S.A. Complex correlates of protection after vaccination. Clin Infect Dis. 2013; 56(10): 1458-65. doi:10.1093/cid/cit048

Return to footnote 69 referrer

Footnote 70

Government of Canada. Vaccine Coverage in Canadian Children: Results from the 2017 Childhood National Immunization Coverage Survey (cNICS). 2020.

Return to footnote 70 referrer

Footnote 71

Krebs E., Nosyk B. Cost-Effectiveness Analysis in Implementation Science: a Research Agenda and Call for Wider Application. Curr HIV/AIDS Rep. 2021; 18(3): 176-85. doi:10.1007/s11904-021-00550-5

Return to footnote 71 referrer

Footnote 72

Andrews N.J., Waight P.A., Burbidge P., Pearce E., Roalfe L., Zancolli M., et al. Serotype-specific effectiveness and correlates of protection for the 13-valent pneumococcal conjugate vaccine: a postlicensure indirect cohort study. Lancet Infect Dis. 2014; 14(9): 839-46. doi:10.1016/s1473-3099(14)70822-9

Return to footnote 72 referrer

Footnote 73

Daniels C.C., Rogers P.D., Shelton C.M. A Review of Pneumococcal Vaccines: Current Polysaccharide Vaccine Recommendations and Future Protein Antigens. J Pediatr Pharmacol Ther. 2016; 21(1): 27-35. doi:10.5863/1551-6776-21.1.27

Return to footnote 73 referrer

Footnote 74

Malagón T., Drolet M., Boily M.C., Franco E.L., Jit M., Brisson J., et al. Cross-protective efficacy of two human papillomavirus vaccines: a systematic review and meta-analysis. Lancet Infect Dis. 2012; 12(10): 781-9. doi:10.1016/s1473-3099(12)70187-1

Return to footnote 74 referrer

Footnote 75

Plotkin S.A. Updates on immunologic correlates of vaccine-induced protection. Vaccine. 2020; 38(9): 2250-7. doi:10.1016/j.vaccine.2019.10.046

Return to footnote 75 referrer

Footnote 76

World Health Organization (WHO). Guidelines on clinical evaluation of vaccines: regulatory expectations. 2017.

Return to footnote 76 referrer

Footnote 77

Centres for Disease Control and Prevention (CDC). Immunogenicity, Efficacy, and Effectiveness of Influenza Vaccines. 2019. Available from: https://www.cdc.gov/flu/professionals/acip/immunogenicity.htm.

Return to footnote 77 referrer

Footnote 78

Plotkin S.A., Gilbert P.B. Nomenclature for immune correlates of protection after vaccination. Clin Infect Dis. 2012; 54(11): 1615-7. doi:10.1093/cid/cis238

Return to footnote 78 referrer

Footnote 79

World Health Organization (WHO). Correlates of vaccine-induced protection: methods and implications. 2013.

Return to footnote 79 referrer

Footnote 80

Cowling B.J., Lim W.W., Perera R., Fang V.J., Leung G.M., Peiris J.S.M., et al. Influenza Hemagglutination-inhibition Antibody Titer as a Mediator of Vaccine-induced Protection for Influenza B. Clin Infect Dis. 2019; 68(10): 1713-17. doi:10.1093/cid/ciy759

Return to footnote 80 referrer

Footnote 81

Gilbert P.B., Luedtke A.R. Statistical Learning Methods to Determine Immune Correlates of Herpes Zoster in Vaccine Efficacy Trials. J Infect Dis. 2018; 218(suppl_2): S99-s101. doi:10.1093/infdis/jiy421

Return to footnote 81 referrer

Footnote 82

Belshe R.B., Gruber W.C., Mendelman P.M., Mehta H.B., Mahmood K., Reisinger K., et al. Correlates of immune protection induced by live, attenuated, cold-adapted, trivalent, intranasal influenza virus vaccine. J Infect Dis. 2000; 181(3): 1133-7. doi:10.1086/315323

Return to footnote 82 referrer

Footnote 83

Bentebibel S.E., Lopez S., Obermoser G., Schmitt N., Mueller C., Harrod C., et al. Induction of ICOS+CXCR3+CXCR5+ TH cells correlates with antibody responses to influenza vaccination. Sci Transl Med. 2013; 5(176): 176ra32. doi:10.1126/scitranslmed.3005191

Return to footnote 83 referrer

Footnote 84

Cooper L.V., Boukary R.M., Aseffa A., Mihret W., Collard J.M., Daugla D., et al. Investigation of correlates of protection against pharyngeal carriage of Neisseria meningitidis genogroups W and Y in the African meningitis belt. PLoS One. 2017; 12(8): e0182575. doi:10.1371/journal.pone.0182575

Return to footnote 84 referrer

Footnote 85

Plotkin S.A. Correlates of protection induced by vaccination. Clin Vaccine Immunol. 2010; 17(7): 1055-65. doi:10.1128/cvi.00131-10

Return to footnote 85 referrer

Footnote 86

Bilcke J., Ogunjimi B., Hulstaert F., Van Damme P., Hens N., Beutels P. Estimating the age-specific duration of herpes zoster vaccine protection: a matter of model choice? Vaccine. 2012; 30(17): 2795-800. doi:10.1016/j.vaccine.2011.09.079

Return to footnote 86 referrer

Footnote 87

Li X., Zhang J.H., Betts R.F., Morrison V.A., Xu R., Itzler R.F., et al. Modeling the durability of ZOSTAVAX® vaccine efficacy in people ≥60 years of age. Vaccine. 2015; 33(12): 1499-505. doi:10.1016/j.vaccine.2014.10.039

Return to footnote 87 referrer

Footnote 88

Zoratti M.J., Pickard A.S., Stalmeier P.F.M., Ollendorf D., Lloyd A., Chan K.K.W., et al. Evaluating the conduct and application of health utility studies: a review of critical appraisal tools and reporting checklists. Eur J Health Econ. 2021; 22(5): 723-33. doi:10.1007/s10198-021-01286-0

Return to footnote 88 referrer

Footnote 89

Brazier J., Ara R., Azzabi I., Busschbach J., Chevrou-Séverac H., Crawford B., et al. Identification, Review, and Use of Health State Utilities in Cost-Effectiveness Models: An ISPOR Good Practices for Outcomes Research Task Force Report. Value Health. 2019; 22(3): 267-75. doi:10.1016/j.jval.2019.01.004

Return to footnote 89 referrer

Footnote 90

Peasgood T., Brazier J. Is Meta-Analysis for Utility Values Appropriate Given the Potential Impact Different Elicitation Methods Have on Values? PharmacoEconomics. 2015; 33(11): 1101-5. doi:10.1007/s40273-015-0310-y

Return to footnote 90 referrer

Footnote 91

Ungar W.J. Challenges in Health State Valuation in Paediatric Economic Evaluation. PharmacoEconomics. 2011; 29(8): 641-52. doi:10.2165/11591570-000000000-00000

Return to footnote 91 referrer

Footnote 92

Oliveira C., de Silva N.T., Ungar W.J., Bayoumi A.M., Avitzur Y., Hoch J.S., et al. Health-related quality of life in neonates and infants: a conceptual framework. Quality of Life Research. 2020; 29(5): 1159-68. doi:10.1007/s11136-020-02432-6

Return to footnote 92 referrer

Footnote 93

Ungar W.J., editor. Economic evaluation in child health: Oxford University Press; 2019.

Return to footnote 93 referrer

Footnote 94

Cookson R., Skarda I., Cotton-Barratt O., Adler M., Asaria M., Ord T. Quality adjusted life years based on health and consumption: A summary wellbeing measure for cross-sectoral economic evaluation. Health Econ. 2021; 30(1): 70-85. doi:10.1002/hec.4177

Return to footnote 94 referrer

Footnote 95

de Vries L.M., van Baal P.H.M., Brouwer W.B.F. Future Costs in Cost-Effectiveness Analyses: Past, Present, Future. Pharmacoeconomics. 2019; 37(2): 119-30. doi:10.1007/s40273-018-0749-8

Return to footnote 95 referrer

Footnote 96

Feeny D., Krahn M., Prosser L.A., Salomon J.A. Valuing Health Outcomes. Cost-Effectiveness in Health and Medicine. 2 ed. New York: Oxford University Press; 2016.

Return to footnote 96 referrer

Footnote 97

Nyman J.A. Should the consumption of survivors be included as a cost in cost-utility analysis? Health Econ. 2004; 13(5): 417-27. doi:10.1002/hec.850

Return to footnote 97 referrer

Footnote 98

Shiroiwa T., Fukuda T., Ikeda S., Shimozuma K. QALY and productivity loss: empirical evidence for "double counting". Value Health. 2013; 16(4): 581-7. doi:10.1016/j.jval.2013.02.009

Return to footnote 98 referrer

Footnote 99

Tilling C., Krol M., Tsuchiya A., Brazier J., Brouwer W. In or out? Income losses in health state valuations: a review. Value Health. 2010; 13(2): 298-305. doi:10.1111/j.1524-4733.2009.00614.x

Return to footnote 99 referrer

Footnote 100

Adarkwah C.C., Sadoghi A., Gandjour A. Should Cost-Effectiveness Analysis Include the Cost of Consumption Activities? AN Empirical Investigation. Health Econ. 2016; 25(2): 249-56. doi:10.1002/hec.3162

Return to footnote 100 referrer

Footnote 101

Basu A. Estimating Costs and Valuations of Non-Health Benefits in Cost-Effectiveness Analysis. Cost-Effectiveness in Health and Medicine. 2 ed. New York: Oxford University Press; 2016.

Return to footnote 101 referrer

Footnote 102

Salomon J.A., Trikalinos T.A., Sanders G.D., Mandelblatt J.S. Identifying and Quantifying the Consequences of Interventions. Cost-Effectiveness in Health and Medicine. 2 ed. New York: Oxford University Press; 2016.

Return to footnote 102 referrer

Footnote 103

Jacobs P., Budden A., Lee K.M. Guidance document fo rthe costing of health care resources in the Canadian setting. 2nd edition. 2016.

Return to footnote 103 referrer

Footnote 104

Feenstra T.L., van Baal P.H., Gandjour A., Brouwer W.B. Future costs in economic evaluation. A comment on Lee. J Health Econ. 2008; 27(6): 1645-9; discussion 50-1. doi:10.1016/j.jhealeco.2008.07.007

Return to footnote 104 referrer

Footnote 105

Morton A., Adler A.I., Bell D., Briggs A., Brouwer W., Claxton K., et al. Unrelated Future Costs and Unrelated Future Benefits: Reflections on NICE Guide to the Methods of Technology Appraisal. Health Econ. 2016; 25(8): 933-8. doi:10.1002/hec.3366

Return to footnote 105 referrer

Footnote 106

Meltzer D. Accounting for future costs in medical cost-effectiveness analysis. J Health Econ. 1997; 16(1): 33-64. doi:10.1016/s0167-6296(96)00507-3

Return to footnote 106 referrer

Footnote 107

Manns B., Meltzer D., Taub K., Donaldson C. Illustrating the impact of including future costs in economic evaluations: an application to end-stage renal disease care. Health Econ. 2003; 12(11): 949-58. doi:10.1002/hec.790

Return to footnote 107 referrer

Footnote 108

Canadian Institutes for Health Information (CIHI). National Health Expenditure Trends. 2021. Available from: https://www.cihi.ca/en/national-health-expenditure-trends.

Return to footnote 108 referrer

Footnote 109

Levy A.R., Sobolev B., James D., Barrable W., Clarke-Richardson P., Sullivan S.D., et al. The costs of change: direct medical costs of solid organ transplantation in British Columbia, Canada, 1995-2003. Value Health. 2009; 12(2): 282-92. doi:10.1111/j.1524-4733.2008.00445.x

Return to footnote 109 referrer

Footnote 110

Manns B.J., Mendelssohn D.C., Taub K.J. The economics of end-stage renal disease care in Canada: incentives and impact on delivery of care. Int J Health Care Finance Econ. 2007; 7(2-3): 149-69. doi:10.1007/s10754-007-9022-y

Return to footnote 110 referrer

Footnote 111

Centres for Disease Control and Prevention (CDC). Guidelines for Evaluating Surveillance Systems. 1988. Available from: https://www.cdc.gov/mmwr/preview/mmwrhtml/00001769.htm.

Return to footnote 111 referrer

Footnote 112

Coleman M.S., Garbat-Welch L., Burke H., Weinberg M., Humbaugh K., Tindall A., et al. Direct costs of a single case of refugee-imported measles in Kentucky. Vaccine. 2012; 30(2): 317-21. doi:10.1016/j.vaccine.2011.10.091

Return to footnote 112 referrer

Footnote 113

Dalton C.B., Haddix A., Hoffman R.E., Mast E.E. The cost of a food-borne outbreak of hepatitis A in Denver, Colo. Arch Intern Med. 1996; 156(9): 1013-6.

Return to footnote 113 referrer

Footnote 114

Dayan G.H., Ortega-Sánchez I.R., LeBaron C.W., Quinlisk M.P. The cost of containing one case of measles: the economic impact on the public health infrastructure--Iowa, 2004. Pediatrics. 2005; 116(1): e1-4. doi:10.1542/peds.2004-2512

Return to footnote 114 referrer

Footnote 115

Parker A.A., Staggs W., Dayan G.H., Ortega-Sánchez I.R., Rota P.A., Lowe L., et al. Implications of a 2005 measles outbreak in Indiana for sustained elimination of measles in the United States. N Engl J Med. 2006; 355(5): 447-55. doi:10.1056/NEJMoa060775

Return to footnote 115 referrer

Footnote 116

Institut national de sante publique du Quebec. Publications. n.d. Available from: https://www.inspq.qc.ca/en/publications/.

Return to footnote 116 referrer

Footnote 117

Musto R., Siever J.E., Johnston J.C., Seidel J., Rose M.S., McNeil D.A. Social equity in Human Papillomavirus vaccination: a natural experiment in Calgary Canada. BMC Public Health. 2013; 13: 640. doi:10.1186/1471-2458-13-640

Return to footnote 117 referrer

Footnote 118

Workplace Safety and Insurance Board. Operational policy manual: travel and related expenses. 2017.

Return to footnote 118 referrer

Footnote 119

Doherty M., Buchy P., Standaert B., Giaquinto C., Prado-Cohrs D. Vaccine impact: Benefits for human health. Vaccine. 2016; 34(52): 6707-14. doi:10.1016/j.vaccine.2016.10.025

Return to footnote 119 referrer

Footnote 120

Koopmanschap M.A., Rutten F.F., van Ineveld B.M., van Roijen L. The friction cost method for measuring indirect costs of disease. J Health Econ. 1995; 14(2): 171-89. doi:10.1016/0167-6296(94)00044-5

Return to footnote 120 referrer

Footnote 121

Krol M., Brouwer W. How to estimate productivity costs in economic evaluations. Pharmacoeconomics. 2014; 32(4): 335-44. doi:10.1007/s40273-014-0132-3

Return to footnote 121 referrer

Footnote 122

Pike J., Grosse S.D. Friction Cost Estimates of Productivity Costs in Cost-of-Illness Studies in Comparison with Human Capital Estimates: A Review. Appl Health Econ Health Policy. 2018; 16(6): 765-78. doi:10.1007/s40258-018-0416-4

Return to footnote 122 referrer

Footnote 123

Goeree R., O'Brien B.J., Blackhouse G., Agro K., Goering P. The valuation of productivity costs due to premature mortality: a comparison of the human-capital and friction-cost methods for schizophrenia. Can J Psychiatry. 1999; 44(5): 455-63. doi:10.1177/070674379904400505

Return to footnote 123 referrer

Footnote 124

Hank K., Stuck S. Volunteer work, informal help, and care among the 50+ in Europe: further evidence for 'linked' productive activities at older ages. Soc Sci Res. 2008; 37(4): 1280-91. doi:10.1016/j.ssresearch.2008.03.001

Return to footnote 124 referrer

Footnote 125

Bouwmans C., Krol M., Severens H., Koopmanschap M., Brouwer W., Hakkaart-van Roijen L. The iMTA Productivity Cost Questionnaire: A Standardized Instrument for Measuring and Valuing Health-Related Productivity Losses. Value Health. 2015; 18(6): 753-8. doi:10.1016/j.jval.2015.05.009

Return to footnote 125 referrer

Footnote 126

Pearce A. Productivity losses and how they are calculated. In: (CHERE) C.f.H.E.R.a.E., editor. 2016.

Return to footnote 126 referrer

Footnote 127

Statistics Canada. Average usual and actual hours worked in a reference week by type of work (full- and part-time), annual. 2021. Available from: https://www150.statcan.gc.ca/t1/tbl1/en/tv.action?pid=1410004301.

Return to footnote 127 referrer

Footnote 128

Statistics Canada. Income of individuals by age group, sex and income source, Canada, provinces and selected census metropolitan areas. 2021. Available from: https://www150.statcan.gc.ca/t1/tbl1/en/tv.action?pid=1110023901.

Return to footnote 128 referrer

Footnote 129

Prosser L.A. SMDM President’s Column: Gender bias and cost-effectiveness analysis. 2020. Available from: https://myemail.constantcontact.com/SMDM-Newsletter---Issue-4--September-2020.html?soid=1116971938232&aid=ywAYbDuCkUY.

Return to footnote 129 referrer

Footnote 130

Zhang W., Bansback N., Boonen A., Severens J.L., Anis A.H. Development of a composite questionnaire, the valuation of lost productivity, to value productivity losses: application in rheumatoid arthritis. Value Health. 2012; 15(1): 46-54. doi:10.1016/j.jval.2011.07.009

Return to footnote 130 referrer

Footnote 131

Grosse S.D., Pike J., Soelaeman R., Tilford J.M. Quantifying Family Spillover Effects in Economic Evaluations: Measurement and Valuation of Informal Care Time. Pharmacoeconomics. 2019; 37(4): 461-73. doi:10.1007/s40273-019-00782-9

Return to footnote 131 referrer

Footnote 132

Hoefman R.J., van Exel J., Brouwer W. How to include informal care in economic evaluations. Pharmacoeconomics. 2013; 31(12): 1105-19. doi:10.1007/s40273-013-0104-z

Return to footnote 132 referrer

Footnote 133

McKibbin W.J., Fernando R. Global macroeconomic scenarios of the COVID-19 pandemic. CAMA Working Papers. 2020.

Return to footnote 133 referrer

Footnote 134

Statistics Canada. Household spending, Canada, regions and provinces. 2021. Available from: https://www150.statcan.gc.ca/t1/tbl1/en/tv.action?pid=1110022201.

Return to footnote 134 referrer

Footnote 135

Statistics Canada. Unit of analysis: the household. 2020. Available from: https://www150.statcan.gc.ca/n1/pub/13-607-x/2016001/952-eng.htm.

Return to footnote 135 referrer

Footnote 136

Roed C., Omland L.H., Skinhoj P., Rothman K.J., Sorensen H.T., Obel N. Educational achievement and economic self-sufficiency in adults after childhood bacterial meningitis. Jama. 2013; 309(16): 1714-21. doi:10.1001/jama.2013.3792

Return to footnote 136 referrer

Footnote 137

Federal Reserve Bank of Cleveland. Educational attainment and employment. 2011. Available from: https://www.clevelandfed.org/en/newsroom-and-events/publications/economic-trends/2011-economic-trends/et-20110302-educational-attainment-and-employment.aspx.

Return to footnote 137 referrer

Footnote 138

US Bureau of Labor Statistics. Earnings and unemployment rates by educational attainment, 2020. 2021.

Return to footnote 138 referrer

Footnote 139

Gallagher-Mackay K., Srivastava P., Underwood K., Dhuey E., McCready L., Born K.B., et al. COVID-19 and education disruption in Ontario: emerging evidence on impacts. 2021.

Return to footnote 139 referrer

Footnote 140

Grimwood K., Anderson V.A., Bond L., Catroppa C., Hore R.L., Keir E.H., et al. Adverse outcomes of bacterial meningitis in school-age survivors. Pediatrics. 1995; 95(5): 646-56.

Return to footnote 140 referrer

Footnote 141

Klein E.Y., Schueller E., Tseng K.K., Morgan D.J., Laxminarayan R., Nandi A. The Impact of Influenza Vaccination on Antibiotic Use in the United States, 2010-2017. Open Forum Infect Dis. 2020; 7(7): ofaa223. doi:10.1093/ofid/ofaa223

Return to footnote 141 referrer

Footnote 142

Klugman K.P., Black S. Impact of existing vaccines in reducing antibiotic resistance: Primary and secondary effects. Proc Natl Acad Sci U S A. 2018; 115(51): 12896-901. doi:10.1073/pnas.1721095115

Return to footnote 142 referrer

Footnote 143

Pruden A., Larsson D.G.J., Amézquita A., Collignon P., Brandt K.K., Graham D.W., et al. Management options for reducing the release of antibiotics and antibiotic resistance genes to the environment. Environ Health Perspect. 2013; 121(8): 878-85. doi:10.1289/ehp.1206446

Return to footnote 143 referrer

Footnote 144

Bilcke J., Beutels P., Brisson M., Jit M. Accounting for methodological, structural, and parameter uncertainty in decision-analytic models: a practical guide. Med Decis Making. 2011; 31(4): 675-92. doi:10.1177/0272989x11409240

Return to footnote 144 referrer

Footnote 145

Briggs A.H., Weinstein M.C., Fenwick E.A., Karnon J., Sculpher M.J., Paltiel A.D. Model parameter estimation and uncertainty: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force--6. Value Health. 2012; 15(6): 835-42. doi:10.1016/j.jval.2012.04.014

Return to footnote 145 referrer

Footnote 146

Mahsin M.D., Deardon R., Brown P. Geographically dependent individual-level models for infectious diseases transmission. Biostatistics. 2020. doi:10.1093/biostatistics/kxaa009

Return to footnote 146 referrer

Footnote 147

Popinga A., Vaughan T., Stadler T., Drummond A.J. Inferring epidemiological dynamics with Bayesian coalescent inference: the merits of deterministic and stochastic models. Genetics. 2015; 199(2): 595-607. doi:10.1534/genetics.114.172791

Return to footnote 147 referrer

Footnote 148

Vreman R.A., Geenen J.W., Knies S., Mantel-Teeuwisse A.K., Leufkens H.G.M., Goettsch W.G. The Application and Implications of Novel Deterministic Sensitivity Analysis Methods. Pharmacoeconomics. 2021; 39(1): 1-17. doi:10.1007/s40273-020-00979-3

Return to footnote 148 referrer

Footnote 149

Sickbert-Bennett E.E., Weber D.J., Poole C., MacDonald P.D., Maillard J.M. Completeness of communicable disease reporting, North Carolina, USA, 1995-1997 and 2000-2006. Emerg Infect Dis. 2011; 17(1): 23-9. doi:10.3201/eid1701.100660

Return to footnote 149 referrer

Footnote 150

Mattoo S., Cherry J.D. Molecular pathogenesis, epidemiology, and clinical manifestations of respiratory infections due to Bordetella pertussis and other Bordetella subspecies. Clin Microbiol Rev. 2005; 18(2): 326-82. doi:10.1128/cmr.18.2.326-382.2005

Return to footnote 150 referrer

Footnote 151

Lipsitch M., Jha A., Simonsen L. Observational studies and the difficult quest for causality: lessons from vaccine effectiveness and impact studies. Int J Epidemiol. 2016; 45(6): 2060-74. doi:10.1093/ije/dyw124

Return to footnote 151 referrer

Footnote 152

Berger M.L., Sox H., Willke R.J., Brixner D.L., Eichler H.G., Goettsch W., et al. Good practices for real-world data studies of treatment and/or comparative effectiveness: Recommendations from the joint ISPOR-ISPE Special Task Force on real-world evidence in health care decision making. Pharmacoepidemiol Drug Saf. 2017; 26(9): 1033-9. doi:10.1002/pds.4297

Return to footnote 152 referrer

Footnote 153

Sterne J.A., Hernán M.A., Reeves B.C., Savović J., Berkman N.D., Viswanathan M., et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016; 355: i4919. doi:10.1136/bmj.i4919

Return to footnote 153 referrer

Footnote 154

Del Valle S.Y., Hyman J.M., Chitnis N. Mathematical models of contact patterns between age groups for predicting the spread of infectious diseases. Math Biosci Eng. 2013; 10(5-6): 1475-97. doi:10.3934/mbe.2013.10.1475

Return to footnote 154 referrer

Footnote 155

Funk S., Bansal S., Bauch C.T., Eames K.T., Edmunds W.J., Galvani A.P., et al. Nine challenges in incorporating the dynamics of behaviour in infectious diseases models. Epidemics. 2015; 10: 21-5. doi:10.1016/j.epidem.2014.09.005

Return to footnote 155 referrer

Footnote 156

Centres for Disease Control and Prevention (CDC). Hepatitis A Questions and Answers for Health Professionals. 2020. Available from: https://www.cdc.gov/hepatitis/hav/havfaq.htm.

Return to footnote 156 referrer

Footnote 157

Gravitt P.E. The known unknowns of HPV natural history. J Clin Invest. 2011; 121(12): 4593-9. doi:10.1172/jci57149

Return to footnote 157 referrer

Footnote 158

Campbell H., Andrews N., Borrow R., Trotter C., Miller E. Updated postlicensure surveillance of the meningococcal C conjugate vaccine in England and Wales: effectiveness, validation of serological correlates of protection, and modeling predictions of the duration of herd immunity. Clin Vaccine Immunol. 2010; 17(5): 840-7. doi:10.1128/cvi.00529-09

Return to footnote 158 referrer

Footnote 159

Lewnard J.A., Grad Y.H. Vaccine waning and mumps re-emergence in the United States. Sci Transl Med. 2018; 10(433). doi:10.1126/scitranslmed.aao5945

Return to footnote 159 referrer

Footnote 160

Centres for Disease Control and Prevention (CDC). About Meningococcal Vaccines. 2021. Available from: https://www.cdc.gov/vaccines/vpd/mening/hcp/about-vaccine.html.

Return to footnote 160 referrer

Footnote 161

Bojke L., Claxton K., Sculpher M., Palmer S. Characterizing structural uncertainty in decision analytic models: a review and application of methods. Value Health. 2009; 12(5): 739-49. doi:10.1111/j.1524-4733.2008.00502.x

Return to footnote 161 referrer

Footnote 162

Jackson C.H., Thompson S.G., Sharples L.D. Accounting for uncertainty in health economic decision models by using model averaging. J R Stat Soc Ser A Stat Soc. 2009; 172(2): 383-404. doi:10.1111/j.1467-985X.2008.00573.x

Return to footnote 162 referrer

Footnote 163

Hofmann B. Toward a procedure for integrating moral issues in health technology assessment. Int J Technol Assess Health Care. 2005; 21(3): 312-8. doi:10.1017/s0266462305050415

Return to footnote 163 referrer

Footnote 164

Krahn M.D., Bielecki J.M., Bremner K.E., de Oliveira C., Almeida N., Clement F., et al. Picturing ELSI+: a visual representation of ethical, legal, and social issues, and patient experiences in Health Technology Assessment in Canada. Int J Technol Assess Health Care. 2020; 36(1): 40-9. doi:10.1017/s0266462319000722

Return to footnote 164 referrer

Footnote 165

Public Health Agency of Canada (PHAC). Framework for ethical deliberation and decision-making in public health: a tool for public health practitioners, policy makers and decision-makers. Ottawa, Canada2017.

Return to footnote 165 referrer

Footnote 166

World Health Organization (WHO). Health equity. n.d. Available from: https://www.who.int/health-topics/health-equity#tab=tab_1.

Return to footnote 166 referrer

Footnote 167

Rawlins M., Barnett D., Stevens A. Pharmacoeconomics: NICE's approach to decision-making. Br J Clin Pharmacol. 2010; 70(3): 346-9. doi:10.1111/j.1365-2125.2009.03589.x

Return to footnote 167 referrer

Footnote 168

Cookson R., Griffin S., Norheim O.F., Culyer A.J. Distributional Cost-Effectiveness AnalysisQuantifying Health Equity Impacts and Trade-Offs: Quantifying Health Equity Impacts and Trade-Offs. Oxford, UK: Oxford University Press; 2020 2020-10.

Return to footnote 168 referrer

Footnote 169

Johri M., Norheim O.F. Can cost-effectiveness analysis integrate concerns for equity? Systematic review. Int J Technol Assess Health Care. 2012; 28(2): 125-32. doi:10.1017/s0266462312000050

Return to footnote 169 referrer

Footnote 170

Norheim O.F., Baltussen R., Johri M., Chisholm D., Nord E., Brock D., et al. Guidance on priority setting in health care (GPS-Health): the inclusion of equity criteria not captured by cost-effectiveness analysis. Cost Eff Resour Alloc. 2014; 12: 18. doi:10.1186/1478-7547-12-18

Return to footnote 170 referrer

Footnote 171

Cookson R., Mirelman A.J., Griffin S., Asaria M., Dawkins B., Norheim O.F., et al. Using Cost-Effectiveness Analysis to Address Health Equity Concerns. Value Health. 2017; 20(2): 206-12. doi:10.1016/j.jval.2016.11.027

Return to footnote 171 referrer

Footnote 172

Converge3. Equity, values, and health decisions in Ontario: a Converge3 methods report. 2019.

Return to footnote 172 referrer

Footnote 173

O'Neill J., Tabish H., Welch V., Petticrew M., Pottie K., Clarke M., et al. Applying an equity lens to interventions: using PROGRESS ensures consideration of socially stratifying factors to illuminate inequities in health. J Clin Epidemiol. 2014; 67(1): 56-64. doi:10.1016/j.jclinepi.2013.08.005

Return to footnote 173 referrer

Footnote 174

Powers M., Powers A.P.P.M., Faden R.R., Faden R.R. Social Justice: The Moral Foundations of Public Health and Health Policy: Oxford University Press; 2006.

Return to footnote 174 referrer

Footnote 175

Flannery B., Schrag S., Bennett N.M., Lynfield R., Harrison L.H., Reingold A., et al. Impact of childhood vaccination on racial disparities in invasive Streptococcus pneumoniae infections. Jama. 2004; 291(18): 2197-203. doi:10.1001/jama.291.18.2197

Return to footnote 175 referrer

Footnote 176

Public Health Agency of Canada (PHAC). Key health inequalities in Canada: a national portrait. 2018.

Return to footnote 176 referrer

Footnote 177

Castle P.E., Zhao F.-H. Population Effectiveness, Not Efficacy, Should Decide Who Gets Vaccinated Against Human Papillomavirus via Publicly Funded Programs. J Infect Dis. 2011; 204(3): 335-7. doi:10.1093/infdis/jir287

Return to footnote 177 referrer

Footnote 178

Blakely T., Kvizhinadze G., Karvonen T., Pearson A.L., Smith M., Wilson N. Cost-effectiveness and equity impacts of three HPV vaccination programmes for school-aged girls in New Zealand. Vaccine. 2014; 32(22): 2645-56. doi:10.1016/j.vaccine.2014.02.071

Return to footnote 178 referrer

Footnote 179

Culyer A.J. Equity of what in healthcare? Why the traditional answers don't help policy--and what to do in the future. Healthc Pap. 2007; 8 Spec No: 12-26. doi:10.12927/hcpap.2007.19216

Return to footnote 179 referrer

Footnote 180

Menzies R.I., Singleton R.J. Vaccine preventable diseases and vaccination policy for indigenous populations. Pediatr Clin North Am. 2009; 56(6): 1263-83. doi:10.1016/j.pcl.2009.09.006

Return to footnote 180 referrer

Footnote 181

Veras M., Zakus D. Equity Aspects of Canadian Immunization Programs: Differences within and between countries. 2011. 2011: 22. doi:10.5195/hcs.2011.45

Return to footnote 181 referrer

Footnote 182

Brearley L., Eggers R., Steinglass R., Vandelaer J. Applying an equity lens in the Decade of Vaccines. Vaccine. 2013; 31 Suppl 2: B103-7. doi:10.1016/j.vaccine.2012.11.088

Return to footnote 182 referrer

Footnote 183

Quilici S., Smith R., Signorelli C. Role of vaccination in economic growth. J Mark Access Health Policy. 2015; 3: 10.3402/jmahp.v3.27044. doi:10.3402/jmahp.v3.27044

Return to footnote 183 referrer

Footnote 184

Avanceña A.L.V., Prosser L.A. Examining Equity Effects of Health Interventions in Cost-Effectiveness Analysis: A Systematic Review. Value Health. 2021; 24(1): 136-43. doi:10.1016/j.jval.2020.10.010

Return to footnote 184 referrer

Footnote 185

Verguet S., Kim J.J., Jamison D.T. Extended Cost-Effectiveness Analysis for Health Policy Assessment: A Tutorial. Pharmacoeconomics. 2016; 34(9): 913-23. doi:10.1007/s40273-016-0414-z

Return to footnote 185 referrer

Footnote 186

van Hoek A.J., Melegaro A., Gay N., Bilcke J., Edmunds W.J. The cost-effectiveness of varicella and combined varicella and herpes zoster vaccination programmes in the United Kingdom. Vaccine. 2012; 30(6): 1225-34. doi:10.1016/j.vaccine.2011.11.026

Return to footnote 186 referrer

Footnote 187

Crosignani P., De Stefani A., Fara G.M., Isidori A.M., Lenzi A., Liverani C.A., et al. Towards the eradication of HPV infection through universal specific vaccination. BMC Public Health. 2013; 13: 642. doi:10.1186/1471-2458-13-642

Return to footnote 187 referrer

Footnote 188

Tinghög G. Discounting, preferences, and paternalism in cost-effectiveness analysis. Health Care Anal. 2012; 20(3): 297-318. doi:10.1007/s10728-011-0188-6

Return to footnote 188 referrer

Footnote 189

Husereau D., Drummond M., Petrou S., Carswell C., Moher D., Greenberg D., et al. Consolidated Health Economic Evaluation Reporting Standards (CHEERS) statement. Eur J Health Econ. 2013; 14(3): 367-72. doi:10.1007/s10198-013-0471-6

Return to footnote 189 referrer

Footnote 190

Government of Canada. National Advisory Committee on Immunization (NACI): membership and representation. 2021. Available from: https://www.canada.ca/en/public-health/services/immunization/national-advisory-committee-on-immunization-naci/naci-membership-representation.html.

Return to footnote 190 referrer

Footnote 191

Government of Canada. National Advisory Committee on Immunization (NACI): methods and process. 2021. Available from: https://www.canada.ca/en/public-health/services/immunization/national-advisory-committee-on-immunization-naci/methods-process.html.

Return to footnote 191 referrer

Page details

Date modified:: 2022-05-03

National Advisory Committee on Immunization (NACI): Guidelines for the economic evaluation of vaccination programs in Canada

On This Page

Acknowledgements

NACI economic guidelines task group members and liaisons

Reviewers

Contributors

Conflicts of interest declaration

Abbreviations

Glossary

Introduction

Guideline statements

1. Decision problem

2. Types of evaluations

3. Study populations

4. Comparators

5. Perspectives

6. Time horizon

7. Discounting

8. Modelling

9. Effectiveness

10. Measurement and valuation of health

11. Resource use and costs

12. Analysis

13. Uncertainty

14. Equity

15. Reporting

Guidelines in detail

1. Decision problem

2. Types of evaluations

3. Study populations

4. Comparators

5. Perspectives

Publicly funded health system perspective

Societal perspective

6. Time horizon

7. Discounting

8. Modelling

Model structure and attributes

Endogenous vs. exogenous infection rate

Other attributes

Deterministic versus stochastic

Aggregate versus individual-based

Open versus closed

Discrete versus continuous time

Model calibration

Model validation

9. Effectiveness

Assessing estimates of vaccine effectiveness

Data synthesis

Surrogate outcomes

Extrapolation

10. Measurement and valuation of health

Health utility data

Quality-adjusted life years in societal perspective economic evaluations

11. Resource use and costs

Health system sector

Healthcare costs

Formal caregiving

Future healthcare costs

Public health costs

Healthcare costs not funded by the health system

Non-healthcare areas

Direct out-of-pocket costs

Productivity

Individual productivity

Informal caregiver productivity

Macroeconomic consequences

Non-Medical consumption

Education

Social services

Environment

Other areas

12. Analysis

13. Uncertainty

Parameter uncertainty

Structural uncertainty

Methodological uncertainty

Value-of-information

14. Equity

15. Reporting