National Advisory Committee on Immunization (NACI): Guidelines for the economic evaluation of vaccination programs in Canada

1^st Edition: April 2023

Acknowledgements
In memory of Murray Krahn
Conflicts of interest declaration
Abbreviations
Glossary
Foreword
Introduction
Guideline statements
Guidelines in detail
Appendix 1: Impact inventory table
- Table 1: Impact inventory table for economic evaluations of vaccination strategies
Appendix 2: Reference case specifications
- Table 2: Recommendations for reference case analyses
References for foreword
References

Acknowledgements

NACI Economic guidelines task group members and liaisons

Task group members were responsible for selecting chapters, identifying and discussing key issues related to each chapter, drafting sections, reviewing all draft chapters, reviewing the draft consolidated report, addressing peer review and stakeholder feedback, reviewing, and approving the final version of guidelines.

Academic members:
Beate Sander, PhD
Co-chair of Task Group and NACI member
Senior Scientist
University Health Network
Toronto, Ontario

Murray Krahn, MD
Co-chair of Task Group (January 2019 - December 2021)
Professor & Director, Toronto Health Economics and Technology Assessment (THETA) Collaborative
University of Toronto
Toronto, Ontario

Stirling Bryan, PhD
Professor
University of British Columbia
Vancouver, British Columbia

Werner Brouwer, PhD
Professor
Erasmus University Rotterdam
Rotterdam, Netherlands

Mark Jit, PhD
Professor
London School of Hygiene and Tropical Medicine
London, United Kingdom

Sachiko Ozawa, PhD
Associate Professor
University of North Carolina at Chapel Hill
Chapel Hill, North Carolina

Lisa Prosser, PhD
Professor
University of Michigan
Ann Arbor, Michigan

Liaison members:
Karen M. Lee, MA
CADTH Liaison
Director, Health Economics
Canadian Agency for Drugs & Technologies in Health (CADTH)
Ottawa, Ontario

Monika Naus, MD
Provincial/ Territorial Liaison and NACI Liaison
Medical Director, Communicable Diseases & Immunization Service
British Columbia Centre for Disease Control
Vancouver, British Columbia

PHAC members:
Man Wah Yeung, MSc
Senior Health Economist
PHAC
Toronto, Ontario

Austin Nam, PhD
Senior Health Economist
PHAC
Toronto, Ontario

Ashleigh Tuite, PhD
Manager
PHAC
Toronto, Ontario

Althea House, BScN
Manager
PHAC
Ottawa, Ontario

Matthew Tunis, PhD
Executive Secretary
PHAC
Ottawa, Ontario

Contractor:
Nina Lathia, PhD
Health Economist
No affiliation
Toronto, Ontario

Reviewers

PHAC would like to acknowledge the academic peer-reviewers and NACI Economics Task Group (sister task group) for their peer-review of the full guidelines. The sister task group's mandate is to develop processes related to health economics for NACI.

Academic peer-reviewers:
Lauren Cipriano, PhD
Western University
London, Ontario

Kim Dalziel, PhD
University of Melbourne
Melbourne, Australia

David Fisman, MD
University of Toronto
Toronto, Ontario

Susan Griffin, PhD
University of York
York, England

Ava John-Baptiste, PhD
Western University
London, Ontario

Christopher McCabe, PhD
Kate Harback, PhD
Erin Kirwin, MA
Ellen Rafferty, PhD
Jeff Round, PhD
Sasha van Katwyk, MSc

On behalf of
Institute of Health Economics
Edmonton, Alberta

Bohdan Nosyk, PhD
Simon Fraser University
Burnaby, British Columbia

Wendy Ungar, PhD
The Hospital for Sick Children Research Institute
Toronto, Ontario

Members of NACI Economics Task Group:
Beate Sander, PhD
University Health Network
Toronto, Ontario

Philippe De Wals, MD, PhD
Université Laval
Québec City, Québec

David Fisman, MD
University of Toronto
Toronto, Ontario

Kristin Klein, MD
Alberta Health Services
Edmonton, Alberta

Joanne Langley, MD
Dalhousie University
Halifax, Nova Scotia

Monika Naus, MD
British Columbia Centre for Disease Control
Vancouver, British Columbia

Ellen Rafferty, PhD
Institute of Health Economics
Edmonton, Alberta

Bernice Tsoi, PhD
Canadian Agency for Drugs & Technologies in Health (CADTH)
Ottawa, Ontario

Contributors

PHAC would like to acknowledge the following individuals for their contributions:

Academic contributors:

Lisa Schwartz, PhD - contributed to the development of the equity chapter
Richard Cookson, DPhil - contributed to the development of the equity chapter
Pieter van Baal, PhD - contributed to the development of the resource use and costs chapter

Indigenous Services Canada contributors:

Kendra Hardy, MA - provided review of the foreword, equity chapter, and types of evaluation chapter
Melanie Knight, RN - provided review of the equity chapter and types of evaluation chapter
Andrea Monahan, RN - provided review of the foreword
Denise Hamilton, MPA - provided review of the foreword
Tom Wong, MD - provided review of the foreword, equity chapter, and types of evaluation chapter
Kim Daly, MN - provided review of the foreword
Pamela Wolfe-Roberge, BA - provided review of the equity chapter, and types of evaluation chapter

Public Health Ethics Consultative Group contributors:

Diego Silva, PhD - provided review of the equity chapter
Béatrice Godard, PhD - provided review of the equity chapter
Boluwaji Ogunyemi, MD - provided review of the equity chapter
Cassandra Opikokew Wajuntah, PhD (c) - provided review of the equity chapter
Maxwell J. Smith, PhD - provided review of the equity chapter
A.M. Viens, PhD - provided review of the equity chapter
Alice Virani, PhD - provided review of the equity chapter

NACI contributors:

Sheela Ramanathan, PhD - provided review of the effectiveness chapter
Kyla Hildebrand, MD - provided review of the effectiveness chapter
Matthew Miller, PhD - provided review of the effectiveness chapter

PHAC contributors:

Alexandra Cernat, MSc - provided support on referencing, collating public feedback, and copyediting
Erica Tice, MSc - provided review of the foreword
Shainoor Ismail, MD - provided review of the equity chapter
Angela Sinilaite, MPH - provided review of the equity chapter
Amanda Sumner, MA - provided technical support early in the project
Christine Mauviel, BA - provided project management support
Chantale Tremblay, BSc - provided project management support
Jennifer Daniel - provided project management support
Siobhan Kelly, BA - provided project management support
Caroline Rodriguez-Charette, BJ - provided project management support
PHAC would like to acknowledge the discussions and review by NACI members, liaisons, and ex-officios. NACI members approved the final version of the guidelines.

NACI members:

Shelley Deeks (Chair), MD, Nova Scotia Health and Wellness
Robyn Harrison (Vice-chair), MD, Alberta Health Services
Melissa Andrew, MD, Dalhousie University
Julie Bettinger, PhD, British Columbia Children's Hospital Research Institute
Nicholas Brousseau, MD, Institut national de santé publique du Québec
Philippe De Wals, MD, PhD, Université Laval
Hélène Decaluwe, MD, PhD, University of Montréal
Eve Dubé, PhD, Université Laval
Vinita Dubey, MD, University of Toronto
Kyla Hildebrand, MD, British Columbia Children's Hospital
Kristin Klein, MD, Alberta Health Services
Miranda O' Driscoll, RN, Disease Control Specialist for Newfoundland and Labrador
Jesse Papenburg, MD, Montreal Children's Hospital
Anne Pham-Huy, MD, Children's Hospital of Eastern Ontario
Beate Sander, PhD, University Health Network
Sarah Wilson, MD, Public Health Ontario

Former NACI members:

Caroline Quach-Thanh (former Chair), MD, University of Montreal
Natalie Dayneka, PharmD, Children's Hospital of Eastern Ontario
Soren Gantt, MD, University of British Columbia
Coleman Rotstein, MD, University of Toronto
Susan Smith, RN, Government of Alberta
Wendy Vaudry, MD, University of Alberta
Martin Lavoie, MD, Fraser Health Authority
Nadine Sicard, MD, Ministère de la Santé et des Services sociaux
Marina Salvadori, MD, Children's Hospital of Western Ontario

NACI liaisons:

Lea Bill, RN, Canadian Indigenous Nurses Association
Lucie Marisa Bucci, MA, Canadian Public Health Association
Eliana Castillo, MD, Society of Obstetricians and Gynaecologists of Canada
Jeannette Comeau, MD, Association of Medical Microbiology and Infectious Disease Control
Martin Lavoie, MD, Council of Chief Medical Officers of Health
Dorothy Moore, MD, Canadian Paediatric Society
Monika Naus, MD, Canadian Immunization Committee
Amanda Ung, BScPhm, Canadian Pharmacists Association
Melanie Osmack, MA, Indigenous Physicians Association of Canada
Jessica MacNeil, MPH, Centers for Disease Control and Prevention

Former NACI liaisons:

Evan Adams, MD, Indigenous Physicians Association of Canada
Loretta Dupuis, RN, Canadian Nurses Association
Jia Hu, MD, College of Family Physicians of Canada
Julie Emili, MD, College of Family Physicians of Canada
Reka Gustafson, MD, Council of Chief Medical Officers of Health
Amanda Cohn, MD, Centers for Disease Control and Prevention
Kathleen Dooling, MD, Centers for Disease Control and Prevention
Jason Brophy, MD, Canadian Association of Immunization Research and Evaluation
Deshayne Fell, PhD, Canadian Association of Immunization Research and Evaluation
Philip Emberley, PharmD, Canadian Pharmacists Association
Sarah Funnell, MD, Indigenous Physicians Association of Canada

Ex-Officio Representatives:

Erin E. Henry, BScN, Public Health Agency of Canada
Vincent Beswick-Escanlar, MD, National Defence and the Canadian Armed Forces
Mireille Lacroix, LLM, Public Health Agency of Canada
Celia Lourenco, PhD, Health Canada
Susanna Ogunnaike-Cooke, MSc, Public Health Agency of Canada
Kelly Robinson, MSc, Health Canada
Michael Routledge, MD, National Microbiology Laboratory
Tom Wong, MD, Indigenous Services Canada
Patrick Fandja, MSc, Health Canada
Diane MacDonald, MSc, Public Health Agency of Canada

Former Ex-Officio Representatives:

Gina Charos, Public Health Agency of Canada
Celia Lourenco, PhD, Health Canada
Robert Pless, MD, Health Canada
Kelly Robinson, MSc, Health Canada
Jim Gallivan, PhD, Health Canada
Guillaume Poliquin, MD, PhD, National Microbiology Laboratory
Jennifer Pennock, MSc, Public Health Agency of Canada

In memory of Murray Krahn

On the passing of a friend, colleague, mentor, and giant in the field
1957 - 2022

Dr. Murray Krahn passed away on July 1st, 2022 at the age of 65. At the time of his death, we were six months away from having the first edition of these Guidelines finalized - work which began in January of 2019. Murray had a long and distinguished history, not only as a staff physician caring for his patients and training new residents, but also as a global leader in health technology assessment.

After living in Kodaikanal, India; Seattle, Washington; and Fresno, California in his youth, Murray's educational training took him to Winnipeg, Manitoba to pursue degrees in philosophy and medicine, and to Toronto to pursue training in internal medicine. During his residency, he developed an interest in medical decision-making, and completed an MSc in Clinical Epidemiology at McMaster University, followed by a research fellowship in Clinical Decision-Making at Tufts University.

Murray held many leadership roles including serving as president of the Society for Medical Decision Making (SMDM) and founding the Toronto Health Economics and Technology Assessment (THETA) Collaborative, where he was director for 13 years. He was also a Professor at the Dalla Lana School of Public Health, and in the Faculties of Medicine and Pharmacy at the University of Toronto, a Senior Scientist at the Toronto General Research Institute, and an Adjunct Scientist at ICES (formerly the Institute for Clinical Evaluative Sciences). In these roles, he championed health technology assessments and economic evaluations. His impact was recognized by a number of prestigious awards including a Tier 1 Canada Research Chair in Health Technology Assessment, the Career Achievement Award from the SMDM, the Distinguished Service Award from the SMDM, the Dr. Jill M. Sanders Award of Excellence in Health Technology Assessment from the Canadian Agency for Drugs and Technologies in Health, the Senior Researcher Award from the Association of Faculties of Pharmacy of Canada, the David Sackett Senior Investigator Award from the Canadian Society for Internal Medicine, a Canadian Institutes for Health Research Investigator Award, and the Arthur Bond Fellowship in Innovative Health Systems Research from Physicians' Service Incorporated.

During the development of these guidelines, Murray co-chaired the Economic Guidelines Task Group, reviewing early drafts of the chapters as they came together, and engaging in friendly philosophical debates. He brought thoughtfulness and kindness to our meetings, sprinkled with humour.

It is evident that Murray's career gave him immense joy and meaning. As noted by his family, Murray treasured his friendships and collaborations with colleagues and students. He was passionate about mentoring and training the next generation of researchers and medical staff. He will be deeply missed.

Conflicts of interest declaration

As part of standard procedures for identifying and addressing affiliations and interests, Economic Guidelines Task Group members completed individual disclosure forms, which were assessed by PHAC to ensure no undue influence or perceived conflict of interest.

Beate Sander had no declarations of interests and affiliations to make.
Murray Krahn had no declarations of interests and affiliations to make.
Stirling Bryan had no declarations of interests and affiliations to make.
Werner Brouwer had no declarations of interests and affiliations to make.
Mark Jit declared receiving research funding from non-profit organizations (European Commission and the World Health Organization).
Karen M. Lee had no declarations of interests and affiliations to make.
Monika Naus had no declarations of interests and affiliations to make.
Sachiko Ozawa declared research funding from Merck Sharp & Dohme Corporation, the Bill & Melinda Gates Foundation, the North Carolina Department of Health and Human Services and the National Cancer Institute.
Lisa Prosser had no declarations of interests and affiliations to make.

Abbreviations

AQoL: Assessment of Quality of Life instrument
CADTH: Canadian Agency for Drugs and Technologies in Health
CBA: Cost-benefit analysis
CEAC: Cost-effectiveness acceptability curve
CEAF: Cost-effectiveness acceptability frontier
CHEERS II: Consolidated Health Economic Evaluation Reporting Standards II
CHU9D: Child Health Utility 9-Dimensions instrument
CoP: Correlate of protection
CUA: Cost-utility analysis
DSA: Deterministic sensitivity analysis
EEFA: Ethics, equity, feasibility, and acceptability
EQ-5D: EuroQol 5-Dimensions instrument
EQ-5D-Y: EuroQol 5-Dimensions Youth instrument
HPV: Humanpapilloma virus
HRQoL: Health-related quality of life
HUI: Health Utilities Index instrument
ICER: Incremental cost-effectiveness ratio
IQ: Inuit Qaujimajatuqangit
NACI: National Advisory Committee on Immunization
PedsQL: Pediatric Quality of Life inventory
QALY: Quality-adjusted life year
RCT: Randomized controlled trial
SF-6D: Short Form 6-Dimensions instrument
SIS: Susceptible-infectious-susceptible
SIR: Susceptible-infectious-removed (immune)
TB: Tuberculosis

Glossary

Age-shifting of infection: A phenomenon that can occur when a particular age cohort of the population is vaccinated, which reduces the force of infection in that cohort, and shifts the average age at infection.
Agent-based model: See definition for individual-based model.
Aggregate model (or population-based model): A model in which groups of individuals are assigned to compartments (or health states) based on their health status or other relevant characteristics. Individuals move from one compartment to another according to parameter values defined at the aggregate level and the model records the number of individuals in each compartment over time.
Basic reproduction number: The average number of secondary cases infected by an infectious person in a completely susceptible population and in the absence of interventions.
Catch-up strategy: A strategy of vaccinating individuals who did not receive a particular vaccination at the recommended age. This strategy may be used in individuals who have not been previously eligible for vaccination, who have missed a scheduled vaccine dose, or who have not completed a vaccine series.
Coverage: The percentage of eligible individuals who have received a particular vaccine.
Canadian Agency for Drugs and Technologies in Health (CADTH): Independent, not-for-profit organization tasked with providing Canada's healthcare decision-makers with evidence related to the optimal use of drugs and medical devices within the public healthcare system.
Closed (population) model: A model that follows a fixed cohort or cohorts of individuals. Individuals are not able to enter or exit the model over time.
Community immunity (or herd immunity): A state where a sufficient proportion of a population is immune to an infectious disease, either from vaccination or prior infection, thereby preventing outbreaks from occurring and making spread between individuals less likely (i.e., when the basic reproduction number R0 falls below zero). The term is commonly used to refer to the indirect protection unvaccinated individuals receive due to the presence of immune individuals in a population.
Confounding bias: A distortion in the estimate of the relationship between an exposure and an outcome in a study, resulting from a third variable, the confounder, which is related both to the exposure and the outcome.
Consumption: Goods and service bought by individuals.
Continuous (time) model: A model in which events can occur at any point in time.
Correlate of protection (CoP): An immune biomarker that predicts vaccine efficacy in vaccinated individuals and can be used as a surrogate endpoint in studies of vaccine efficacy or effectiveness.
Cost-effectiveness acceptability curve (CEAC): A graphic summary of the uncertainty in results of an economic evaluation, where the probability that an intervention is cost-effective is plotted against a range of cost-effectiveness thresholds.
Cost-effectiveness acceptability frontier (CEAF): A graphical summary of the uncertainty in results of an economic evaluation, which indicates the strategy that is economically preferred at different cost-effectiveness threshold values; and the probability of that strategy being cost-effective. As the threshold increases, the economically preferred treatment may change (switch point). CEAFs are most useful when three or more alternatives are being compared, in which case there may be two or more switch points at different threshold values.
Cost-benefit analysis (CBA): An economic evaluation in which health outcomes are valued in monetary terms.
Cost-effectiveness analysis (CEA): An economic evaluation in which health outcomes are measured as natural units (e.g., infections avoided).
Cost-utility analysis (CUA): An economic evaluation in which health outcomes are valued as quality-adjusted life years (or other generic measure of health-related utility). It is sometimes referred to as a cost-effectiveness analysis (CEA), or "CEA with quality-adjusted life-years (QALYs)". This form of economic evaluation is favoured by Public Health decision-makers in Canada.
Decision problem: An explicit statement of the interventions, study populations, outcome measures, and perspective adopted in an economic evaluation, related specifically to the decision(s) that the evaluation is designed to inform.
Deterministic model: A model that describes what happens on average and in which events cannot occur randomly (by chance). For a defined set of parameters and starting conditions, these models will always generate the same results each time they are run.
Deterministic sensitivity analysis (DSA): A method used to explore uncertainty in results of a model-based economic evaluation, where one or more parameters are changed across a pre-specified range while holding the remaining parameters fixed, to determine the extent to which the parameter values impact the results of the analysis.
Discount rate: Costs and outcomes occurring in the future are generally considered to be valued less than those occurring presently, and so they are discounted in an economic evaluation to ascertain their present-day value. The factor by which costs and health outcomes are discounted is expressed as the discount rate.
Discrete (time) model: A model in which events can only occur at pre-specified points in time.
Disease control: The state in which incidence, prevalence, morbidity, or mortality of a particular disease has been reduced locally but continued efforts are required to maintain this reduction.
Disease elimination: Local incidence of a particular infection has been reduced to a level below that necessary to sustain ongoing transmission in a given geographic area, but continued efforts are required to maintain this reduction.
Disease eradication: The incidence of an infection has been permanently reduced to zero worldwide, the causative organism is no longer present in the wild and efforts are no longer required to maintain this reduction.
Distributional cost-effectiveness analysis: An extension to the conventional CEA framework that quantifies the distributional impacts of health interventions based on different equity criteria such as socioeconomic status or disease severity.
Dominance: Refers to a scenario in which a strategy results in greater benefits and fewer costs compared to its alternative. The strategy resulting in greater benefits and fewer costs is said to dominate the alternative strategy.
Dose completion: For multi-dose vaccination regimes, the receipt of the required number of doses of the vaccine during a specified time period.
Dynamic (transmission) model: A model in which the force of infection can vary over time. Incidence is a function of the number (or proportion) of infected and susceptible individuals and the transmissibility of the virus.
Effectiveness: The extent to which an intervention provides the desired outcome(s) in the relevant study population in a real-world setting.
Effective reproduction number: The average number of secondary cases infected by an infectious person in a population where some of the individuals are immune, due to vaccination or infection and/ or in the presence interventions to reduce transmission.
Efficacy: The benefit of an intervention produced in an experimental and controlled setting, such as in a randomized controlled trial (RCT). To be distinguished from effectiveness (see definition above).
Efficiency: The extent to which the greatest benefit from interventions is maximized using the available resources, or to which a given benefit is obtained minimizing the resource use and costs.
Epidemiologic equilibrium: A situation where the rate of new infections circulating in a population is equal to the rate of recovery from the infection, resulting in a stable or unchanging state.
Expected value of perfect information (EVPI): The value, in monetary terms, of removing all uncertainty about the parameters in the analysis (that is, of obtaining perfect information on all parameters). Estimates are calculated using a pre-specified cost-effectiveness threshold, which can be varied in sensitivity analyses. If EVPI exceeds the cost of further research, it can be considered cost-effective to collect more information (data) to remove uncertainty from the model parameters. EVPI can also be expressed for the total population who stand to benefit over the expected lifetime of the intervention (known as population EVPI). The use of EVPI assumes that the funding decision is entirely based on whether the intervention is cost-effective at a specific cost-effectiveness threshold.
Expected value of partial perfect information (EVPPI): The value, in monetary terms, of removing uncertainty of one or more inputs to an economic model (that is, of obtaining perfect information on a subset of parameters). Estimates are calculated using a pre-specified cost-effectiveness threshold, which can be varied in sensitivity analyses. If EVPPI exceeds the cost of further research, it can be considered cost-effective to collect more information (data) to remove uncertainty from the subset of model parameters. The use of EVPPI assumes that the funding decision is entirely based on whether the intervention is cost-effective at a specific cost-effectiveness threshold.
Extended dominance: When comparing three or more strategies, extended dominance occurs when the total cost of strategy (A) is more than strategy (B), which is more than strategy (C); and the incremental cost-effectiveness ratio of strategy (A) compared to strategy (C) is less than that of strategy (B) compared to strategy (C). Thus, a decision-maker should eliminate strategy (B) from further consideration. This is because delivering a more effective strategy (A) to a fraction of the population and an inferior strategy to the remainder of the population dominates delivering a strategy eliminated by extended dominance (B) to the entire population.
Extended cost-effectiveness analysis: An extension to the conventional cost-effectiveness analysis (CEA) framework that quantifies the distributional impacts of health interventions based on equity criteria as well as financial risk protection. Financial risk protection refers to households' access to needed healthcare services without experiencing undue financial hardship.
Externalities: Costs and consequences of an intervention such as a vaccination program that fall on other members of the population beyond those receiving the intervention (e.g., community immunity, age-shifting of disease).
Equity: See definition for health equity.
First-order uncertainty: Uncertainty related to random variability in outcomes between identical people. This type of uncertainty is also referred to as stochastic uncertainty.
Force of infection: The rate at which susceptible individuals become infected per unit time. It is a function of the number of infectious individuals in the population at a given time and the transmissibility of the infection.
Health: A state of complete physical, mental and social well-being and not merely the absence of disease or infirmity. This definition is from the World Health Organization constitution.
Health technology: The application of organized knowledge and skills in the form of devices, medicines, vaccines, procedures, and systems developed to solve a health problem and improve quality of lives. This definition is from the World Health Organization.
Health technology assessment (HTA): The multi-disciplinary process that uses explicit methods to determine the value of a health technology at different points in its lifecycle. The purpose is to inform decision-making in order to promote an equitable, efficient, and high-quality health system.
Health equity: The absence of unfair and avoidable or remediable differences in health among population groups defined by any relevant characteristic (e.g., medical, social, economic, demographic, geographic). Horizontal equity refers to individuals with like characteristics (of ethical relevance) being treated the same way, while vertical equity allows for individuals with different characteristics (of ethical relevance) to be treated differently in order to achieve more equitable outcomes.
Health-related quality of life (HRQoL): A multidimensional concept which describes the physical, role functioning, social, and psychological aspects of well-being and functioning.
Health system: For the purposes of these guidelines, the term "health system" refers to both healthcare clinical services and Public Health.
Health utility: A measure of health-related quality of life that represents preference values that individuals attach to their overall health status. Conventionally the valuations are anchored by 0 (representing a health state equivalent to being dead) and 1 (representing a health state equivalent to perfect health). Health utilities are also referred to as preference-based measures of health-related quality of life.
Herd immunity: See definition for community immunity.
Heterogeneity: Differences between individuals that can, in part, be explained. This differs from the random chance that individuals with the same underlying characteristics will experience a different outcome.
Incremental costs: Difference in expected costs associated with the use of an intervention compared with the use of an alternative. This is a key output of an economic evaluation.
Incremental cost-effectiveness ratio (ICER): A ratio that is calculated by dividing the difference in expected costs by the difference in expected health outcomes (or effects) between two alternatives being compared in an economic evaluation. The comparator usually represents the current standard of care.
Incubation period: The time from infection to onset of clinical disease.
Indigenous Peoples: The earliest known people groups of any land around the world. For the purposes of these Guidelines, the term 'Indigenous Peoples' refers to individuals who are First Nations, Inuit, and Métis.
Infectious period: The time from the end of the latent or pre-infectious period until the host is no longer able to transmit the infection to other individuals.
Individual-based model (including microsimulation and agent-based model): A model that simulates the progression of each individual. Individual-based models can be with or without interactions (e.g., interactions with other individuals or with the environment, including the availability of resources). Individual-based models include individual sampling models, individual event history models, and discrete event simulations. In individual-level decision-analytic models, the probability of transition, risk for events, or time-to-event apply to each individual. Each simulation represents an individual, introducing heterogeneity and stochasticity. In individual-based infectious disease models, the biologic structure, demographic characteristics, and risk factors are applied at the individual level, so that natural history, risk level, and contacts/interactions can vary between people.
Intergenerational equity: The concept that people of different generations should benefit equitably from policy decisions such as expenditures on vaccination programs.
Latent period: The time period from when a host acquires an infection until they are able to transmit it to another host. It is sometimes referred to as the pre-infectious period.
Methodologic uncertainty: Uncertainty related to the different methods that can be used to conduct an economic evaluation.
Microsimulation: See definition for individual-based model.
National Advisory Committee on Immunization (NACI): A national advisory committee of experts in the fields of pediatrics, infectious diseases, immunology, pharmacy, nursing, epidemiology, pharmacoeconomics, social science and public health. NACI makes recommendations for the use of vaccines currently or newly approved for use in humans in Canada, including the identification of groups at risk for vaccine-preventable diseases who would benefit from vaccination.
Net health benefit: A summary statistic that represents the impact of an intervention on population health, expressed in quality-adjusted life-year (QALYs). It estimates the health that is foregone elsewhere when funds are moved to pay for the intervention. A positive net health benefit implies that the overall population health would increase with the introduction of the intervention.
Net monetary benefit: A summary statistic that represents the value of an intervention, expressed in monetary terms. It estimates the benefit derived from an intervention (monetized based on a cost-effectiveness threshold) and compares to the cost of intervention. A positive net monetary benefit implies a greater benefit than cost (i.e., the intervention is cost-effective compared to the alternative strategy).
Open (population) model: A model that allows new individuals to enter the model either through birth or in-migration, or to exit the model through death or out-migration over time.
Parameters: Inputs in a model such as the rates of movement between model states, probabilities of events, health utilities, and costs.
Parameter uncertainty: Uncertainty in parameter estimates that are used to populate a model. This type of uncertainty is also referred to as second-order uncertainty.
Pathogen variations: Differences between strains of a pathogen related to serotypes, serogroups, or genotypes.
Perspective: The viewpoint from which an economic evaluation is conducted. The perspective determines the outcomes and costs that will be included in the analysis.
Positive time preference: The preference for present benefits over benefits occurring in the future.
Probabilistic analysis: A method used to quantify parameter uncertainty in an economic analysis where a probability distribution is assigned to uncertain parameter estimates, and values are randomly sampled from each distribution repeatedly to generate a distribution of outcomes that can be analyzed.
Probabilistic sensitivity analysis: See definition for probabilistic analysis.
Productivity: A measure of how efficiently production inputs, such as labour and capital, are being used in an economy to produce a given level of output.
Public Health: The organized efforts of society to keep people healthy and prevent injury, illness and premature death. This definition is from the Canadian Public Health Association. In these Guidelines, Public Health is stylized in capitals to better distinguish it from "public healthcare", which refers to healthcare being publicly funded.
Randomized controlled trial (RCT): A comparative study, designed to ascertain the efficacy of a health intervention, in which individuals are randomly assigned to either the intervention or control group.
Real-world evidence: Evidence used for decision-making that is collected through non-experimental studies.
Reference case: A set of methods for conducting an economic evaluation specified by the decision-maker. The purpose of the reference case analysis is to ensure consistency between methods underpinning analyses and the decision-making process that is employed.
Quality-adjusted life year (QALY): A summary outcome measure used to quantify the health outcomes associated with a particular intervention. QALYs combine the impact of benefits related to both survival and health-related quality of life expressed as health utilities, and allow for comparisons between interventions across disease states.
Scenario analysis: An analysis that tests alternate model scenarios underpinned by different plausible structural assumptions.
Second-order uncertainty: See definition for parameter uncertainty.
Selection bias: Bias in a non-randomized study resulting from systematic differences in sampling of individuals who are in the exposed group compared to those who are not, resulting in a distribution of exposures and outcomes that is no longer representative of the source population. Selection bias can also occur in randomized studies due to attrition post-randomization.
Serotype replacement: The increase in prevalence of non-vaccine serotypes of a pathogen resulting from the removal of vaccine-specific serotypes from the population that compete with them for colonization of hosts.
Spillover effects: The unintended effects of the intervention on caregivers, family members, household members, and others (i.e., the social network of patients). Spillover effects are a type of externality. See definition for externalities.
Static model: A model in which the force of infection is constant over time, and not on the number of other individuals who are infectious.
Stochastic model: For the purposes of these Guidelines, a stochastic model is one that accounts for first-order uncertainty where events are programmed to occur randomly.
Structural uncertainty: Uncertainty related to the structure of a model and other unparameterized sources of uncertainty. Scenario analysis is one approach for assessing this type of uncertainty.
Time horizon: The time period over which outcomes and costs are quantified in an economic evaluation.
Value of information analysis: An analysis used to estimate the value, in terms of cost and health outcomes, of collecting more information (data) on key parameters influencing a funding decision - assuming the funding decision is entirely based on whether the intervention is cost-effective at a specific cost-effectiveness threshold. It is most useful where the output of an economic evaluation is uncertain, but close to a decision threshold, and one or more key parameters are uncertain. See definitions for expected value of perfect information (EVPI) and expected value of partial perfect information (EVPPI).

Foreword: How First Nations, Inuit, and Métis perspectives can be considered in economic evaluation of vaccination programs

Economic evaluations and health technology assessments of vaccination programs

In these guidelines, we suggest how economic evaluations of vaccination programs should be conducted in Canada. Federal vaccination program recommendations are made by the National Advisory Committee on Immunization (NACI) following its decision framework, where cost-effectiveness (as assessed through economic evaluations) is one factor for consideration. The NACI decision framework is based on the Erickson and De Wals analytical framework, which was developed for the "analysis and comparison of potential immunization programs".^Footnotea The Erickson and De Wals framework includes 13 decision-making criteria, of which eight are within NACI's scope and thus included into the NACI decision framework (italicized): (i) burden of disease, (ii) vaccine characteristics (e.g., efficacy, safety), (iii) research questions, (iv) immunization strategy and program, (v) cost-effectiveness, (vi) ethical considerations, (vii) equity, (viii) feasibility, (viii) acceptability, (ix) ability to evaluate programs, (xi) legal considerations, (xii) conformity of program, and (xiii) political considerations.

Decision-makers may note the significant overlap between the factors above and the decision-making attributes often included in health technology assessment (HTA) frameworks. HTAs determine the value of health interventions such as devices, medicines, vaccines, procedures, or systems. While several HTA frameworks exist, the HTA Core Model (Figure 1) represents a useful way of identifying the full range of decision attributes used internationally in HTA. The Core Model was developed by the European Network for HTA (EUneHTA), which includes 70 institutions across 32 countries. This framework includes nine separate domains of technology assessment, including the description of the health problem, technical characteristics, safety, effectiveness, costs and economic evaluation, ethical analysis, organizational aspects, patient and social aspects, and legal aspects, and as such is broader than most conceptualizations of HTA. The authors state that the HTA Core Model, while developed in Europe, is generic and designed for global use.^{Footnote b} The HTA Core Model represents a broad conceptualization of the potential range of HTA applications, rather than detailed guidance about how each of these domains are to be defined and implemented in a multi-attribute decision framework. Whether using the NACI decision framework or the HTA Core Model, many of the decision attributes are the same, and the intent of both approaches is to support decision-making for publicly funded interventions. In both approaches, there is no distinction drawn between attributes that are descriptive (e.g., vaccine characteristics), attributes that are normative (e.g., appraisal criteria such as effectiveness, cost-effectiveness), and attributes related to feasibility considerations (e.g., budget impact, organizational feasibility).

**Figure 1: European Network for health technology assessment (EUneHTA) Core Model**

Preliminary thoughts on accounting for and incorporating Indigenous perspectives into the process of technology assessment

However broad, is this "conventional Western" approach fit for purpose? In other words, are existing approaches to HTA, as described in the NACI decision framework or HTA Core Model, adequate to conduct and appraise HTAs in a way that considers First Nations, Inuit, and Métis knowledge and ways of knowing regarding health and well-being? Are the attributes, decision rules for combining attributes, and decision processes using conventional HTA approaches appropriate? How might we bring Indigenous perspectives more fully into decisions about health? ^{Footnote c}^{Footnote d}^{Footnote e} What about funding mechanisms?

These are challenging questions, for which there are few answers at present based on our literature review. As the literature related to the evaluation of health of Indigenous Peoples continues to grow,^{Footnote c}little is published regarding technology assessment from Indigenous perspectives, including how technology assessment should be specifically informed by Indigenous perspectives. We do not attempt, therefore, to fully account for or outline Indigenous perspectives that should be included in HTA in these Guidelines. We offer, instead, some resources and preliminary thoughts, while recognizing Indigenous perspectives vary, including by distinction, region, and Nation, and there is much work to be done to better understand how to meaningfully incorporate Indigenous perspectives into HTA. The following resources may inform an analysis of health and wellness that incorporates Indigenous perspectives:

1) Select frameworks on social determinants of health

One resource is the National Collaborating Centre for Indigenous Health (NCCIH) State of knowledge of Aboriginal health: A review of Aboriginal public health in Canada.^{Footnote f} The NCCIH document conceptualizes Indigenous health broadly within a social determinants of health framework. Social determinants of health impacting Indigenous populations include colonialism (including irrevocable loss of land, language, culture, and livelihood), globalization, migration, cultural continuity, territory, access, poverty, racism, social exclusion, self-determination, land, and environmental stewardship. These determinants are not discrete categories, but rather have a complex and interconnected relationship.

The document also notes the broad cultural and geographic variation (including by urban, remote, on reserve, off-reserve locations) among First Nations, Inuit, and Métis peoples. Disparities exist across the populations in socio-economic status (including employment, type of work, income, educational attainment, and economic development), healthy early development and family structures, women's wellness (including violence and criminalization), access to health services, housing conditions, and loss of language and culture.

Another resource is the Framework for Understanding the Social Determinants of Health and Indigenous People by Dr. Margo Greenwood.^{Footnote g} The framework also highlights the historical and contemporary contexts of social, cultural, political, and economic relationships and realities. It covers non-indigenous systems, structures, and disciplines; Indigenous communities, nations, and culture; and their interface.

Note that work focused on social determinants of health may not completely align in scope and intent with traditional HTA. Work on social determinants of health identifies factors associated with health and well-being. Technology assessment usually focuses somewhat more narrowly on health and reflects a greater focus on economic questions such as cost-effectiveness and affordability.^{Footnote f} Nonetheless, there are recent efforts to apply dimensions of equity (including social determinants of health) to HTA, ^{Footnote h} including within the NACI decision framework (see NACI's Ethics, Equity, Feasibility, and Acceptability (EEFA) Framework)^{Footnote i} and within these Guidelines on economic evaluations (see chapter 14 on Equity).

**Figure 2: First Nations perspectives on health and well-being**

2) Inuit perspectives on health and well-being

Another resource from NCCIH describes the role of Inuit Qaujimajatuqangit (IQ), the knowledge system characteristic of the Inuit culture, in supporting wellness in Inuit communities in Nunavut.^{Footnote j} As documented by Inuit Elders in Nunavut, the framework for IQ is grounded in: (i) working for the common good, (ii) respecting all living things, (iii) maintaining harmony and balance, and (iv) continually planning and preparing for the future. IQ is the foundation for social, emotional, spiritual, cognitive, and physical well-being, supporting both personal and cultural wellness. Cultural health is the "basis for every other kind of health because in it resides the sense of identity, the collective social supports for the individual, and the sense of belonging."^{Footnote j} The document recognizes the need for a cultural health approach for the Inuit population and recognizes economics as an aspect of the social-cultural environment to examine.

Implications for Canadian economic guidelines of vaccination programs

As noted in the resources described above, the conceptualizations of health and well-being are broad.^{Footnote c}^{Footnote f}^{Footnote g}^{Footnote j} The emphasis on the ecologic, social, and communal dimensions of health stands out. Individual health is only possible when the community is healthy. In HTA, a range of definitions is used for health, some of which align more closely with Indigenous conceptualizations of health and well-being, and some of which do not. For example, developers of the Health Utilities Index (HUI) and other commonly used health status indexes deliberately take an individual approach to the measurement of health.^{Footnote k}^{Footnote l}^{Footnote m}^{Footnote n} They specifically exclude the social and cultural dimensions of health. This contrasts the broader definition of health from the World Health Organization (also used by the guidelines here), which considers physical, mental and social well-being.^{Footnote o} Further, in the Indigenous resources, there is an explicit recognition of the importance of "the land" in sustaining individual and community life. Environmental considerations are receiving increasing attention in health decision-making generally. The British Columbia First Nations Health Authority and NCCIH resources highlight the need for attention to land, food, air, water, and housing as fundamental determinants of health. There is an acknowledgment that resources are constrained, but this problem is framed differently than conventional economic approaches. Rather than thinking about decision-making in the context of scarce resources as an optimization problem, the resources above frame the problem of limited resources as a challenge to sharing and managing the resources that are available. The problem is the same, but the emphasis is on sharing and collaborative decision-making.

We do not intend to suggest that our new Guidelines on economic evaluation are definitive. In particular, we recognize that future versions of these guidelines will need to evolve to take account of First Nations, Inuit, and Métis perspectives more holistically. However, our brief overview suggests to us that the decision attributes and decision processes used in conventional HTA, while at present insufficient, are not irrelevant to the task of informing sound public decisions regarding the use of vaccines. Many of the themes reflected in the EUneHTA document are also touched upon in the resources above on Indigenous health and well-being. We also recognize that economic evaluations are not independent, and sit within decision frameworks, including specific decision attributes, decision rules, and decision processes.

This high-level review has revealed some of the gaps in the HTA literature and suggests to us that there is significant work to be undertaken, as a committee and as a field of study, in understanding Indigenous perspectives on health and technology. This iteration of the Guidelines involved review from Indigenous Services Canada of this foreword, chapter 2 on Types of evaluation, and chapter 14 on Equity. Future iterations of the Guidelines, as well as NACI's decision framework more broadly, will require collaboration and engagement with Indigenous governments, organizations, academic groups, among others. Future versions may benefit from considering the Etuaptmumk conceptual framework (Mi'kmaw for "Two-Eyed Seeing"), which provides guidance on considering both Indigenous and Western perspectives in parallel to produce an enriched and mutual understanding. Mi'kmaw Elder Albert Marshall of the Eskasoni First Nation defines Two-Eyed Seeing as "learning to see from one eye with the strengths of Indigenous knowledges and ways of knowing, and from the other eye with the strengths of mainstream knowledges and ways of knowing, and to use both these eyes together, for the benefit of all."^{Footnote p} In the next iteration of the Guidelines, we plan to incorporate Indigenous knowledge to ensure the perspectives of Indigenous Peoples inform public healthcare decision-making on vaccination programs in Canada. We envision this work on economic evaluations to influence and inform future developments in HTA decision frameworks.

Introduction

This is the first edition of the National Advisory Committee on Immunization (NACI) Guidelines on the Economic Evaluation of Vaccination Programs in Canada; in the text hereafter, they will be referred to as the guidelines. These Guidelines have been established to articulate best practices for conducting and reporting economic evaluations of vaccination programs in Canada, be they regional, provincial, territorial, or national programs. Adherence to common best practices can allow decision-makers in Canada's publicly funded health system to have access to consistent and credible information to inform funding decisions related to vaccination programs. The Canadian Agency for Drugs and Technologies in Health's (CADTH's) Guidelines for the Economic Evaluation of Health Technologies: Canada^{Footnote 1} present more general information applicable to health technologies in Canada. Health technologies can refer to devices, medicines, procedures, and systems. Where CADTH's Guideline Statements are applicable to economic evaluations of vaccination programs, they have been included in these guidelines.

The main feature that distinguishes vaccination programs from other health technologies is their population-level effects, which result from their potential to affect both vaccinated and unvaccinated individuals. These Guidelines present detailed information on how to incorporate these population-level effects into economic evaluations of vaccination programs, including methods for addressing their non-health sector impacts.

Development process

The recommendations contained in these Guidelines were formulated by NACI's Economic Guidelines Task Group, comprised of Canadian and international experts in infectious diseases and health economics. Over four years, the group engaged in a series of discussions that led to decisions made by consensus, which were supported by literature reviews for selected topics. During the development process, the group consulted with relevant stakeholders including the Canadian Immunization Committee, and the Council of Chief Medical Officers of Health- committees with representation from each Canadian province and territory, with the former focusing on immunization program planning. During the review process, federal government partners provided input on their respective areas of speciality including Indigenous Services Canada, and the Public Health Agency of Canada (PHAC) Public Health Ethics Consultative Group. A peer-review process engaged both Canadian and international academics with representation across gender, geography, and areas of expertise. A public consultation process solicited feedback from stakeholders including, but not limited to: industry, patient groups, economic guideline groups, health technology assessment agencies, national immunization technical advisory groups, and the general public. These consultations and reviews informed the final recommendations and text contained in this document. The guidelines have been approved by the Economic Guidelines Task Group, NACI, and PHAC.

Overview and structure of the guidelines

A social decision-making framework has been adopted for these guidelines. The basis of this framework is that the health decision-maker acts on behalf of a socially legitimate higher authority (e.g., a democratically elected government) to achieve an explicit policy objective (e.g., improving overall population health). The function of an economic evaluation within this framework is to inform social decisions.^{Footnote 2}^{Footnote 3}^{Footnote 4}

The guidelines use the World Health Organization's definition of health, which is that "health is a state of complete physical, mental and social well-being and not merely the absence of disease or infirmity".^{Footnote 5} Critics of this broad definition argue that the concept is unattainable and is difficult to operationalize as a goal for health services. Proponents of a broad definition argue that biomedical definitions, often focused on disability or deficit, are inadequate. The task group is in line with the Public Health community which sees merit to including psychological, social, spiritual, and well-being domains. The task group also recognizes the pragmatic needs of researchers measuring and valuing health through commonly used generic instruments. Hence, the guidelines recommend that researchers assess and justify the data source(s) of health state utility values based on fitness for purpose, credibility, and consistency. This includes assessing how well the various domains of health are captured for the infectious disease of interest, and if supplementing generic instruments with disease-specific instruments can identify health impacts that are not captured or are not adequately captured.

Economic evaluations have traditionally focused on the trade-offs between incremental costs and incremental effects of different health interventions to enable decision-makers to make judgments related to efficiency. These guidelines expand on the traditional approach by encouraging the integration of equity considerations into economic evaluations of vaccination programs. Such integration is consistent with NACI's Ethics, Equity, Feasibility, and Acceptability (EEFA) Framework, which provides a mechanism for decision-makers to systematically consider these four factors, alongside effectiveness and cost-effectiveness, when making recommendations about vaccination programs.^{Footnote 6}

Recommendations are presented for the following aspects of economic evaluations of vaccination programs: decision problem, types of evaluations, study populations, comparators, perspectives, time horizon, discounting, modelling, effectiveness, measurement and valuation of health, resource use and costs, analysis, uncertainty, equity, and reporting. Guidance on each of these topics is contained in a separate chapter. Guideline Statements are presented in the next section of this document and at the beginning of each chapter for ease of use, followed by a detailed discussion of the recommendations. Contained in these statements are recommendations for two reference case analyses. The reference cases encourage the use of a standard set of methods when conducting economic evaluations of vaccination programs and enable decision-makers to compare results between different vaccination programs. The reference cases are not intended to be a checklist to follow uncritically. Hence, researchers who cannot (or strongly believe that they should not) follow a particular recommendation should document the deviation, present a rationale, and discuss the likely implications to the inferences of the economic evaluation. In addition, any deviation should be prominently noted (e.g., in the abstract or executive summary) to ensure users of the study who do not read the full details do not misinterpret the main conclusions.

The guidelines are written for researchers conducting economic evaluations, as well as end-users of economic evidence such as decision-makers - in other words, those who are technically proficient in the methods of economic evaluation. As such, background on methods has been omitted. The guidelines are not intended to be a primer or a methodological treatise (e.g., on selecting the model structure, building the model, evidence grading). Similarly, the guidelines omit detailed background information related to vaccines and immunization, as it is expected that researchers will consult with subject matter experts. The references contained in this document provide sources for researchers to obtain additional information when required.

The guidance presented in this document represents NACI's current recommendations for conducting economic evaluations of vaccination programs. The guidelines have attempted to reflect current best practices, but the recommendations contained here will evolve alongside scientific and methodological advancements. Topics for which there is no current consensus on best practices and require further research have been identified. As such, the function of these guidelines is not only to recommend current best practices for the economic evaluation of vaccination programs, but also to suggest directions for future research. The Guidelines are not meant to be prescriptive for these topics.

The remit of these Guidelines is to specify methods for conducting economic evaluations of vaccination programs, and not to provide guidance or insights into the decision-making process. For those interested in the decision-making process at the Canadian federal level, please refer to NACI's "Process for incorporating economic evidence into federal vaccine recommendations".^{Footnote 7}

Motivation for two reference cases

These guidelines recommend the adoption of two reference case analyses for the economic evaluation of vaccination programs: one conducted from the publicly funded health system perspective and the other conducted from the societal perspective. The latter is to account for the full range of impacts associated with vaccination programs, including those that accrue to non-health sectors and to those not directly vaccinated (i.e., via non-health spillovers and externalities). Some of these broad impacts are unique to or are unusually large with vaccination programs. Among academic groups as well as the World Health Organization, there is a growing consensus that vaccines generate value through impacts traditionally not included in health technology assessments.^{Footnote 8}^{Footnote 9} Failure to account for them can lead to undervaluation of vaccination programs, and subsequently suboptimal funding decisions and allocation of resources. Note that broad impacts do not only refer to benefits; for instance, there are environmental costs to implementation that the Guidelines recommend capturing in an economic evaluation of vaccination programs.

The task group has given serious consideration to the adoption of two perspectives, with the acknowledgment that there are arguments against the use of the societal perspective as a reference case (e.g., the lack of an empirically estimated societal cost-effectiveness threshold; the lack of established methods to quantify select broader impacts into monetary terms). The task group recognizes that many arguments for or against this reference case involve normative judgment and social preferences; hence, there is no "correct" choice. During consultation, Canadian Public Health decision-makers expressed the value of deliberating results from the societal perspective alongside the publicly funded health system perspective. Results from both the publicly funded health system perspective and the societal perspective can provide decision-makers a more fulsome account of the vaccination program impacts.

A general note to decision-makers in consideration of two perspectives

Decision-makers at NACI do not use explicit weighting of various decision determinants when deliberating vaccination decisions at the federal level. Similarly, they do not use explicit weights when interpreting economic evaluation results from the publicly funded health system perspective versus the societal perspective. For decision-makers in general, however, some may choose to put more emphasis on results from one perspective over another, given their unique circumstances. For example, the health system perspective may be stressed when there is a fixed health budget from which vaccines are funded and other health technologies from that same budget are assessed purely from a health system perspective. The societal perspective may be stressed when vaccines are funded from a general budget, or when decision-makers are interested in overall welfare and not only health gains. Decision-makers should not be left unaware of the many impacts of vaccination programs that fall outside of the health sector. The same can be said for other health technologies, albeit that their non-health impacts may be smaller. As such, the task group would even argue that assessments of other health technologies in Canada could benefit from an additional reference case conducted from the societal perspective.

Again, there are normative judgments and social preferences involved; hence, there is no "correct" approach on how/ if to weigh results from the two perspectives. The recommendations on the two reference cases and multiple sensitivity analyses found in these Guidelines bolster evidence-informed decision-making for vaccination program.

Guideline statements

1. Decision problem

1.1 "The decision problem addressed by the economic evaluation should be clearly stated." [CADTH Guideline Statement]
1.2 The decision problem statement should provide a comprehensive specification of the interventions to be compared, the setting(s) in which they are to be delivered, the perspectives of the evaluation, which costs and outcomes are to be considered (including externalities), the time horizon, and the population(s) for the evaluation (including populations directly and indirectly affected by the vaccination program). [CADTH Guideline Statement with amendment]

2. Types of evaluations

2.1 In the reference cases, the economic evaluation should be a cost-utility analysis (CUA) with outcomes expressed as quality-adjusted life-years (QALYs). Any departure from this approach should be clearly justified. [CADTH Guideline Statement with amendment]
2.2 A cost-benefit analysis (CBA) may be used alongside the reference case CUAs in situations where the vaccination program may be compared to a non-health intervention.

3. Study populations

3.1 Researchers should identify the population(s) intended for the vaccination program, the population(s) at risk for the disease of interest, and any populations that may be indirectly affected by the vaccination program, either through externalities or spillover effects.
3.2 Researchers should present an overall analysis that includes the costs and outcomes for all affected populations. When relevant, researchers should also summarize the results separately for each affected group (e.g., intended population, population experiencing externalities or spillover effects) that was included in the overall analysis.
3.3 Where there are factors that could lead to differences in costs and outcomes related to the vaccination program across affected populations, researchers should stratify analyses and report by subgroups. These factors could include demographic factors, behavioural factors, disease-related factors, and effectiveness of the vaccine or comparator(s).

4. Comparators

4.1 The choice of comparator(s) should be related to the scope of the decision problem. As such, the comparators should reflect the population(s) intended for the vaccination program and the jurisdiction for which the decision is being made. [CADTH Guideline Statement with amendment]
4.2 Researchers should consider both preventive and treatment-based approaches when selecting comparators for economic evaluations of vaccination programs. Preventive interventions could include vaccine-based measures, screening programs, preventive medication-based interventions, and preventive non-medical interventions.

5. Perspectives

5.1 Two reference case analyses should be presented as part of the economic evaluation of vaccination programs: one conducted from the publicly funded health system perspective, and the other conducted from the societal perspective.
5.2 "Both costs and outcomes should be consistent with the stated perspective." [CADTH Guideline Statement]

6. Time horizon

6.1 In the reference cases, the time horizon should be long enough to capture all relevant differences in the future costs and outcomes associated with the interventions being compared. Thus, the time horizon should be based on the infectious disease and the likely impact of the intervention. [CADTH Guideline Statement with amendment]
6.2 Researchers should justify their choice of time horizon. Where it spans a long period of time (i.e., multiple decades), researchers should report incremental costs, incremental effects and incremental cost-effectiveness ratio (ICER) estimates from various time points throughout the time horizon.

7. Discounting

7.1 In the reference cases, costs and outcomes that occur beyond one year should be discounted to present values at a rate of 1.5% per year. [CADTH Guideline Statement with amendment]
7.2 "The impact of uncertainty in the discount rate should be assessed by comparing the results of the reference cases to those from non-reference case analyses, using discount rates of 0% and 3% per year." [CADTH Guideline Statement]

8. Modelling

8.1 "Model conceptualization and development should address the decision problem." [CADTH Guideline Statement]
8.2 Researchers should consider any existing well-constructed and validated models that appropriately capture the clinical or care pathway for the infectious disease of interest when conceptualizing their model. [CADTH Guideline Statement with amendment]
8.3 The model structure should reflect the natural history of disease, the clinical or care pathway, and account for susceptibility, infectiousness, immunity, morbidity and mortality related to the infection.
8.4 Relevant behavioural dynamics including contact patterns between individuals and behaviours related to infection prevention and control should be incorporated into the model where appropriate.
8.5 Dynamic models should be considered in economic evaluations of vaccines that are associated with externalities such as prevention of human-to-human transmission of infection and age-shifting of disease.
8.6 Other model attributes should be considered in the context of the decision problem such as whether the model is deterministic or stochastic, whether the population is modelled at the aggregate level or individual level, and whether the population is open or closed.
8.7 Researchers should transparently report on model calibration and validation processes that were undertaken and on their results.

9. Effectiveness

9.1 A comprehensive search of the available data sources should be conducted to inform the estimates of effectiveness and harms associated with the interventions. Report the included studies and methods used to select or combine the data. Researchers should assess sources used for effectiveness based on their fitness for purpose, credibility, and consistency. Describe the trade-offs among these criteria and provide justification for the selected source(s). [CADTH Guideline Statement with amendment]
9.2 The following criteria should be considered when assessing estimates of vaccine effectiveness: vaccine effectiveness by dose and time (e.g., waning protection); pathogen variation-specific effectiveness (i.e., serotypes, serogroups, strains); and geographic and vaccine recipient factors that may affect effectiveness.
9.3 Researchers should ensure that immune biomarkers used as surrogate outcomes in studies of vaccine efficacy or effectiveness meet the criteria for correlates of protection.

10. Measurement and valuation of health

10.1 In both reference cases, the quality-adjusted life year (QALY) should be used as the method for valuing health outcomes.
10.2 "Health preferences should reflect the general Canadian population." [CADTH Guideline Statement]
10.3 In the reference cases, researchers should use health preferences obtained from an indirect method of measurement that is based on a generic classification system (e.g., EuroQol 5-Dimensions questionnaire [EQ-5D], Health Utilities Index [HUI], Short Form 6-Dimensions [SF-6D], Child Health Utility 9-Dimensions [CHU9D], Assessment of Quality of Life [AQoL]). Researchers must justify where an indirect method is not used. [CADTH Guideline Statement with amendment]
10.4 A comprehensive search of the available data sources should be conducted to inform the health state utility values. Report the included studies and methods used to select or combine the data. Researchers should assess sources used for health state utility values based on their fitness for purpose, credibility, and consistency. Describe the trade-offs among these criteria and provide justification for the selected sources. [CADTH Guideline Statement with amendment]

11. Resource use and costs

11.1 For each reference case analysis, researchers should systematically identify, measure, value, and report all relevant resources consumed or saved as a result of the delivery or implementation of the vaccination program under consideration.
11.2 Where possible, researchers should value the consumption of relevant resources identified for all sectors in monetary terms. In situations where this is not possible, researchers should present the relevant resources that have been identified in the Impact inventory table for economic evaluations of vaccination strategies (Appendix I) for consideration by decision-makers.
11.3 Resource use, unit prices, and costs should be based on Canadian sources and reflect the jurisdiction(s) of interest (as specified in the decision problem). [CADTH Guideline Statement with amendment]
11.4 When valuing and monetizing resources, researchers should select data sources that most closely reflect the opportunity cost, given the perspective of the analysis. [CADTH Guideline Statement with amendment]
11.5 A comprehensive search of the available data sources should be conducted to inform the resource use and cost values. Report the included studies and methods used to select or combine the data. Researchers should assess sources used for cost data based on their fitness for purpose, credibility, and consistency. Describe the trade-offs among these criteria and provide justification for the selected sources.

12. Analysis

12.1 Incremental costs, incremental effectiveness, incremental cost-effectiveness ratios (ICERs) and, where useful for interpretation, net monetary benefits, or net health benefits, should be calculated for both reference case analyses.
12.2 "For analyses with more than two interventions, a sequential analysis of cost-effectiveness should be conducted following standard rules for estimating ICERs, including the exclusion of dominated interventions." [CADTH Guideline Statement]
12.3 The expected values of costs and outcomes, where possible, should be generated probabilistically to reflect the overall uncertainty in the model parameters.

13. Uncertainty

13.1 Researchers should address parameter uncertainty using a probabilistic reference case analysis, where possible, as well as deterministic sensitivity analyses.
13.2 "Methodological uncertainty should be explored by comparing the reference case results to those from a non-reference case analysis that deviates from the recommended methods in order to examine the impact of methodological differences." [CADTH Guideline Statement]
13.3 Cost-effectiveness acceptability curves (CEACs) and cost-effectiveness acceptability frontiers (CEAFs) should be used to represent the uncertainty in the estimates of costs and outcomes when these estimates have been generated probabilistically. [CADTH Guideline Statement with amendment]
13.4 When the decision problem includes the option of commissioning or conducting future research, value-of-information analysis may be helpful to characterize the value of these options and design future research, and may be included in the reference case analyses. [CADTH Guideline Statement with amendment]
13.5 Scenario analyses should be used to assess structural uncertainty. [CADTH Guideline Statement with amendment]

14. Equity

14.1 Researchers and decision-makers should work together to establish which equity dimensions and goals should be included in the economic evaluation of the vaccination program considered. Equity dimensions should be considered in the context of NACI's Ethics, Equity, Feasibility, and Acceptability (EEFA) framework.
14.2 Analyses that incorporate relevant equity concerns should accompany the reference case analyses (e.g., distributional cost-effectiveness analysis, extended cost-effectiveness analysis, or other emerging methods), and presented alongside the reference cases.

15. Reporting

15.1 "The economic evaluation should be reported in a transparent and detailed manner with enough information to enable the reader or user (e.g., decision-maker) to critically assess the evaluation." [CADTH Guideline Statement with amendment]
15.2 "A summary of the evaluation written in non-technical language should be included." [CADTH Guideline Statement]
15.3 "Results of the economic evaluation should be presented in graphical or visual form, in addition to tabular presentation." [CADTH Guideline Statement]
15.4 "Details and/ or documents describing quality assurance processes and results for the economic evaluation should be provided. An electronic copy of the model should be made available for review with accompanying documentation in adequate detail to facilitate understanding of the model, what it does, and how it works." [CADTH Guideline Statement]
15.5 "Funding and reporting relationships for the evaluation should be described, and any conflicts of interest disclosed." [CADTH Guideline Statement]
15.6 Researchers should use NACI's Guidelines for Reporting Economic Evaluations of Vaccination Programs in Canada (supplemental document) and complete the Impact inventory table for economic evaluations of vaccination strategies (Appendix 1).

Guidelines in detail

1. Decision problem

Guideline Statements

1.1 "The decision problem addressed by the economic evaluation should be clearly stated." [CADTH Guideline Statement]
1.2 The decision problem statement should provide a comprehensive specification of the interventions to be delivered, the setting(s) in which they are to be compared, the perspectives of the evaluation, which costs and outcomes are to be considered (including externalities), the time horizon, and the population(s) for the evaluation (including populations directly and indirectly affected by the vaccination program). [CADTH Guideline Statement with amendment]

The decision problem being addressed by an economic evaluation of a vaccination program should address concerns relevant to decision-makers and be clearly articulated at the outset of the analysis. Following a standardized approach can ensure consistency with other economic evaluations of vaccines where possible.

In Canada, various decision-makers assess economic evaluations of vaccination programs. At the federal level, they include NACI; at the provincial and territorial level, they include health ministries and, depending on the jurisdiction, provincial immunization technical advisory groups. NACI provides non-binding advice to provincial and territorial governments that in turn make implementation decisions. Such decisions include whether to publicly fund a new vaccination program, how to roll-out programs, and how to strengthen and monitor existing programs.

The decision problem should provide a detailed description and justification of the vaccination program being evaluated, including: 1) the interventions to be compared; 2) the setting(s) in which they are to be compared; 3) the type of evaluation being conducted; 4) the perspectives from which the analysis is being carried out; 5) the costs and outcomes to be quantified in the analysis; 6) the time horizon over which the analysis is to be carried out; 7) and the population(s) for the evaluation.

A clear description of the vaccine being evaluated, including the dosage of vaccine, the number of doses required, dose schedule, whether any booster doses are required, expected dose completion, handling of vaccine wastage, assumptions on waning, coverage estimates, and setting of vaccine delivery, should be provided along with detailed descriptions of comparators. Comparators could include other existing preventive vaccines, non-vaccine-based preventive approaches, and current treatment approaches including best supportive care.

The type of economic evaluation should be specified and justified. The type of economic evaluation, along with the perspective, will determine which costs and outcomes should be included (and how). The included outcome measures, which should be the same for each comparator, should be explicitly stated in the decision problem and listed by area of impact (e.g., health outcomes, educational achievement). Similarly, the included costs should be explicitly stated and listed by area of impact (e.g., healthcare costs, education-related costs, productivity-related costs).

Researchers should seek out and engage with decision-makers to gain an understanding of the concerns they are intending to address with the vaccination program. Some aspects of the decision problem that are particularly relevant to decision-makers include, but are not limited to: 1) population(s) for the evaluation; 2) vaccine delivery setting(s); 3) the time horizon of the evaluation; 4) possible impacts of the vaccination program beyond the health sector; and 5) health inequities that could potentially be affected by introduction of the vaccination program.

Researchers, in collaboration with decision-makers, should determine the population(s) to be included in the evaluation. The study population may be the entire population for which the vaccine is indicated, a subset of the indicated population, or a different population for off-label vaccine use, depending on the decision-makers' needs. The study population should not be limited to those intended for the vaccination program. Where applicable, those at risk for the disease of interest, and those who experience externalities and spillover effects (e.g., informal caregivers) should be included. Where there are heterogeneities between groups of individuals that may affect the results of an economic evaluation (e.g., age groups, clinical risk groups, geographic location), researchers should identify the subgroups and identify the different vaccination strategies that can improve coverage in each of the subgroups. All options of interest to the decision-maker should be evaluated together using the principles of full incremental analysis. Where applicable, stratified analyses and stratified reporting should accompany the full analysis.

Researchers should ensure the vaccine delivery setting(s) of interest to decision-makers are captured in the economic evaluation. Examples include physician clinics, pharmacies, schools, Public Health clinics, and workplaces. The choice of setting by decision-makers can vary depending on the disease and vaccine-related factors. Moreover, they may choose to deliver the vaccination program across multiple settings.

Ensuring a time horizon that is relevant to decision-makers is particularly important when a vaccination program results in protection for unvaccinated individuals through community immunity (also known as herd immunity) with the potential for disease elimination. Often a very long time horizon (sometimes many decades) is required for the full costs and effects of a vaccination program to become apparent. In these cases, researchers should ensure that results of an economic evaluation are reported from several time points to allow decision-makers to determine when payoffs of the program become positive.

Given the broader, non-health-related outcomes that are associated with many vaccination programs, a perspective broader than the health system perspective will usually be relevant. In these Guidelines, health system refers to both healthcare clinical services and Public Health. Researchers should attempt to gain an understanding of the broader costs and outcomes related to the vaccination program that may fall outside of the health sector, and that are relevant to the decision-maker. For example, because measles can lead to neurologic damage, preventing measles through childhood vaccination improves educational outcomes. Similarly, preventing influenza in the population through a vaccination program leads to productivity-related benefits. Further details on this topic are found in Chapter 5 on Perspectives.

Certain groups are at high risk of infection and at risk of the adverse impacts of infectious disease mitigation policies due to historical harms and socially constructed barriers. Vaccines have been identified as a strategy to potentially reduce inequities relating to risk of infection or burden of the disease in question. Researchers, in collaboration with decision-makers, should identify specific groups who may especially benefit from the vaccination program. For example, individuals of lower socioeconomic status and those belonging to minorities experience a higher incidence of cervical cancer and greater mortality related to the disease. Such outcomes could be prevented through a human papillomavirus (HPV) vaccination program.^{Footnote 10} Conversely, researchers and decision-makers should also consider whether the vaccination program may not benefit some groups. A further discussion on this topic is found in chapter 14 on Equity.

2. Types of evaluations

Guideline Statements

2.1 In the reference cases, the economic evaluation should be a cost-utility analysis (CUA) with outcomes expressed as quality-adjusted life-years (QALYs). Any departure from this approach should be clearly justified. [CADTH Guideline Statement with amendment]
2.2 A cost-benefit analysis (CBA) may be used alongside the reference case CUAs in situations where the vaccination program may be compared to a non-health intervention.

In the reference cases, the economic evaluation should be a cost-utility analysis (CUA) with outcomes expressed as quality-adjusted life-years (QALYs). There is recognition that there are populations in whom CUAs cannot be robustly conducted because valid instruments for direct utility elicitation do not exist. Such is the case for children under 8 years of age. In these cases, alternative analytic approaches such as cost-effectiveness analysis (CEA) with a relevant outcome measure in natural health units may be considered and should be justified.

In addition to the reference case CUA, a cost-benefit analysis (CBA) may be presented in cases where broader impacts beyond health are important factors for decision-makers. CBA has been proposed as a method to evaluate vaccination programs associated with consequences that fall outside of the health sector.^{Footnote 11}^{Footnote 12}^{Footnote 13} NACI's recommendation to conduct a reference case CUA from the societal perspective should enable researchers to account for non-health sector outcomes by monetizing them, including them in the incremental costs, and subsequently the numerator of the incremental cost-effectiveness ratio (ICER) estimate. This approach, however, does not enable decision-makers to compare non-health outcomes of alternative programs, or to compare vaccination programs to non-health programs. This is because the denominator of the ICER estimate is reported in QALYs. In cases where a decision-maker may be interested in comparing the economic attractiveness of a vaccination program to a non-health intervention (e.g., school lunch program), researchers could present a CBA alongside the societal perspective reference case analysis to enable such a comparison.^{Footnote 11} Researchers should be aware that different approaches can be used to monetize outcomes in a CBA, and that this could lead to wide variations in the results of a CBA.^{Footnote 12} For instance, researchers may consider the human capital approach, friction cost approach, value of a statistical life using revealed or stated preferences, or monetization of QALYs using a cost-effectiveness threshold. The choice of a particular approach needs to be specified and justified. While these Guidelines do not prescribe a particular approach, researchers may wish to consult the Reference Case Guidelines for Benefit-Cost Analysis in Global Health and Development,^{Footnote 14} or other CBA guidance outside of the health sector.^{Footnote 15}^{Footnote 16}^{Footnote 17}

3. Study populations

Guideline Statements

3.1 Researchers should identify the population(s) intended for the vaccination program, the population(s) at risk for the disease of interest, and any populations that may be indirectly affected by the vaccination program, either through externalities or spillover effects.
3.2 Researchers should present an overall analysis that includes the costs and outcomes for all affected populations. When relevant, researchers should also summarize the results separately for each affected group (e.g., intended population, population experiencing externalities or spillover effects) that was included in the overall analysis.
3.3 Where there are factors that could lead to differences in costs and outcomes related to the vaccination program across affected populations, researchers should stratify analyses and report by subgroups. These factors could include demographic factors, behavioural factors, disease-related factors, and effectiveness of the vaccine or comparator(s).

The results of any economic evaluation of a vaccination program depend on the intervention's impact on three populations: 1) the population(s) intended for the vaccination program; 2) the population(s) at risk for the disease of interest; and 3) population(s) that may experience externalities or spillover effects. The decision problem should include, at minimum, the first study population, and where applicable, the latter two populations. Researchers should identify any externalities associated with vaccination programs (e.g., community immunity, age-shifting of disease), and the population(s) they are expected to affect. For example, a measles vaccination program intended for infants and children may result in community immunity that could potentially lead to population-wide disease elimination. Another example is varicella vaccine intended for young children for the prevention of chickenpox, which could increase the incidence of herpes zoster in the general population. Further details on incorporating externalities into economic evaluations are provided in chapter 8 on Modelling. Additionally, researchers should identify any population(s) that may experience spillover effects related to the vaccination program such as caregivers.

Researchers should provide a detailed description of each population being considered including age, gender, and geographic location. Researchers should also describe any other factors that determine eligibility for the vaccination program being evaluated, and factors that may affect the magnitude of the externalities experienced.

Researchers should present an overall analysis that includes the costs and outcomes for all affected populations including the groups intended for the vaccination program, groups at risk of the disease, and any groups that may experience externalities or spillover effects. Such an analysis should be presented for each implementation strategy that is being considered (e.g., universal vaccination, vaccination of high-risk groups only, vaccination of children only). When relevant, researchers should also summarize the results separately for each affected group (e.g., intended population, population experiencing externalities or spillover effects) that was included in the overall analysis.

In situations where heterogeneities between groups of individuals may affect the results of the economic evaluation, researchers should consider different strategies that improve coverage in each of the subgroups. Researchers should conduct stratified analyses and report by subgroups. This should ideally be based on an underlying mathematical model that considers all the subgroups and interactions between them. Important heterogeneities with respect to vaccination programs could include demographic factors (e.g., age, gender, ethnicity, profession, clinical risk groups, frailty, biomarkers, genetic profiles, geographic location, living in congregate setting), behavioural factors (e.g., expected uptake of the vaccination program, risk-taking behaviours), disease-related factors (e.g., natural history of the disease, risk of disease transmission), effectiveness of the vaccine or comparator(s), and health utilities or costs associated with the health states or interventions included in the analysis. Some of these factors are interconnected with equity. Certain subgroups bear a disproportionate burden of disease or adverse impact of infectious disease control policies for various historical, social, and political reasons. A further discussion on this topic is found in chapter 14 on Equity.

4. Comparators

Guideline Statements

4.1 The choice of comparator(s) should be related to the scope of the decision problem. As such, the comparators should reflect the population(s) intended for the vaccination program and the jurisdiction for which the decision is being made. [CADTH Guideline Statement with amendment]
4.2 Researchers should consider both preventive and treatment-based approaches when selecting comparators for economic evaluations of vaccination programs. Preventive interventions could include vaccine-based measures, screening programs, preventive medication-based interventions, and preventive non-medical interventions.

When selecting comparators for the economic evaluation of vaccination programs, researchers should consider: 1) all current interventions, 2) those that may become available in the near future, and 3) those that may be displaced by the vaccination program being evaluated. Interventions used for prevention or treatment of the disease of interest should be considered. Preventive interventions could include vaccine-based measures, screening programs, preventive medication-based interventions, and preventive non-medical interventions. Often, more than one comparator will be relevant for the economic evaluation, and therefore, all relevant comparators should be included. The included outcome measures should be the same for each comparator.

Vaccine-based measures could include alternative vaccines against the same pathogen (e.g., parenteral trivalent inactivated vaccine versus intranasal live attenuated vaccine for influenza, whole-cell versus acellular vaccines for pertussis, mRNA versus viral vector vaccines for coronavirus disease 2019). They can include vaccines with additional valency (e.g., 10-, 13-, 15-, or 20-valent pneumococcal conjugate vaccines). They can also refer to different implementation or delivery with the same vaccine product. Examples include universal vaccination versus vaccination of high-risk groups only; vaccination of the intended age group with no catch-up strategy versus vaccination of the intended age group with a catch-up strategy for other age groups; vaccination strategy with no booster doses versus strategy with booster doses; vaccination strategies based on a differing number of doses or differing administration schedules; or different settings for delivery such as a school-based strategy versus a Public Health clinic-based strategy versus a mass vaccination strategy.^{Footnote 18}^{Footnote 19}^{Footnote 20}^{Footnote 21} Vaccine-based measures may be affected by concurrent health promotion, information campaigns and other efforts to build community trust.

Screening programs (also known as secondary prevention) could include regular exams and tests to detect disease in its earliest stage. They may be changed by the introduction of vaccination programs. For example, human papillomavirus (HPV) vaccination may change the value and frequency of routine cervical cancer screening.

Preventive medication-based interventions may involve pre- or post-exposure administration of medications. Examples include anti-malarial medications for travellers to malaria-endemic regions, and pre- and post-exposure prophylaxis for human immunodeficiency virus.

Preventive non-medical interventions could include physical measures such as condoms to prevent sexually transmitted infections, face masks to prevent transmission of respiratory infections, or behavioural modifications such as physical distancing and hand washing to prevent infections that are transmitted through close personal contact between individuals.

When treatment-based comparators are being considered, researchers should be aware that best supportive care should be considered the relevant comparator in cases where no curative treatments exist for the disease of interest.

In some cases, the goal of these interventions may be different and there may be limited comparative efficacy data or different endpoints used in clinical studies. Researchers should assess sources based on their fitness for purpose, credibility, and consistency, describe the trade-offs among these criteria and provide justification for the selected source(s). This is discussed further in chapter 9 on Effectiveness, chapter 10 on Measurement and valuation of health, and chapter 11 on Resource use and costs.

5. Perspectives

Guideline Statements

5.1 Two reference case analyses should be presented as part of the economic evaluation of vaccination programs: one conducted from the publicly funded health system perspective, and the other conducted from the societal perspective.

5.2 "Both costs and outcomes should be consistent with the stated perspective." [CADTH Guideline Statement]

Two reference case analyses should be conducted as part of the economic evaluation of vaccination programs: one conducted from the publicly funded health system perspective, and the other conducted from the societal perspective. In these Guidelines, "health system" refers to both healthcare clinical services and Public Health. Public Health is stylized in capital letters to better distinguish it from "public healthcare". Public Health refers to "the organized efforts of society to keep people healthy and prevent injury, illness and premature death".^{Footnote 22} It is a combination of programs, services and policies that protect and promote health of the population (as opposed to clinical management of individual patients). Public healthcare refers to healthcare being publicly funded.

Publicly funded health system perspective

The scope of this perspective should be clearly defined, for instance, as a single provincial/ territorial publicly funded health system, multiple regional publicly funded health systems, or a national health system. Researchers should include: 1) health outcomes experienced by vaccinated and unvaccinated individuals; and 2) costs incurred by the health system. It must be recognized that when the reference case analysis includes multiple publicly funded health systems, the publicly funded cost items may vary from jurisdiction to jurisdiction (e.g., prescription medications), or even within a jurisdiction (e.g., long-term care). Variations in which items are included across systems should be made transparent.

In cases where vaccines are associated with externalities, the health outcomes and costs considered in the analysis also include those experienced by unvaccinated individuals since vaccine plays a critical role in population health.^{Footnote 23} Population-level health outcomes that should be considered include: 1) incidence of infection and disease in vaccinated and unvaccinated individuals (including disease elimination or eradication); 2) changes in the age distribution of individuals who are infected as a result of age-shifting related to the vaccination program (when this has consequences on the overall disease burden as a result of age-dependency in severity of disease); and 3) emergence of new diseases related to variations of the pathogen (i.e., serotypes, serogroups, strains) or unrelated pathogens that may replace the one(s) targeted by the vaccine.

Health system costs that should be considered from this perspective include: 1) publicly funded healthcare services (e.g., physician visits, hospitalization); 2) future related and unrelated healthcare costs; 3) program-related costs funded by Public Health (e.g., vaccine delivery, investigation and management of outbreaks); and 4) intervention-related costs funded by Public Health (e.g., vaccine doses, administration). More examples and guidance on quantifying costs are found in chapter 11 on Resource use and costs.

Societal perspective

A societal perspective reference case analysis is also recommended because many vaccines prevent diseases that have impacts in areas beyond health. For example, the Haemophilus influenzae type b vaccine administered to infants prevents neurological sequelae (e.g., deafness, blindness, developmental delays), all of which would affect a child's school attendance, future productivity and consumption, as well as broader well-being.^{Footnote 24}^{Footnote 25} Even relatively mild diseases, such as childhood diarrhea resulting from rotavirus infection, can lead to impacts outside of health. In many cases, medical attention is not required to treat these infections; nonetheless, a parent is required to take time off work to care for the sick child.^{Footnote 26}^{Footnote 27} Finally, diseases such as coronavirus disease 2019 have tremendous health and economic impacts that extend to every area of the economy,^{Footnote 28} and their impacts could be mitigated through vaccination programs.^{Footnote 29}^{Footnote 30} Failing to consider the full range of benefits associated with vaccines underestimates the role of health as a driver of well-being and economic activity, and could lead to undervaluation of vaccination programs.^{Footnote 23} Some may argue that a vaccination program can be overvalued when comparing a reference case conducted from the societal perspective with a reference case of a non-vaccine health technology, which is normally conducted from the health system perspective (under current 4^th edition CADTH guidelines). Hence, these Guidelines recommend two reference case analyses using the two perspectives. The health system perspective can allow for comparability with other health technologies for which only evaluations from health systems perspectives exist, and the societal perspective can better capture the broader impacts of vaccination programs.

The societal perspective captures all health outcomes and health system costs from the health system perspective. In addition, it captures impacts that fall outside of the publicly funded health system, including healthcare costs not publicly funded by the health system, direct out-of-pocket costs, losses in productivity, consumption, education, social services and community services, and environment. Longer-term impacts such as the effect of childhood illness on an individual's neurodevelopmental impairment, educational achievement, and subsequent long-term productivity (and consumption) should be considered where relevant and feasible. The impacts above are listed along with examples in Appendix I, Table 1: Impact inventory table for economic evaluations of vaccination strategies. The table was adapted from the impact inventory published by the 2^nd Panel on Cost-Effectiveness^{Footnote 31} to further include broader impacts not previously captured. Many of these impacts relate to health technologies in general, and some are specific to vaccines or infectious diseases (e.g., age-shifting of disease, disease eradication).^{Footnote 8}^{Footnote 23}^{Footnote 31}^{Footnote 32}^{Footnote 33} Note that vaccine-specific value frameworks have been recently published, highlighting the broad impacts of vaccination programs on society.^{Footnote 34}^{Footnote 35}^{Footnote 36} The impact table provides a list of health and non-health impacts that could result from vaccination programs. The intent is to allow researchers to systematically consider all impacts when planning for and conducting economic evaluations of vaccination programs. Specific guidance on quantifying these impacts and their associated costs is found in chapter 11 on Resource use and costs.

Researchers should complete and present Table 1 as part of their analysis to explicitly indicate which impacts are included and excluded in the economic evaluation for each of the two reference case analyses. The comments column could be used to provide justification for including or excluding certain impacts or to provide additional information.

6. Time horizon

Guideline Statements

6.1 In the reference cases, the time horizon should be long enough to capture all relevant differences in the future costs and outcomes associated with the interventions being compared. Thus, the time horizon should be based on the infectious disease and the likely impact of the intervention. [CADTH Guideline Statement with amendment]
6.2 Researchers should justify their choice of time horizon. Where it spans a long period of time (i.e., multiple decades), researchers should report incremental costs, incremental effects and incremental cost-effectiveness ratio (ICER) estimates from various time points throughout the time horizon.

Models used to estimate the cost-effectiveness of vaccination programs can be closed or open models. Closed models follow a cohort of individuals over a length of time and do not allow for the entry of new individuals into the model. Most Markov (state-transition) models are closed models. Closed models are usually static, meaning that they do not account for disease transmission dynamics between individuals. Open models, on the other hand, do allow for entry of new individuals into the model over time (e.g., via new births, migration), specifically to account for disease transmission dynamics within a population over time.^{Footnote 37}

Since closed models of vaccination programs often follow a single cohort of individuals, these models should follow the cohort for a long enough time horizon to capture all important differences in future costs and outcomes related to the vaccination strategies being compared. For example, meningitis B acquired during infancy can have lasting consequences throughout adulthood.

Open models may have time horizons that extend beyond the life of any individual alive at the start of the simulation. This is particularly true for vaccines that provide population-level protection through community immunity over multiple birth cohorts. For example, a cohort of individuals vaccinated against measles today may prevent transmission of this infection to another cohort years later. Individuals who are not vaccinated would benefit from this protection for the rest of their lives, in turn, not infecting future cohorts with measles who would also benefit for the rest of their lives.^{Footnote 38}^{Footnote 39} In cases where future generations are implicated, there may be intergenerational equity considerations to account for in the economic evaluation. Intergenerational equity refers to the consequences of decisions today for people not yet born. More information can be found in chapter 14 on Equity.

There are typically three phases in open models: 1) the run-in phase; 2) the evaluation phase; and 3) the steady-state phase. The run-in phase must account for epidemiologic characteristics of the disease prior to introduction of the vaccine to obtain baseline disease dynamics. The epidemiologic estimates used during the run-in phase should be validated based on historical data pertaining to the disease of interest. More information on validation can be found in chapter 8 on Modelling. The evaluation phase begins when the vaccination program is implemented in the intended population and should be long enough to account for externalities associated with the program. The steady-state phase begins once epidemiologic variation terminates.^{Footnote 40}

For the measles example above, and for similar vaccines, the model time horizon should continue until the undiscounted incremental cost-effectiveness ratio (ICER) reaches a steady-state. This is when the ratio between cumulative incremental costs and cumulative incremental health outcomes (QALYs) between the interventions being compared stabilizes. When ICER estimates are not appropriate to calculate such as in the case of dominance, the stability of the incremental costs and incremental outcomes should be checked. The appropriate duration of the model time horizon should be ascertained during, rather than prior, to the analysis.^{Footnote 13}

Stability of the undiscounted incremental estimates and ICER as a criterion should mean that the time horizon of the model will be long enough to capture the full costs and benefits of community immunity as well as any other externalities (e.g., age-shifting of disease) associated with a vaccination program. Researchers should note that models might achieve epidemiologic steady-state prior to the incremental estimates and ICER estimates stabilizing. For example, one program may continue accruing costs or QALYs relative to the other even after epidemiologic equilibrium has been realized. In these cases, the model time horizon needs to continue until the ICER estimate and variance (i.e., credible interval) have stabilized. For some vaccinations programs, modelling a very large number of birth cohorts may be required to achieve stable ICER estimates. [39] If the model is not run until the incremental estimates and ICER have stabilized, researchers should justify why this is the case, and define the run time in terms of time horizon or number of cohorts, and provide justification for this choice.^{Footnote 41}

In justifying the time horizon and number of cohorts, researchers should be aware of the trade-offs between bias and uncertainty. Shorter time horizons may introduce bias into cost-effectiveness estimates since they do not allow for enough time to account for epidemiologic changes resulting from the vaccination program. Shorter time horizons place a greater weight on upfront costs related to initiation of the vaccination program relative to later or annual costs, and reduce the consequences of discounting on measured outcomes. Shorter time horizons also may not quantify all outcomes accrued to all cohorts vaccinated.

For models with long time horizons, researchers should consider the potential for future changes that might alter the costs and benefits of the vaccine (e.g., technological change, long-term estimates of vaccine effectiveness, demographic projections).^{Footnote 18} ^{Footnote 42} ^{Footnote 43} While some of this uncertainty may be accounted for in the discount rate (in particular, for the "catastrophic risk" - the risk of an unanticipated event removing much of the value of the intervention), researchers may wish to consider context-specific, long-term uncertainties such as the emergence of treatment-resistant disease. Where the time horizon spans a long period (i.e., multiple decades), researchers should report ICER estimates from various time points throughout the time horizon. The time points selected may need to differ across economic evaluations of different vaccination programs. The choice can be context-specific, depending on the infectious disease and when health effects set in in the model. Researchers should consider engaging with the decision-makers to determine time points of interest to report.

7. Discounting

Guideline Statements

7.1 In the reference cases, costs and outcomes that occur beyond one year should be discounted to present values at a rate of 1.5% per year. [CADTH Guideline Statement with amendment]
7.2 "The impact of uncertainty in the discount rate should be assessed by comparing the results of the reference cases to those from non-reference case analyses, using discount rates of 0% and 3% per year." [CADTH Guideline Statement]

Discounting costs, health outcomes, and non-health outcomes in economic evaluations reflects the societal preference for present consumption over future consumption. This is because discounting reduces the value of future costs and outcomes compared to their present value. The discount rate accounts for the social rate of time preference, growth rates in healthcare resources and the consumption value of health, and uncertainty about whether future health outcomes will be realized.^{Footnote 44}^{Footnote 45}

Discounting in economic evaluations of vaccination programs can have a profound effect on the cost-effectiveness of programs, particularly in situations where the expected benefits of the vaccination program may not become apparent for years or even generations (e.g., prevention of cervical cancer through a human papillomavirus (HPV) vaccination program). Pediatric populations are particularly susceptible to these effects. Discounting of health and non-health outcomes that accrue in the distant future may lead to a considerably reduced present value of outcomes. For example, in a cost-effectiveness analysis of an HPV vaccination program, the authors report an undiscounted incremental cost-effectiveness ratio (ICER) of €7,600/QALY, which increased to €59,100/QALY when a 4% discount rate was applied to both costs and benefits (3,462 undiscounted QALYs versus 438 discounted QALYs).^{Footnote 46}

The two most common approaches to discounting in economic evaluations of vaccination programs are: 1) constant discounting, where the same fixed discount rate is applied to both outcomes and costs; and 2) differential discounting, where a lower discount rate is applied to outcomes compared to costs.^{Footnote 44}^{Footnote 45}^{Footnote 46}^{Footnote 47}

Researchers should use constant discounting in economic evaluations of vaccination programs. This approach is more commonly employed than differential discounting in evaluations of vaccine and non-vaccine health interventions.^{Footnote 45} Differential discounting is only recommended by select national immunization technical advisory group guidelines and health technology assessment guidelines in uncertainty analyses or in special circumstances.^{Footnote 48}^{Footnote 49}

Arguments in favour of constant discounting of outcomes and costs include consistency and horizontal equity.^{Footnote 45} The consistency argument posits that health technologies associated with the same outcomes and costs over the same analytic time horizon receive equal priority by decision-makers, regardless of the time at which they are initiated.^{Footnote 50} This is because of the constant value of health over time. The horizontal equity argument posits that all individuals who potentially benefit from a vaccination program are treated equally, regardless of when they experience the benefits relative to when the program was initiated. Constant discounting prevents vaccination programs that span multiple generations from being given preference over programs that span a shorter time.^{Footnote 44} Constant discounting may also prevent the strategic use of time horizons and the number of included cohorts included to alter cost-effectiveness estimates. For instance, under a differential discounting strategy, ICERs can be decreased by increasing the number of birth cohorts included; however, this does not occur under a constant discounting strategy.^{Footnote 51}^{Footnote 52}

One argument for differential discounting with a lower discount rate for health and non-health outcomes compared to costs is to normatively place more weight on future benefits. Another main argument for differential discounting is the increasing value that future health expectedly represents, or changing cost-effectiveness thresholds.^{Footnote 53} Discount rates could be adjusted to reflect these changes, though they could also be dealt with more explicitly in uncertainty analysis. With respect to vaccination programs, long time horizons - often generations, may be required to achieve outcomes related to indirect effects of community immunity, which benefit not only the vaccinated individuals but also future cohorts through disease elimination or eradication.^{Footnote 45} Using constant discounting, particularly with higher discount rates, may render the present value of these programs close to zero. This is less of a concern with lower discount rates, as recommended in these guidelines.

Researchers should discount health outcomes, non-health outcomes, and costs occurring beyond one year at a rate of 1.5% per year in the reference case analyses. This value represents the real cost of long-term borrowing for Canadian provinces or territories, who are the authorities responsible for funding most of the Canadian healthcare system,^{Footnote 4} and approximates the rate at which society is willing to trade-off consumption today for consumption in the future.^{Footnote 45}

Sensitivity analyses should be conducted using constant rates of 0% and 3% per year applied to both outcomes and costs to account for any uncertainty in the discount rate. The low discount rate in the reference case analyses mitigates some of the concerns regarding present values of expected outcomes in the distant future. In situations where effects of a vaccination program span multiple generations and may be affected by the discounting strategy or rate used in the analysis, presenting undiscounted results will help decision-makers to assess potential intergenerational effects. A discussion of intergenerational equity is found in chapter 14 on Equity.

8. Modelling

Guideline Statements

8.1 "Model conceptualization and development should address the decision problem." [CADTH Guideline Statement]
8.2 Researchers should consider any existing well-constructed and validated models that appropriately capture the clinical or care pathway for the infectious disease of interest when conceptualizing their model. [CADTH Guideline Statement with amendment]
8.3 The model structure should reflect the natural history of disease, the clinical or care pathway, and account for susceptibility, infectiousness, immunity, morbidity and mortality related to the infection.
8.4 Relevant behavioural dynamics including contact patterns between individuals and behaviours related to infection prevention and control should be incorporated into the model where appropriate.
8.5 Dynamic models should be considered in economic evaluations of vaccines that are associated with externalities such as prevention of human-to-human transmission of infection and age-shifting of disease.
8.6 Other model attributes should be considered in the context of the decision problem such as whether the model is deterministic or stochastic, whether the population is modelled at the aggregate level or individual level, and whether the population is open or closed.
8.7 Researchers should transparently report on model calibration and validation processes that were undertaken and on their results.

This chapter presents considerations related to constructing models used to estimate the cost-effectiveness of vaccination programs. Models with non-human hosts are outside the scope of NACI, and consequently, the scope of these guidelines. An overview of dynamic versus static models in the context of infectious disease modelling is presented, followed by an overview of other model attributes. Finally, recommendations related to model calibration and validation are presented.

Model structure and attributes

The model's structure and attributes should reflect the natural history of disease and include all relevant health states and transitions between these states. There are two primary considerations when conceptualizing a model used to estimate the cost-effectiveness of a vaccination program: whether transmission of infection between individuals is important in estimating the effects of a vaccination program; and whether individual behaviours and characteristics are important in understanding outcomes related to a vaccination programs. Researchers should refer to more detailed model taxonomies by Brennan et al.^{Footnote 54} Kim and Goldie^{Footnote 37} Stahl et al.^{Footnote 55} and Mac et al.^{Footnote 56} for additional details if required.

Further guidance on constructing models for economic evaluation of vaccination programs can be found in chapter 13 on Uncertainty (e.g., ensuring that the model structure accounts for factors related to transmission of infection between individuals, the natural history of the disease being modeled, as well as the direct and indirect effects of the vaccination program).^{Footnote 39}^{Footnote 57}

Dynamic versus static models

Models need to capture externalities related to vaccination programs such as community immunity and age-shifting of disease. In these guidelines, the terms "dynamic model" and "static model" refer to the nature of the incidence rate being dynamic or not (i.e., changing over time based on the proportion of the population that is infectious). While some may alternatively use the terminologies "endogenous" or "exogenous infection incidence rates", these guidelines follow the conventions set out in the World Health Organization economic evaluation guidelines.^{Footnote 58}

Static models, which typically use a constant risk of exposure, do not explicitly represent dynamic infection transmission. These models are acceptable for use in economic evaluations of vaccination programs where there is no human-to-human transmission (e.g., tetanus or rabies).^{Footnote 13} They are also acceptable in situations where the intended group for vaccination is not epidemiologically influential with respect to transmission (e.g., hepatitis A vaccination of healthcare workers, influenza or pneumococcal vaccination in the elderly).^{Footnote 13}^{Footnote 39} Static models may also be acceptable for infections where the individual is already a "host" (e.g., some pneumococcal strains; varicella-zoster virus where herpes zoster (shingles) can occur later in life due to reactivation of latent infection that follows primary varicella (chickenpox) infection). Finally, a static model is acceptable when: 1) a vaccination program is demonstrated to be cost-effective, and a dynamic model would only serve to reinforce this conclusion by accounting for infections prevented through indirect protection or secondary transmission; or 2) there are epidemiological or modelling data available that will allow estimation of the magnitude of community immunity or secondary transmission in the same or very similar setting.^{Footnote 39}^{Footnote 59}

Although the scenarios above outline situations where static models may be acceptable for estimating the cost-effectiveness of vaccination programs, researchers should be aware of their limitations. First, when a static model has demonstrated the cost-effectiveness of a vaccine despite not accounting for the effects of community immunity or secondary transmission, the true cost-effectiveness of the intervention may be underestimated, and could result in biased resource allocation decisions.^{Footnote 39} Second, when epidemiological or modelling data are used to estimate the magnitude of community immunity or secondary transmission in the context of static models, there may be biased cost-effectiveness estimates when the data used are from a different population than the one considered in the model and when there are other important differences. Also, if the data used are from epidemiologic equilibrium, the fluctuation in prevalence in the initial post-vaccination period will not be accounted for. This limitation is particularly important for vaccination programs with a positive time preference given that the initial time period is when most of the costs and benefits related to the vaccine are accrued.^{Footnote 59} Examples of static models include decision trees and cohort-based Markov models.

Dynamic models, which explicitly represent infection transmission, should be considered in economic evaluations of vaccination programs where human-to-human transmission is an important factor. For example, dynamic models should be employed when a large-scale vaccination program is expected to change the force of infection leading to control, elimination, or eradication of a disease by preventing its transmission.

Dynamic models should also be employed when serotype replacement and age-shifting of disease could potentially result from a vaccination program. Vaccines that are specific for certain pathogen variation (i.e., serotype, serogroup, or strain) may reduce one pathogen variation of the disease, but in the presence of multiple variations, the prevalence of infection from non-vaccine variations may still increase.^{Footnote 60}^{Footnote 61} For diseases caused by multiple variations of a pathogen, researchers should include each variation separately within the model so that infection and disease related to the emergence of new variations can be accounted for. Situations where a vaccination program leads to an increase or decrease in the average age of individuals who are infected may lead to a change in disease severity, treatment costs, and mortality, which should also be accounted for in a cost-effectiveness analysis.^{Footnote 62}^{Footnote 63} Examples of dynamic models include dynamic cohort models and individual-based simulation models.

When choosing between a dynamic or static model, researchers should consider the trade-offs between the need to represent transmission, and the additional complexities associated with dynamic models. In some situations, the decision to select one model type over another may not always be straightforward. Dynamic models are conceptually and computationally more complex than static models. Decision-makers who are the end-users of the results generated must be able to understand and interpret the structure of the model. They also need to trust that the results are a reasonable representation of what would be expected to transpire in the real-world setting after the implementation of the vaccination program. There is also a trade-off between the complexity (and realism) of a model, and the ease by which it can be understood, communicated, and validated. In some cases, transmissibility between individuals may result in spread of an infection, but the nature of the vaccination program may negate the need to represent transmission in an economic model. For example, for a universal vaccination program that is expected to achieve a high level of coverage in the population, a static model may be adequate in predicting its effects. For further guidance on whether to use a static or dynamic model when estimating the cost-effectiveness of a vaccination program, researchers could consult published schematic diagrams that delineate considerations related to this choice by Jit and Brisson and the World Health Organization (Figure 4, Table 8).^{Footnote 58}^{Footnote 59}

It should be noted that there are "hybrid" models between dynamic and static models, in which researchers do not fully account for infection transmission. Rather, they estimate the average number of secondary infections averted through the prevention of a case and incorporate the costs and benefits of preventing those cases into the analysis.

Other attributes

Although the fundamental choice facing researchers who are modelling the cost-effectiveness of vaccines is between selecting static versus dynamic modelling techniques, they must also consider other attributes related to the model structure.

Deterministic versus stochastic

Deterministic models describe what happens on average in a population and does not account for first-order uncertainty since events cannot occur randomly (by chance). In stochastic models, on the other hand, events are programmed to occur randomly, accounting for first-order uncertainty.^{Footnote 37}^{Footnote 64} For a discussion of second-order (parameter) uncertainty, researchers should refer to chapter 13 on Uncertainty.

Average parameter values used in deterministic models may realistically approximate the processes being modeled if the population at risk is large, and the infection is not close to elimination or global eradication (e.g., human papillomavirus (HPV)). For small populations, (e.g., college outbreak of meningococcal B infection), or when modelling the rise of an emerging infection or a rare infection that is on the verge of elimination (e.g., measles and polio in some countries) models that incorporate individual variability and first-order uncertainty (e.g., individual-based models) are more appropriate since they are able to account for random transmission events that are important in these situations.^{Footnote 13}^{Footnote 39}

Aggregate versus individual-based

In aggregate models (also referred to as population-based), groups of individuals are aggregated into compartments representing health states based on their characteristics. Examples include cohort-based Markov models and dynamic compartmental models. Changes over time represent shifts in the proportion of the population in each health state based on average parameter values.^{Footnote 37}^{Footnote 65}

In individual-based models, the progression of each individual with different characteristics is simulated. Models that simulate transmission between infected and susceptible individuals are dynamic, in that they have a changing risk of infection over the time horizon of the simulation.^{Footnote 37} In contrast, models that assume an exogeneous risk of infection independent of the whether there are infected people in the population are static.^{Footnote 37} An individual-based model is generally more complex, requiring more data than an aggregate model, and can be programmed stochastically so that an individual's probability of future events accounts for uncertainty related to randomness.^{Footnote 56}

The behaviour of individuals may influence the effects of a vaccination program, and the economic evaluation should account for these behaviours when relevant. These models may be programmed such that the individuals have memory and are able to alter their behaviours over time based on their previous interactions. ^{Footnote 64}

Individual-based models are also appropriate when there are significant heterogeneities between individuals in a population. These heterogeneities may be related to genetic factors, socioeconomic status, age, access to healthcare services, and behaviour changes in response to disease outbreaks, just to name a few. See chapter 14 on Equity for more equity-relevant differences. Individual-based models account for these characteristics and the effect that they could have on outcomes related to the introduction of a vaccination program.^{Footnote 66}

Aggregate models, on the other hand, are appropriate for vaccination programs for relatively homogeneous groups of individuals (e.g., a pneumococcal vaccination program for elderly individuals in one geographic area)^{Footnote 67} since they have similar characteristics that could be reasonably represented by average values as they transition through different health states. Note that aggregate models can nonetheless incorporate some heterogeneity through stratifying by risk, and/ or incorporating assortative mixing by age groups and on other risk factors.

When modelling heterogeneities between groups or individuals (including equity-relevant differences), researchers should consider how much detail is required to adequately model the cost-effectiveness of a vaccination program, and consider the trade-offs between different model types that could be used to account for these heterogeneities.

Open versus closed

Models can represent open or closed populations. Open models allow new susceptible individuals, through births and immigration, to enter and exit the model over time, while closed models do not. Although open models may be computationally more complex, they allow researchers to estimate the demographic changes of the population intended for vaccination and account for its characteristics such as risk exposure, age, and disease severity.^{Footnote 37}^{Footnote 68}

Open models should be used when the effects of vaccination programs in one cohort will affect other population cohorts (e.g., childhood immunization programs for diseases such as measles or polio). Closed models are appropriate when examining vaccination programs in small groups of individuals that are unlikely to be epidemiologically influential in the wider population (e.g., hepatitis A vaccination program for healthcare workers), or where the effects of the vaccine are short-lived (e.g., seasonal influenza vaccination program). However, note that open models are sometimes necessary in small relatively homogeneous populations with high population turnover, and that closed models can be sufficient in many cases when the population turnover is approximately net neutral. Also note that closed models with long time horizons may undercount potential costs and outcomes.

Discrete versus continuous time

Continuous time models are recommended when multiple events need to be modelled simultaneously. In disease outbreaks, for example, transmission of infection between individuals may depend on multiple factors such as contact patterns between individuals, as well as the number of infectious individuals in a given population.^{Footnote 54} Although continuous time models may provide more accurate results in such situations, these models are computationally more complex. They require use of ordinary differential equations for which solutions may be difficult to obtain. Results of continuous models may be approximated by employing discrete time models with a small time steps and appropriately rescaling parameters.^{Footnote 37}^{Footnote 54}

Model calibration

Model calibration is the process used to infer unknown model parameters by adjusting them to ensure that model outputs represent a good fit with observed data (calibration targets).^{Footnote 69} In infectious disease modelling, many parameters may be unknown or cannot be directly estimated based on available data. These could include parameters related to the natural progression of the infection or disease, details related to sexual behaviours in the case of sexually transmitted infections, and data related to uptake and screening interventions.^{Footnote 70} Calibration targets should be independent data that are accurately reported with a high degree of both internal and external validity. When appropriate, these data should be stratified by relevant subgroups to ensure adequate model performance across key population strata.^{Footnote 71} Researchers could also consider eliciting expert opinion when selecting calibration targets.

Researchers should be aware that because subjective decisions are required during the calibration process, such as selecting calibration targets, goodness-of-fit measures, and calibration method, there is uncertainty related to the calibration methods that are employed. These uncertainties can lead to considerable differences in results of economic evaluations. Although calibration is often computationally intensive, when possible, researchers may consider using multiple goodness-of-fit statistics and more than one approach for model calibration.^{Footnote 72}^{Footnote 73} Researchers should retain uncertainty in the calibration estimated parameters, which then can be used in probabilistic analysis.

Difficulty calibrating multiple model parameters may indicate that the model structure or its underlying assumptions are incorrect. It may also suggest a limited understanding of the natural history of the disease being modelled, or of the behaviours that affect its transmissibility, detection, or treatment. Alternatively, it may reveal biases, inconsistency, or imprecision in the data being used as calibration targets. As such, it should not be minimized or ignored, but rather used to help establish future research priorities.^{Footnote 39}

Model validation

Validation is the process that is used to ensure the accuracy of results generated from models used in economic evaluations. The validity of a model should be examined within a relevant decision-making context so that decision-makers are able to determine whether the model under consideration addresses the decision problem at hand.^{Footnote 74} Researchers should assess various aspects of model validity using different methods.

Face validity concerns whether a model reflects the current understanding and evidence related to the disease and vaccination program being considered. It involves the subjective assessment of a model's structure, assumptions, data sources, and results. This is best conducted by clinical experts in the field, and can also be done by comparing the model structure to standard of care. Internal validity is often referred to as verification, and refers to whether the model behaves as it should. It involves verifying that the mathematical equations used in the model have been programmed correctly. It ensures that there are no computational errors in the model. Cross-validation involves comparing the results generated from one model, and determining the extent to which they correspond to results of other models.^{Footnote 75} External validity involves comparing results generated from a model with existing data from independent sources such as clinical trials, epidemiologic studies, and routinely available population statistics such as mortality data, or electronic health records. External validation is not possible in situations where the model makes use of all relevant known data. It may be difficult in situations where these types of data do not exist, or when they are not sufficiently detailed to allow appropriate comparison.^{Footnote 76} Predictive validity refers to whether a model is performing its intended purpose, which is to predict outcomes related to a vaccination program. It is also the most difficult type of validation to perform since results must relate to events or studies conducted in the future. This type of validation is usually not applicable to decision-making related to a new vaccination program.^{Footnote 76} However, it may be relevant when developing a model based on older models. Researchers can assess the older models prior to re-use. As with model calibration, researchers could consider eliciting expert opinion when undertaking model validation processes.

9. Effectiveness

Guideline Statements

9.1 A comprehensive search of the available data sources should be conducted to inform the estimates of effectiveness and harms associated with the interventions. Report the included studies and methods used to select or combine the data. Researchers should assess sources used for effectiveness based on their fitness for purpose, credibility, and consistency. Describe the trade-offs among these criteria and provide justification for the selected source(s). [CADTH Guideline Statement with amendment]
9.2 The following criteria should be considered when assessing estimates of vaccine effectiveness: vaccine effectiveness by dose and time (e.g., waning protection); pathogen variation-specific effectiveness (i.e., serotypes, serogroups, strains); and geographic and vaccine recipient factors that may affect effectiveness.
9.3 Researchers should ensure that immune biomarkers used as surrogate outcomes in studies of vaccine efficacy or effectiveness meet the criteria for correlates of protection.

This chapter details factors that should be considered when assessing the effectiveness of vaccines, and considerations related to data synthesis, interpretation and use of surrogate outcomes, and extrapolation of effectiveness estimates.

Assessing estimates of vaccine effectiveness

There are several factors specific to vaccines that should be considered when interpreting effectiveness data. These factors are discussed below.

Efficacy is established through randomized controlled trials (RCTs), which evaluate changes in immune markers, reductions in disease severity, and improvements in health outcomes in vaccinated individuals. Effectiveness of vaccines in individuals is often different from efficacy. For example, there are often higher rates of vaccine series completion in RCTs compared to the real-world setting; there are limitations of the RCT design in capturing community immunity; and there are other differences between RCT populations and the real-world populations in which the vaccine is used.

Vaccine series completion is an important consideration for the many vaccines that require administration of multiple doses at defined time intervals. For example, the human papillomavirus (HPV) vaccine was initially administered on a 3-dose schedule, although a 2-dose schedule is now recommended for most. For the 3-dose schedule, the second dose is given 1-2 months after the first dose, and the third dose 6 months after the first dose.^{Footnote 77} Researchers should be mindful that individuals who do not receive all doses of a recommended vaccine series might experience lower rates of vaccine effectiveness than those who receive the full series. Researchers should assess both clinical trial data and expected real-world dose completion estimates, as both have strengths and limitations. Real-world data may be obtained from acceptability studies on vaccine series completion or from data on completion of other vaccine series used in similar populations with a similar number of doses. Researchers should keep in mind that residual confounding may affect results of observational studies that examine the relationship between dose completion rates and vaccine effectiveness. Specifically, factors that predict a lower probability of dose completion may also increase the underlying risk of infection (e.g., earlier sexual exposure in girls who receive fewer than three doses of HPV vaccine).^{Footnote 21} Researchers should use expected real-world dose completion estimates based on the relevant jurisdiction(s) and population intended for the vaccination program for the reference case analyses.

In terms of community immunity, RCTs may underestimate a vaccine's population-level effects. That is, community immunity is not observed in RCT participants since they represent a very small proportion of the population. Community immunity is dependent on the distribution of immunity conferred by the vaccine and natural infection within the population, the transmissibility of the infection, and contact patterns of individuals in the population.^{Footnote 78} Population-level effectiveness is usually established through observational studies, which would normally capture the indirect effects of a vaccine. Researchers should be aware, however, that studies using surveillance data are subject to the same limitations as other observational studies, and may not be appropriate to extrapolate to different settings.^{Footnote 39} In such cases, dynamic models parameterised using local epidemiological data can be used to estimate indirect effects of vaccines. When assessing whether to include estimates of vaccine efficacy from RCTs or effectiveness from observational studies in the reference case analyses, researchers should justify which data sources best represent results in populations most similar to the population(s) affected by the vaccination program to be implemented.

Geographic variation should be considered with respect to vaccine efficacy and effectiveness. Several studies have found that vaccine efficacy and effectiveness can vary by country. Several factors have been postulated to account for these differences, including: 1) differences in serotype or strain prevalence; 2) the role of climate and daily mean temperatures; 3) population heterogeneities with respect to social and demographic factors that influence vaccine efficacy and effectiveness; 4) co-administration of other vaccines (e.g., oral rotavirus and polio vaccines co-administration); and 5) differences in prevalence of other endemic infections.^{Footnote 79}^{Footnote 80}

Vaccine recipient factors should also be considered when evaluating fitness for purpose of vaccine efficacy and effectiveness data. Examples of such factors include age, genetic susceptibility to infection, inborn errors of immunity, the effect of nutrition on recipient responses, previous sensitization to organisms antigenically related to the pathogen, comorbidities, particularly those that can affect immune response, secondary immunodeficiencies due to medications, and possible genetic differences in response to a particular vaccine.^{Footnote 81} RCTs tend to include only healthy adults, especially for pivotal trials or trials conducted early in the product development lifecycle. This contrasts real-world studies which tend to include at-risk populations who would otherwise be excluded from RCTs such as pregnant persons, children, and those who are immunocompromised.

Vaccination coverage may differ between groups of individuals or by geographic area. For example, diphtheria, pertussis and tetanus vaccine coverage of four or more doses in two-year-old children differs between Canadian provinces, with Newfoundland and Labrador achieving the highest coverage (89%) and Manitoba the lowest (66%), based on the 2017 Canadian Immunization Coverage Survey.^{Footnote 82} Coverage is an important factor in determining effectiveness at the population-level through community immunity. Achieving high levels of vaccination coverage depends on the implementation strategy undertaken when a new vaccination program is introduced, as well as the ongoing strategies employed to scale up and sustain the program. For instance, health promotion, information campaigns and other efforts to build community trust may counteract vaccine hesitancy. The success of these strategies will depend on the capacity of the resources deployed, ease of access to vaccine doses in the intended population, preparedness of healthcare providers, and attitudes of both healthcare providers and the public. Incorporating these factors into economic evaluations can better align these evaluations with decision-makers' practical needs. Further, including these factors focuses decision-makers' attention on specific implementation strategies, the relative time and effort needed to execute each one, the inherent trade-offs posed by these alternative courses of action, and their independent and joint effects on population coverage.^{Footnote 83} This reinforces the importance of engaging decision-makers during the conceptualization of the economic evaluation.

Some vaccines provide protection only against some variations of a pathogen. For example, 13-valent pneumococcal conjugate vaccine is active against 13 out of over 90 known pneumococcal serotypes,^{Footnote 13}^{Footnote 84} and 23-valent pneumococcal polysaccharide vaccine is active against 23 pneumococcal serotypes.^{Footnote 85} HPV vaccines are available in bivalent, quadrivalent, and nonavalent forms, although there are over 100 HPV serotypes.^{Footnote 86} For these types of vaccines, researchers should ensure that efficacy and effectiveness data being considered are specific to the diseases caused by the variations of the pathogen targeted by the vaccines. Researchers should also be aware that, in some cases, vaccines that are specific to certain pathogen variations may also confer some degree of protection against variations of the pathogen not covered by the vaccine. An example of this cross-protection has been shown with both bivalent and quadrivalent HPV vaccines demonstrating some protection against infections and lesions associated with HPV 31, 33, and 45, which are non-vaccine serotypes.^{Footnote 86}

Data synthesis

Researchers should be mindful of vaccine-specific considerations when combining data from different sources. These include potential geographic and host factors outlined above that may be different between study populations and the population being considered in an economic analysis conducted in the Canadian setting.

Surrogate outcomes

Whenever possible, the efficacy or effectiveness of vaccines should be determined with comparative studies (either RCTs or observational studies) that report the incidence of the infectious disease targeted by the vaccine, in the vaccinated group versus relevant comparator(s).

The primary endpoint of these studies should be defined as clinically apparent infection that meets clinical and laboratory diagnostic criteria. In some situations, it may not be possible to measure cases of clinically apparent infection. One example of such a situation arises when the incidence of the infection is too low to measure in a study, typically constrained by its study time period and study population size. This situation occurs with rare infectious diseases (e.g., meningitis due to meningococcal group B infection), or those that rarely afflict the population because current vaccines provide effective prevention.^{Footnote 87}^{Footnote 88} Another example arises with seasonal influenza vaccines, many of which receive provisional approval based on immunogenicity alone.^{Footnote 89}

In these situations, correlates of protection (CoPs), which are immune biomarkers (antibodies or T-cells) that predict vaccine efficacy in vaccinated individuals, can be used as surrogate endpoints.^{Footnote 87}^{Footnote 90}^{Footnote 91} Researchers should be aware that multiple CoPs can exist for a single vaccine,^{Footnote 92}^{Footnote 93} and that different vaccine types and formulations indicated for the same disease may be associated with different CoPs.^{Footnote 94}^{Footnote 95} For multivalent vaccines that provide protection against multiple variations of a pathogen, higher titers of the CoP may be required for protection against some variations compared to others.^{Footnote 96} Finally, it is important for researchers to identify which dimension of prevention (e.g., preventing infection, preventing disease, reducing severity of disease) is linked to a CoP since correlates may differ quantitatively and qualitatively based on the preventive outcome being considered.^{Footnote 97}

Extrapolation

The duration of clinical trials is often not long enough to ascertain the duration of protection provided by a vaccine, and researchers have to extrapolate estimates of duration of protection from clinical trial data.^{Footnote 98}^{Footnote 99} Several different modelling techniques (e.g., logarithmic waning, exponential waning) can be used to generate duration of protection estimates, which can vary widely based on the technique chosen. Consequently, cost-effectiveness estimates can be sensitive to assumptions on duration of protection.^{Footnote 98} This has been demonstrated with cost-effectiveness analyses of herpes zoster vaccine (Zostavax^®) in Belgium, where the authors found that cost-effectiveness estimates varied considerably based on the choice of model used estimate to vaccine efficacy.^{Footnote 42} Specific guidance on addressing uncertainty of the estimates of duration of protection is provided in chapter 13 on Uncertainty.

10. Measurement and valuation of health

Guideline Statements

10.1 In both reference cases, the quality-adjusted life year (QALY) should be used as the method for valuing health outcomes.
10.2 "Health preferences should reflect the general Canadian population." [CADTH Guideline Statement]
10.3 In the reference cases, researchers should use health preferences obtained from an indirect method of measurement that is based on a generic classification system (e.g., EuroQol 5-Dimensions questionnaire [EQ-5D], Health Utilities Index [HUI], Short Form 6-Dimensions [SF-6D], Child Health Utility 9-Dimensions [CHU9D], Assessment of Quality of Life [AQoL]). Researchers must justify where an indirect method is not used. [CADTH Guideline Statement with amendment]
10.4 A comprehensive search of the available data sources should be conducted to inform the health state utility values. Report the included studies and methods used to select or combine the data. Researchers should assess sources used for health state utility values based on their fitness for purpose, credibility, and consistency. Describe the trade-offs among these criteria and provide justification for the selected sources. [CADTH Guideline Statement with amendment]

Decision-makers are concerned with health outcomes as they are the key output of health interventions. Quality-adjusted life-years (QALYs) are the metric used to quantify health outcomes in a cost-utility analysis (CUA). QALY estimates are generated by combining data on survival and health-related quality of life (HRQoL). To estimate QALYs, HRQoL data in the form of a summary measure, often referred to as a health utility, is required.

Health utility data

The utilities obtained from HRQoL instruments should represent the preferences of the general Canadian population, consistent with the social decision-making viewpoint adopted by these Guidelines. Population preferences for health states defined in an HRQoL instrument are normally elicited from a sample of the general population using methods such as standard gamble or time trade-off.

Although it is possible to elicit health utilities directly from respondents, instruments designed to capture health utilities indirectly provide a more efficient and consistent method of obtaining this information. Both disease-specific and generic HRQoL instruments are available to obtain indirect health utility measurements. The most commonly used generic instruments are the EuroQol 5-Dimensions questionnaire (EQ-5D), Health Utilities Index (HUI), and the Short Form 6-Dimensions (SF-6D). Instruments for children's HRQoL include Child Health Utility 9-Dimensions (CHU9D), KIDSCREEN Quality of Life Questionnaire, Pediatric Quality of Life Inventory (PedsQL) Generic Cores Scales, EQ-5D-Youth (EQ-5D-Y), and Assessment of Quality of Life (AQoL). Researchers should use HRQoL data obtained from a generic instrument to estimate QALYs to ensure comparability between vaccination programs being considered by decision-makers. Where multiple estimates of utilities are available, source studies should be subjected to formal quality appraisal using a suitable quality appraisal tool.^{Footnote 100}

It must be recognized that there are no valid instruments for directly measuring utility in neonates, newborns, infants or young children, although this is an active area of current research.^{Footnote 101} Moreover, the construct of HRQOL for children differs by age group and is conceptually different than adults.^{Footnote 102} While several pediatric-specific, preference-based measures of HRQoL have been developed recently (e.g., EQ-5D-Y, CHU9D, AQoL), all have lower age limits and typically rely on tariff sets derived from adult populations. The convergent validity of pediatric-specific and adult preference-based HRQOL measures requires study. Despite the limitations, researchers should ideally use utilities for child health states sourced from a pediatric-specific generic instrument, as opposed to using adult utilities. If a pediatric-specific generic instrument is not used for a child health state, this should be justified and its impact tested in sensitivity analysis. The use of generic instruments is encouraged in pediatrics, despite direct elicitation methods being frequently used. This is because utilities generated from direct elicitation for health states are sensitive to framing. In cases where utilities may be missing due to a child's young age (e.g., under 5 years), assumptions used should be explicit and justified. Preferences should be from the general population, supplemented with child valuations if available. Proxy respondents (e.g., by parents or healthcare providers) are often required in pediatrics because valuation methods can be cognitively difficult or require reading comprehension. However, proxy responses can systematically differ from child self-report, and the directionality of the discrepancy is difficult to predict.^{Footnote 103} Researchers should use child utilities from instruments that are self-reported where possible, and specify if proxies are used. Further, many vaccines are given in infancy or childhood, some of which prevent diseases in childhood and others in diseases that emerge in adulthood. Researchers should explicitly state which health states in a model are related to child health states and which relate to future adult health states. In economic evaluations where adults and children are modelled, consistency in the use of instrument across ages is encouraged. In cases where data are unavailable such as for vaccines that prevent rare diseases, researchers should provide justification for how they are handling the data limitations.

In these Guidelines, the concept of health is based on the World Health Organization's definition: "a state of complete physical, mental and social well-being and not merely the absence of disease or infirmity".^{Footnote 5} The breadth of the definition necessarily makes valuation of this conceptualization of health difficult. Hence, generic instruments which primarily focus on physical and sometimes mental attributes are used to capture HRQoL for many diseases for pragmatic reasons. Researchers should assess and justify the data source(s) based on fitness for purpose, credibility, and consistency. This includes assessing the extent to which the instrument's descriptive system captures what individuals and decision-makers want to measure for the infectious disease of interest. Example domains that may not be well captured consistently across different instruments include mental health, tiredness, and the senses. Where the instrument is deemed to omit important domains or to inadequately capture them, researchers should consider complementing the generic instrument(s) with a disease-specific instrument. This can identify any omissions and the extent of the discrepancy in measuring health loss between generic and disease-specific instruments.

Health utility data used to populate an economic model are often derived from published literature. To ensure consistency within a model, health utility valuations for all health states included in the model should be obtained from the same instrument and use preference weights obtained from the same population, whenever possible.^{Footnote 104} When this is not possible, researchers should consider trade-offs between fitness for purpose, credibility, and consistency for the available data. In these cases, researchers may also consider pooling health utility data using techniques such as meta-analysis or meta-regression, although the usefulness of these methods may be limited by the considerable heterogeneity in the valuation methods and study populations.^{Footnote 105} Researchers should explore uncertainty in health utilities in sensitivity analyses.

In addition to including health utility data for the population intended for the vaccination program and any population(s) that may experience externalities related to the program, researchers should include health utility data for informal caregivers in cases where potential spillover effects have been identified that could affect the health states of this population. A discussion of methodological issues of QALY-related spillover effects and a catalogue of preference-based values for spillover effects can be found elsewhere.^{Footnote 106} The analysis should not include health utility data for formal caregivers to avoid double counting of their costs and QALYs. This is discussed further in chapter 12 on Analysis.

A more detailed discussion on HRQoL measurement and data can be found in CADTH's Guidelines for the Economic Evaluation of Health Technologies: Canada 4th Edition under chapter 10 Measurement of valuation of health.^{Footnote 1}

Quality-adjusted life years in societal perspective economic evaluations

There is uncertainty about whether QALYs capture only health benefits, or whether they also, implicitly or explicitly, capture non-health-related effects. This uncertainty is particularly relevant to CUAs conducted from the societal perspective since these analyses are concerned with not only costs and outcomes borne by the health system, but also with costs and outcomes that fall onto non-health sectors. Specifically, uncertainties exist on how to include the impacts of productivity and consumption in the incremental cost-effectiveness ratio (ICER) estimate.

As noted above, QALYs are estimated using survival and HRQoL data. The HRQoL data are often elicited based on health domains included in common HRQoL instruments. However, it is unclear whether, or to what extent, respondents implicitly consider non-health-related factors when valuing these health states. In particular, the degree to which respondents consider how changes in productivity and consumption may affect their HRQoL, and the extent to which these considerations are implicitly incorporated into respondents' valuation of their health states have received attention. The available evidence suggests their influence is limited.^{Footnote 107}^{Footnote 108}^{Footnote 109}^{Footnote 110}

If individuals were to account for the economic effects of productivity when valuing their health states, then including monetary estimates of productivity in the numerator of the ICER estimate, along with costs of other resources, double counts these impacts.^{Footnote 109} The current consensus is that productivity and income changes are not likely to be captured in QALY estimates.^{Footnote 111}^{Footnote 112} This supports the inclusion of losses in productivity in the numerator of the ICER estimate.

Similarly, questions have been posed about whether respondents in health state valuations consider and value non-medical consumption such as clothing and housing. One argument posits if the utility value of this consumption is not (implicitly) captured in QALYs, it would be inconsistent to include changes in such consumption on the costs side of the evaluation and therefore, these costs should be excluded.^{Footnote 110} Another argument posits that non-medical consumption (e.g., daily food intake) is necessary to stay alive. Even if respondents would not consider this in their valuations of health states, it should nonetheless be included as a cost. This would be even more evident if respondents would assume usual levels of consumption in answering health state valuation questions. The same argument applies to other non-medical consumption, which to some extent may also contribute to an individual's HRQoL.^{Footnote 108}

In contrast to findings related to respondents' consideration of productivity changes when valuing health states, evidence suggests that respondents do consider utility of consumption when valuing health states.^{Footnote 113} This suggests that health improvements may lead to increases in the marginal utility of non-health-related consumption. Although further research is required to corroborate these findings, they do provide justification for including consumption costs in the numerator of the ICER estimate.

In summary, for the societal perspective reference case analysis, changes related to productivity and costs of non-health-related consumption should be included in the numerator of the ICER estimate. Further details on quantifying the costs of productivity and consumption are found in chapter 11 on Resource use and costs.

11. Resource use and costs

Guideline Statements

11.1 For each reference case analysis, researchers should systematically identify, measure, value, and report all relevant resources consumed or saved as a result of the delivery or implementation of the vaccination program under consideration.
11.2 Where possible, researchers should value the consumption of relevant resources identified for all sectors in monetary terms. In situations where this is not possible, researchers should present the relevant resources that have been identified in the Impact inventory table for economic evaluations of vaccination strategies (Appendix I) for consideration by decision-makers.
11.3 Resource use, unit prices, and costs should be based on Canadian sources and reflect the jurisdiction(s) of interest (as specified in the decision problem). [CADTH Guideline Statement with amendment]
11.4 When valuing and monetizing resources, researchers should select data sources that most closely reflect the opportunity cost, given the perspective of the analysis. [CADTH Guideline Statement with amendment]
11.5 A comprehensive search of the available data sources should be conducted to inform the resource use and cost values. Report the included studies and methods used to select or combine the data. Researchers should assess sources used for cost data based on their fitness for purpose, credibility, and consistency. Describe the trade-offs among these criteria and provide justification for the selected sources.

Both increases and decreases in consumption of resources and services may result from vaccination programs. They can be related to the implementation of the vaccination program and its ongoing delivery, as well as to downstream effects of the program. Resource consumption may fall upon vaccinated individuals including the population(s) at risk for the disease of interest when the vaccination program is associated with externalities, and the population that experiences spillover effects (e.g., informal caregivers). Furthermore, resources consumed as a result of vaccination programs may fall within the health system sector or outside the health system. Researchers should use the Impact inventory table for economic evaluations of vaccination strategies to identify all potential resources and services associated with the vaccination program under consideration. When the resources and services occurring as a result of a vaccination program have been identified, researchers should determine which of the resources consumed can be measured and valued in monetary terms.^{Footnote 114}^{Footnote 115} Where resources or services are not related to production or consumption, they can be considered transfer costs-that is the monetary transfer from one person or group of people in society to another. Transfer costs are discussed at the end of the chapter after the discussion of key resource use and costs below.

Health system costs

For the health system perspective reference case analysis, researchers should identify and include all resources within the publicly funded health system that are consumed through the delivery of the vaccination program, and resources that are consumed or saved as a result of its implementation. All health system costs incurred over the analytic time horizon should be included. As previously noted, for the purposes of these Guidelines, "health system" refers to both healthcare clinical services and Public Health; and Public Health is stylized in capital letters to better distinguish it from "public healthcare".

Healthcare costs

When assigning local unit costs to resources that have been identified as relevant to the decision problem being modelled, researchers should consult the Guidance Document for the Costing of Healthcare Resources in the Canadian Setting: Second Edition.^{Footnote 116} The document provides key recommendations and data sources for identifying, valuing, and measuring costs within the Canadian healthcare system. For analyses that are conducted within or across multiple jurisdictions, variations in the public financing of specific resources and services should be indicated. Researchers should report whether a single price set or multiple jurisdictional price sets are used, as well as report methods used for assigning price sets to multi-jurisdictional data. For an analysis from the national health system perspective, ideally Canada-wide sources should be used. Where this information is not available, researchers may use costs from a representative Canadian jurisdiction or a weighted average of costs (costs from jurisdictions weighted by the proportion of use in Canada represented by the jurisdiction). Where Canadian data are not available, researchers may consult experts or use cost estimates for similar or new interventions, and look to international sources as a guide. Scenario analyses should be conducted to test the sensitivity of these alternative data sources on the results of the economic evaluation.

Among the healthcare costs listed in the Impact Inventory Table, formal caregiving and future healthcare costs are described in further detail to supplement the Canadian costing guidance.^{Footnote 116}

Formal caregiving

Individuals requiring a caregiver may receive this care from either a professional caregiver such as a nurse or a nursing assistant who is hired to perform these services, or an informal caregiver, usually a family member. Professional caregiver time should be valued at the hourly wage rate that would be paid to an individual who performs this service. Formal caregiving costs may or may not be incurred by the publicly funded health system, depending on the precise nature of these costs and the relevant jurisdiction. Hence, formal caregiving costs appear in the Impact Inventory Table under both "publicly funded health system costs" and "healthcare costs not funded by the health system". Informal caregiving is discussed later in the chapter under Productivity.

Future healthcare costs

For vaccination programs that confer a survival benefit in relation to the comparator(s) being considered in the economic evaluation, researchers should include future related healthcare costs (i.e., related to the infection and disease of interest) and unrelated healthcare costs in the reference case analyses. This recommendation is underpinned by the following considerations: 1) there is an opportunity cost associated with life-prolonging interventions since they increase future health expenditures in those individuals- expenditures which could have been used towards other individuals' healthcare needs2) it is often difficult to distinguish related costs from unrelated costs, such as in the case of different diseases that share overlapping physiologic pathways (e.g., diabetes and coronary heart disease), potentially leading to arbitrary decisions about which costs are related versus unrelated; and 3) internal consistency- the benefits related to future medical spending are already included in cost-utility analyses (CUAs) through estimates of survival and quality of life, and are based on the assumption that the individual will receive future medical care, both related and unrelated to the condition of interest.^{Footnote 108}^{Footnote 114}^{Footnote 117}^{Footnote 118}

Excluding future costs leads to lower incremental cost estimates and incremental cost-effectiveness ratio (ICER) estimates for life-prolonging interventions, and may make them seem more economically attractive than those that improve quality-of-life. Including future costs, however, increases incremental cost estimates and ICERs for life-prolonging interventions, leading in some cases to a do-nothing option (i.e., where patients do not survive) being more cost-effective than providing treatment for a disease.^{Footnote 119} In some cases even relatively inexpensive life-prolonging interventions in patients with high costs of ongoing care may not be cost-effective when future costs are considered in an economic evaluation.^{Footnote 120} Researchers should present outcomes and costs in a disaggregated manner so that decision-makers are aware of how individual components included in the analysis contribute to the overall cost-effectiveness of the vaccination program. Researchers should present: 1) the expected outcomes of the vaccination program and the comparator(s); 2) the direct health system costs resulting from the vaccination program and the comparator(s) but excluding costs of future care; and 3) the expected increase in costs of ongoing care resulting from improved survival for the vaccination program and comparator(s).^{Footnote 120}

Estimates of future healthcare costs may be obtained from data published by the Canadian Institute for Health Information's National Health Expenditure Trends.^{Footnote 121} In situations where cost estimates are required for populations with high costs of ongoing care (e.g., dialysis patients, solid organ transplant recipients),^{Footnote 122}^{Footnote 123} researchers may have to consult the published literature to obtain these estimates.

Public health costs

Public Health costs may represent a large share of the costs associated with vaccination programs and management of infectious diseases. Accurately quantifying these costs is necessary to ensure that results generated from economic evaluations of vaccination programs are valid, and lead to optimal funding decisions. Public Health costs can be categorized as either program-related costs or intervention-related costs. Program-related costs are the costs of program implementation, delivery and sustainment costs including Public Health campaigns and health promotion activities; transaction costs related to introduction of new vaccines or switching between vaccines; and costs related to population-based screening, epidemiological surveillance, contact tracing, and investigation and management of outbreaks. Specific components that should be considered when quantifying these costs include personnel costs, overhead costs, travel costs, and other service-related and administrative costs.^{Footnote 116}^{Footnote 124} Specific components that should be considered when quantifying costs of disease outbreaks include laboratory serologic testing; personnel time related to contact tracing, symptom screening, travel, monitoring, and follow-up; post-exposure prophylactic vaccines or immune globulin doses and associated administration costs.^{Footnote 125}^{Footnote 126}^{Footnote 127}^{Footnote 128} Intervention-related costs include costs of vaccine doses, distribution (e.g., transportation of vaccines and cold storage), and administration of the vaccine, including any wastage and ancillary supplies required. Researchers should present costs related to different aspects of implementation and ongoing delivery of the vaccination program in a disaggregated manner. Further, researchers should elaborate on the different levels of intensity of the implementation strategy, which is especially relevant for Public Health campaigns and health promotion activities for instance, as they can produce different levels of benefit.

Given the paucity of published data on program-related Public Health resource use and prices in the Canadian setting, researchers may have to rely on data obtained from local Public Health authorities or provincial/ territorial ministries of health through personal communication. Although costs from local Public Health authorities and provincial/ territorial ministries are jurisdiction-specific, they may be generalizable to other areas. When determining the applicability of data from one jurisdiction to another, researchers should consider factors such as geographic similarities, population characteristics, and epidemiologic patterns.

There are limited Canadian data available on intervention-based Public Health resource use and prices. Some provincial Public Health agencies, such as the Institut national de santé publique du Quebec,^{Footnote 129} publish findings of their work online, which may include epidemiologic surveillance and cost data relevant to the economic evaluation of a vaccination program. The actual price paid by governments for vaccine doses is confidential. Researchers should use the manufacturer's list price in the reference case analyses and conduct deterministic sensitivity analyses using plausible discounted prices. See chapter 13 on Uncertainty for further discussion. Researchers should also consider cost items related to the administration of vaccine doses, as they may vary considerably based on the setting of delivery. For example in Alberta, community-based delivery of human papillomavirus (HPV) vaccine is considerably more expensive than school-based delivery.^{Footnote 130} Resources and services related to providing culturally safe access to healthcare and vaccination program communication materials should also be considered in situations where they are applicable.

Healthcare costs not funded by the health system

Some services associated with vaccination programs may not be reimbursed or publicly funded by the healthcare system. Services that are excluded from the publicly funded health system may vary by jurisdiction or region. Examples of such costs include formal caregiver services, long-term care services, rehabilitation in a facility or at home, home care, drug treatments for individuals who do not have coverage through a publicly funded drug insurance program, out-of-pocket costs such as non-prescription drugs, as well as ancillary costs related to items such as private insurance copayments, dental and vision care, assistive devices, physiotherapy and others. These costs may be funded through private insurance plans, by the individual(s), or a combination of both. Regardless of how these costs are funded, they should be quantified and included in the incremental costs and ICERs for the societal perspective reference case analysis.

Non-healthcare system costs

Researchers should identify all resources consumed as a result of the implementation or ongoing delivery of the vaccination program that fall outside of the health system and quantify their corresponding costs. For example, relevant non-health impact areas for the societal perspective reference case analysis could include: direct out-of-pocket costs (e.g., transportation costs, accommodation costs), paid and unpaid labour time losses, non-medical consumption, and services funded by other sectors including education, social services and community services, and environment. Guidance on identifying resources and quantifying costs for non-healthcare sectors is presented below.

Direct out-of-pocket costs

Estimates of direct out-of-pocket costs (e.g., transportation costs, accommodation costs) should be included in economic evaluations of vaccination programs. Transportation costs include costs related to public transit, including fully accessible barrier-free transportation when necessary, taxis, personal vehicle use, and parking fees.^{Footnote 131}

Losses in productivity

Losses in productivity (also known as productivity costs) are output losses associated with productive time spent in paid labour, or unpaid labour (e.g., volunteering, helping, mentoring) including household production (e.g., cooking, cleaning, shopping, raising children).^{Footnote 114}

Researchers should consider the effects of vaccination programs on the productivity of vaccinated individuals and caregivers, and where applicable, on macroeconomic consequences. Vaccine-related productivity improvements may occur through: 1) increased paid and unpaid labour productivity related to either prevention of illness, or decreased severity of illness in vaccinated individuals; and 2) increased productivity of caregivers related to decreased care needs for sick individuals.^{Footnote 8}^{Footnote 11}^{Footnote 132} Productivity may decrease from time off paid and unpaid work for attending or accompanying others to vaccination appointments. When productivity gains for life-prolonging interventions are included in an analysis from the societal perspective, they may attenuate or offset increased incremental costs due to increased future healthcare consumption in survivors. Each type of productivity loss listed in the Impact Inventory Table is described in further detail below.

Paid and unpaid work

There are two primary methods for quantifying lost productivity related to paid work: the human capital approach, and the friction cost approach. The human capital approach is based on the cost of forgone productive time, whereas the friction cost approach attempts to estimate overall societal production losses, assuming replacement of ill workers in the formal labour market. ^{Footnote 114}^{Footnote 133}^{Footnote 134}^{Footnote 135}

The human capital approach is commonly used to value lost production. It typically requires estimates of time lost from paid work, and averages wage rates of the involved individuals. As such, it may be seen as estimating the lost production (or income) from an individual perspective, due to illness, disability, or death. As the human capital method does not account for societal replacement mechanisms, especially for longer periods of absence (e.g., in case of disability or premature death), it has been suggested that it likely overestimates the true cost of lost production from a societal perspective.^{Footnote 136} This is a particularly important consideration in situations where childhood death or lifelong disability may be avoided as a result of a vaccination program.

The friction cost approach, on the other hand, attempts to quantify lost productivity on a societal level based on the assumption that production levels can be restored by substituting labour for labour (e.g., in case of unemployment) or for capital.^{Footnote 133} This implies that after some 'friction period' production losses cease to occur from a societal perspective. Macro-economic consequences of changes in labour supply and unemployment benefits have been estimated to be small for typical health care programs. Applying this method requires more detailed information on periods of absence, the available labour pool, and the relevant friction period in a jurisdiction.^{Footnote 135}

While both methods primarily focus on valuing lost production in the context of paid work, changes in productivity related to unpaid work should also be captured. Lost productivity in the context of unpaid work can be captured by valuing lost hours with an appropriate value. Estimations of (changes in) productive time in unpaid work for the relevant population may be difficult to obtain in some cases.^{Footnote 137} Other than using general estimates from existing sources, questionnaires may be used to estimate these changes. ^{Footnote 138}

Researchers should calculate total change in productive time, related to both paid and unpaid work, attributable to the vaccination program. Researchers should account for losses of an individual's productive time related to obtaining a vaccine, seeking treatment, illness, disability, and death of vaccinated or otherwise affected individuals. Changes in productivity associated with vaccination programs should be quantified using the human capital approach. Given that it is the most commonly recommended approach in pharmacoeconomic guidelines across different countries,^{Footnote 139} it allows increased comparability between economic evaluations of vaccination programs undertaken in different jurisdictions.

For the societal reference case analysis, researchers should include the full-time period over which affected individuals are expected to incur paid production losses. These losses should be valued based on age-specific average employment income and number of hours worked based on Statistics Canada data ("Income of individuals by age group, sex and income source, Canada, provinces and selected census metropolitan areas", and "Average usual and actual hours worked in a reference week by type of work (full- and part-time), annual") ^{Footnote 140}^{Footnote 141} combined with the disease-specific likelihood of an individual participating in the labour force. Using the same wage rate for both genders is a correction for measurement bias because females are on average paid less than male for the same work. ^{Footnote 142} This recommendation attempts to circumvent existing discrimination issues in society translating into health policy decisions.

In most cases, there will be equity considerations related to whether and how productive time is valued. If productive time is differentially valued based on attributes such as age, gender, or health status, results of an economic evaluation could favour groups with the greatest income-earning potential and disadvantage other groups such as children who do not work or individuals with disabilities or severe health conditions that prevent them from holding high-income jobs. ^{Footnote 114} In these situations, researchers should conduct an additional sensitivity analysis using the average income and the average number of full-time hours worked for all Canadians based on Statistics Canada data. ^{Footnote 140}^{Footnote 141} Although the measurement of these losses is imperfect and biased towards high-wage earners, this approach reveals the efficiency losses that decision-makers need to be prepared to accept each time they choose an option that is neutral to individual characteristics with respect to production.

To account for the likely overestimation of production losses associated with the human capital approach, researchers should include an additional sensitivity analysis using a naïve friction cost approximation. That is, the sensitivity analysis should account for production changes for a single year. After one year, the production changes are set to zero. Average yearly income and average yearly number of hours worked for all Canadians should be used for this analysis based on Statistics Canada data. ^{Footnote 140}^{Footnote 141} The sensitivity analysis will allow decision-makers to assess the impact of the valuation method on the results of the economic evaluation.

Although productivity losses may result from both absenteeism (time off work) and presenteeism (continuing to work but with reduced productivity), researchers are not required to account for the effects of presenteeism in their estimates of lost productivity in the reference case analysis. It is often difficult to collect this information given that it requires survey data from affected individuals and recall can be subjective in many cases. ^{Footnote 138}^{Footnote 143}

Lost unpaid production should be valued by estimating lost hours of unpaid work, and valuing this using the replacement cost method (described below). Although unpaid work may differ in terms of tasks performed and required skills, for the reference case analysis, lost hours should be valued using the wage-rate of a professional. Researchers should exclude costs of lost leisure time from the economic evaluation of vaccination programs.

Informal caregiver productivity

Individuals requiring a caregiver may receive this care from either a professional caregiver, or an informal caregiver, usually a family member. Two approaches have been proposed for valuing informal caregiver time: 1) the replacement cost approach; and 2) the opportunity cost approach. The replacement cost approach is based on the estimated cost of hiring a paid caregiver should informal care not be available. The opportunity cost approach is based on the cost of displaced productive time that results from time spent providing informal care.^{Footnote 144} Since individuals may receive a mix of formal and informal care, researchers should use the replacement cost approach to value caregiver time for the societal perspective reference case analysis. Such estimates can be used alongside estimates of potential health spillover effects due to informal care, captured in terms of caregiver QALYs.^{Footnote 145}

Macroeconomic consequences

Although most vaccination programs are unlikely to have large macroeconomic impacts, those that are designed to prevent widespread disease pandemics, such as the coronavirus disease 2019 pandemic, could attenuate important consequences. Macroeconomic impacts include labour supply shocks and widespread business closures, which may affect labour pools and workforce participation rates, and changes in household consumption preferences.^{Footnote 146} In cases where the macroeconomic costs are substantial, researchers should discuss the impacts qualitatively. Researchers may also consider addressing the costs quantitatively by engaging macroeconomic modellers and other experts.

Overall, the recommendations on productivity advocate for transparency. The stratification of subgroups in addition to an overall analysis can reveal any large effects from heterogeneity. Further, the recommendations aim to provide decision-makers with several sensitivity analyses and a choice of how to weigh efficiency gains in various subgroups (e.g., high-wage earners).

Non-medical consumption

Non-medical consumption represents costs of non-health-related items that contribute to overall welfare. These items include individual financial expenditures, and consumption of public goods and services such as clean water and safe roads.^{Footnote 108}^{Footnote 114} Researchers should include consumption costs whenever they will be altered by the vaccination program.

Researchers should use Statistics Canada data on household spending as the information source for non-medical consumption ("Household spending, Canada, regions and provinces").^{Footnote 147} In order to obtain an estimate of non-medical consumption from this table, researchers should subtract the "health care" estimate from the "total current consumption" estimate. Estimates of individual consumption should be obtained by adjusting household consumption estimates using the Oxford-modified equivalence scale, to account for consumption by household size.^{Footnote 148} The adjustment reflects the fact that one-person households would have higher per-person consumption compared to multi-person households. The Oxford-modified equivalence scale assigns a value of 1 consumption unit to the first adult, of 0.5 to each additional adult and of 0.3 to each child under 14 years of age. Further information on equivalence scales can be found in the Organisation for Economic Co-operation and Development Framework for Statistics on the Distribution of Household Income, Consumption and Wealth (pages 67 to 68).^{Footnote 149} For vaccination programs that result in changes to consumption, researchers should subtract individual estimates of consumption from individual estimates of productivity during the relevant time period. To ensure consistency between estimates of productivity and consumption, estimates of consumption should not be stratified by gender for the reference case analyses.

Education

Vaccination programs may affect educational outcomes by preventing diseases that lead to serious morbidities that, in turn, could affect an individual's level of educational achievement. For example, a Danish study found that children with bacterial meningitis experienced lower levels of educational achievement and economic self-sufficiency in adulthood.^{Footnote 150}

Higher levels of educational achievement are associated with a greater likelihood of labour market participation and higher labour market earnings.^{Footnote 151}^{Footnote 152} In Canada, it is estimated that each additional year of schooling increases lifetime earnings by approximately 11 to 12%. Assuming that decreasing an individual's education level similarly decreases lifetime earnings, it is estimated that each month of education loss could result in an approximately 1% drop in lifetime earnings.^{Footnote 153} Changes in earnings related to educational achievement should be accounted for in estimates of lost (or gained) productivity, and researchers should ensure that these costs are not double counted when considering educational impacts of vaccination programs. While there is recognition that the methods are still emerging, there is value to presenting impacts on the education sector to decision-makers.

In addition to effects on educational achievement and labour market productivity, vaccination programs may result in direct effects on the education sector. For example, children who have recovered from bacterial meningitis may experience cognitive impairment, hearing loss, seizures, and learning disabilities,^{Footnote 154} and may require in-school special education resources. Boards of education and schools may also invest in vaccination delivery programs as well as ancillary programs to, for example, improve indoor air quality during the coronavirus disease 2019 pandemic.

Researchers should consider potential education-related outcomes and direct effects on the education sector that could result from the vaccination program and the comparator(s) being considered. Where possible, these effects should be monetized for inclusion in the ICER estimate. For outcomes that may be difficult to monetize (such as disruptions to learning as a result of school-based vaccine delivery, pediatric disease and disability, or death/disability of a close family member), researchers should nonetheless identify them and include in them in the Impact inventory table for economic evaluations of vaccination strategies for consideration by decision-makers.

Social services and community services

Vaccination programs may affect social services, community services, and child and youth services by preventing diseases that lead to serious morbidities. Examples include disability support, awareness programs, family respite, and programs to improve access to vaccination programs. Researchers should identify, and where feasible monetize, consequences of social services and community services.

Environment

Vaccination programs and the comparator(s) included in the analysis may have environmental impacts related to the manufacture or distribution of vaccine doses, as well as to implementation the program. For example, vaccines have been shown to reduce antibiotic use,^{Footnote 155}^{Footnote 156} which may lead to decreased levels of residual antibiotics from sources such as households, the pharmaceutical industry, and hospitals in wastewater- which has been identified as an environmental source of antibiotic resistant organisms.^{Footnote 157}

Environmental impacts should be identified and included in the Impact inventory table for economic evaluations of vaccination strategies for consideration by decision-makers. They should be monetized where possible, although this is sometimes difficult to do. For example, further research is needed to determine the social value of antibiotic consumption and antibiotic resistance prevalence. On the other hand, there are other environmental costs to society that can be more easily valued. For instance, in cases where the intervention is life-prolonging, carbon consumption as well as food and non-food waste can be included in the societal perspective reference case analysis. Carbon consumption data are available in Canada (such as through the Hot or Cool Institute, and the Government of Canada Canadian Environmental Sustainability Indicators directorate).^{Footnote 158}^{Footnote 159} Food waste data and municipal costs of waste management are available in Canada (such as through Community Research Connections Research, and Environment and Climate Change Canada, respectively) and can be converted to per kilogram costs.^{Footnote 160}^{Footnote 161}

Other areas

Researchers should consider other sectors that may offer services or programs relevant to specific vaccination programs. Examples of such sectors could include the housing sector (e.g., changes in type of housing or adaptations to housing required because of functional disabilities resulting from infection, improvements to ventilation or reduction in crowding to reduce infection transmission).

Transfer costs

Pragmatically, researchers can equate costs with prices when parameterizing economic evaluations for vaccination programs, unless there is information to make the distinction- in which case, researchers should use costs. Cost is the amount required to produce, acquire, distribute, or sell resources and services, whereas price is the amount of money expected in exchange for resources or services. Profit margins come from the difference in price charged to payers and costs incurred, and represent transfer costs. Transfer costs apply to both the publicly funded health system perspective and the societal perspective. Take, for example, vaccine price, which would normally cover research and development costs, marginal or average production costs, distribution costs, and a profit margin. The exact cost components are often unknown to researchers and decision-makers; hence, the distinction between the cost and price cannot be made. Furthermore, it is not known how transfer costs may be redistributed or reinvested. For instance, part of the monetary transfer from health authorities to the manufacturer for the purchase of vaccine doses (profit) can be used by the manufacturer in any manner to subsequently yield utility or welfare. The counterfactual is that the health authorities keep what would have been the manufacturer's profit to subsequently use, again, in any manner to yield utility or welfare. The utility generated from either the manufacturer's reinvestment or the health authorities' reinvestment is unknown. Further discussion on transfer costs in relation to vaccine pricing can be found in chapter 14 on Equity insomuch as they relate to perceptions of health equity and social justice.

12. Analysis

Guideline Statements

12.1 Incremental costs, incremental effectiveness, incremental cost-effectiveness ratios (ICERs) and, where useful for interpretation, net monetary benefits or net health benefits, should be calculated for both reference case analyses.
12.2 "For analyses with more than two interventions, a sequential analysis of cost-effectiveness should be conducted following standard rules for estimating ICERs, including the exclusion of dominated interventions." [CADTH Guideline Statement]
12.3 The expected values of costs and outcomes, where possible, should be generated probabilistically to reflect the overall uncertainty in the model parameters.

Researchers should generate two sets of estimates of expected values for costs related to each intervention considered in the economic evaluation: one for the publicly funded health system perspective reference case analysis, and the other for the societal perspective reference case analysis. One estimate of expected values for outcomes (i.e., quality-adjusted life-years (QALYs)) should be generated for use in both reference case analyses.

Costs and outcomes (i.e., QALYs) should be generated probabilistically, where possible, so that the values reflect the overall uncertainty in the model parameters. In most cases, the probabilistic analysis will take the form of a Monte Carlo simulation, where an appropriate point estimate, range, and probability distribution are applied to each parameter. Researchers can refer to guidance elsewhere on probability distributions to use.^{Footnote 162} Each simulation should produce estimates for mean costs and mean effectiveness for each comparator, as well as estimates of incremental costs and incremental effectiveness. All values, including incremental estimates, must be reported with 95% confidence or credible intervals as indicators of precision. These intervals can be obtained from the 2.5% and 97.5% bounds from the generated simulations. Additional indicators of precision may also be appropriate if the distribution of uncertain outcomes is not approximately Gaussian. In cases where probabilistic analyses are not possible, estimates of these values should be generated deterministically. This scenario is most likely to occur when the computational power required for a probabilistic analysis is a limiting factor, especially for agent-based models.

For the publicly funded health system reference case analysis, the following costs and outcomes should be incorporated: all costs directly borne by the publicly funded health system in Canada, and QALYs that accrue to individuals who are vaccinated, individuals who experience externalities related to the vaccination program, as well as QALYs that accrue to informal caregivers. For close relations of the patient (e.g., parents of a sick child), there may be QALY losses due to anxiety or bereavement. As noted in chapter 10 on Measurement and valuation of health, the analysis should not include QALYs that accrue to formal caregivers. This is to avoid double counting the non-monetary valuation of effects (i.e., disutility of formal care burden) and the monetary value of their time (where the wage should already be adjusted to compensate the QALY losses such as burnout and emotional strain of providing care). Costs of formal caregivers are accounted for and valued at the hourly wage rate that would be paid to an individual who performs this service, as described in Chapter 11 on Resource use and costs.

For the societal perspective reference case analysis, costs and outcomes from the publicly funded health system perspective should be included, along with the following, at minimum: patient-borne costs, caregiver costs, and losses in productivity. Non-health impacts such as consumption, social services, education, and environment should also be included when relevant.

Incremental cost-effectiveness ratios (ICERs) should be reported for each reference case perspective. However, depending on the position of scenarios in the cost-effectiveness plane, it may not be sensible to calculate ICERs, such as in the case of dominance of the vaccination strategy or the comparator. In all cases, however, mean values for costs, effectiveness, incremental costs and incremental effectiveness should be reported with 95% confidence or credible intervals. Where there are more than two interventions being considered in the analysis, sequential ICERs should be presented. This approach involves comparing each intervention to the next most costly intervention and excluding all interventions that are either dominated or subject to extended dominance. Graphically, results should be presented as health production functions or cost-effectiveness efficacy frontiers. When net monetary benefits or net health benefits are useful for interpretation, they should be calculated and presented using a range of commonly used cost-effectiveness thresholds.

In cases where subgroup analyses have been conducted (as described in chapter 3 on Study populations), mean values for costs and outcomes as well as ICERs should be generated for each relevant subgroup in accordance with the guidance presented in this chapter. In cases where multiple regional or provincial/ territorial publicly funded health system perspectives have been analyzed, findings should be reported for each one.

13. Uncertainty

Guideline Statements

13.1 Researchers should address parameter uncertainty using a probabilistic reference case analysis, where possible, as well as deterministic sensitivity analyses.
13.2 "Methodological uncertainty should be explored by comparing the reference case results to those from a non-reference case analysis that deviates from the recommended methods in order to examine the impact of methodological differences." [CADTH Guideline Statement]
13.3 Cost-effectiveness acceptability curves (CEACs) and cost-effectiveness acceptability frontiers (CEAFs) should be used to represent the uncertainty in the estimates of costs and outcomes when these estimates have been generated probabilistically. [CADTH Guideline Statement with amendment]
13.4 When the decision problem includes the option of commissioning or conducting future research, value-of-information analysis may be helpful to characterize the value of these options and design future research, and may be included in the reference case analyses. [CADTH Guideline Statement with amendment]
13.5 Scenario analyses should be used to assess structural uncertainty. [CADTH Guideline Statement with amendment]

Decision-makers need information about uncertainty related to the results of economic evaluations of vaccination programs to avoid making suboptimal funding decisions. Specifically, three types of uncertainty should be examined and reported: parameter, structural, and methodological uncertainty.

Parameter uncertainty

Parameter uncertainty, also called second-order uncertainty, refers to uncertainty in parameter estimates that are used to populate a model.^{Footnote 57}^{Footnote 163}^{Footnote 164} This differs from random variability, also called first-order uncertainty or stochastic uncertainty. Most guidelines on conducting economic evaluations of healthcare interventions recommend using probabilistic analysis either in the reference case analysis or as sensitivity analysis to explore parameter uncertainty, but in rare situations, this technique may not be feasible with dynamic models. Such situations arise when models are particularly complex (e.g., agent-based simulations), or when only limited computing power is available. In dynamic models, many parameters related to transmission, such as contact patterns between individuals and prevention-related behaviours, may be correlated and these correlations must be preserved in the models to generate sensible results that fit to existing data (e.g., epidemiologic surveillance data). In some cases, correlations between parameters may be unknown, ^{Footnote 39}^{Footnote 64} although they can sometimes be established using Bayesian parameter inference methods.^{Footnote 165}^{Footnote 166} In these instances, researchers may be required to choose between a complex model structure that does not allow for probabilistic analysis, and a simpler but less realistic structure that allows exploration of the impact of parameter uncertainty.

Where feasible, parameter uncertainty should be addressed probabilistically through probabilistic reference case analyses. Researchers should ensure that results converge if iterative algorithms are used for parameter inference. Parameter ranges chosen to assess uncertainty should, where possible, be based on estimates from observational studies or surveillance data. Results of these analyses should be presented as cost-effectiveness acceptability curves (CEACs) or cost-effectiveness acceptability frontiers (CEAFs). CEACs summarize uncertainty by graphing the probability of an intervention being cost-effective in relation to possible values of cost-effectiveness thresholds. CEAFs summarize the uncertainty of multiple interventions (usually three or more) by plotting the intervention that is economically preferred at different cost-effectiveness thresholds. Scatter plots on the cost-effectiveness plane may be provided alongside CEACs and CEAFs. Scatter plots are useful to observe the density and spread of the iterations, and to assess inflection points and the shape of the ellipses produced.

In addition to quantifying uncertainty probabilistically in the reference cases, researchers should conduct deterministic sensitivity analyses (DSA) on individual model parameters to gain insight into the isolated effects of variations in these parameters that is provided by deterministic methods. Results of one-way (or univariate) DSAs can be presented using a tornado diagram, and two-way DSAs can be presented using two-way threshold graphs.^{Footnote 164} At minimal, there are two key input parameters to examine through DSA. Researchers should conduct a DSA on vaccine price using several plausible values since the actual unit price of vaccine doses in Canada is often confidential. For the public payer, the negotiated price can be lower than the manufacturer's list price. DSAs should also be conducted on estimates of vaccine effectiveness as there is often a high degree of uncertainty in these parameters. Further, researchers should conduct a two-way sensitivity analysis assessing vaccine price and vaccine effectiveness. Other parameters to test for in DSA include estimates where Canadian data are not used.

Researchers should consider conducting threshold analyses on the most uncertain parameters that may not be based upon empirical evidence (e.g., implementation parameters such as vaccine price), so that decision-makers are able to ascertain plausible ranges of parameter values that are likely to result in a cost-effective vaccination program.

When conducting DSAs researchers should identify parameter regions associated with distinct model behaviours such as epidemic spread or extinction of disease, and state whether the sensitivity analysis has been confined to a single region. If the sensitivity analysis spans more than one region, researchers should indicate the probability of achieving different disease equilibrium states as the parameter values vary.^{Footnote 39}

When probabilistic analyses are not undertaken in the context of non-linear dynamic models, researchers should conduct comprehensive DSAs on uncertain parameters. In these situations, researchers may consider using novel DSA methods such as stepwise DSA and distributional DSA.^{Footnote 167}

The effects of parameter uncertainty may be particularly pronounced in dynamic models compared to static models because of non-linearity in these models that can lead to more variable population outcomes in different parameter regions. For example, a small change in parameter values may alter the model state from a disease-free state to a state of endemic equilibrium when the basic reproduction number (R₀) is near a value of 1. These model behaviours have implications for the effectiveness of vaccination programs. If the program is introduced near a threshold state (e.g., beginning of an epidemic), its indirect effects may be substantial compared to a program introduced at disease equilibrium where its effectiveness may exhibit a linear relationship between the number of individuals vaccinated and prevention of the disease of interest.^{Footnote 39}

It can often be difficult to obtain accurate parameter estimates for infectious disease models since researchers frequently must rely on observational studies or surveillance data.^{Footnote 39} Parameter values derived from Public Health surveillance data may be biased because the proportion of cases detected may be low and can vary considerably among different diseases, even for infectious diseases that are reportable as part of Public Health surveillance requirements.^{Footnote 168} Severity of the infectious disease impacts detection. For example, infection with pertussis may be asymptomatic, associated with mild symptoms, or severe coughing or even death.^{Footnote 169} Thus, surveillance systems that rely upon passive reporting often overestimate disease severity, morbidity and mortality, while underestimating the true incidence of infection in the population.^{Footnote 39}^{Footnote 169}

Uncertainty in estimates of vaccine effectiveness may arise from differences between data obtained from randomized controlled trials (RCTs) compared to data obtained from large observational studies. In most RCTs, the force of infection does not change and leads to an underestimate of the true population-based effectiveness of the vaccine because these studies do not account for indirect effects of vaccination (i.e., impact of community immunity). On the other hand, large population-based observational studies of vaccine effectiveness do account for indirect effects, but they are limited by the potential for selection bias and unmeasured confounding.^{Footnote 39}^{Footnote 170} Selection bias may result from systematic differences in sampling of individuals who are vaccinated compared to those who are not, resulting in a distribution of exposures and outcomes that is no longer representative of the source population. Confounding occurs when the association between the vaccination program and outcomes (e.g., hospitalizations averted) is distorted or spuriously produced. Examples of confounding factors include level of access to healthcare services, socioeconomic status, and prevalence of natural immunity.^{Footnote 170} Researchers should note that observational studies of vaccine effectiveness are difficult to conduct, and usually cannot be undertaken prior to a vaccine being licensed. Also, effectiveness of vaccination programs for preventing disease in both vaccinated and unvaccinated individuals at the population-level (community immunity) is dependent upon vaccine coverage and dose completion rates. Uncertainty in these parameters should be accounted for. When considering observational studies of vaccine effectiveness for inclusion in economic evaluations, researchers may consider referring to published guidelines for assessing evidence from comparative effectiveness studies.^{Footnote 171}^{Footnote 172}

Uncertainty in parameters related to transmission of infection between individuals should be reflected in an uncertainty analysis. These parameters include contact patterns between individuals, as well as other behaviours that may influence disease prevention and control. Researchers should account for any differences in these parameters between groups. For example, in diseases where asymptomatic or mildly symptomatic individuals can transmit infection to others, these individuals are less likely to modify their behaviours to reduce transmission compared to individuals whose symptoms are more severe.^{Footnote 39}

In some cases, parameter values are estimated using models, which could be considered sub-models of the primary decision-analytic model.^{Footnote 163} For example, a predictive model may be required to establish the relationship between immune biomarkers that are vaccine correlates of protection (CoPs), and the incidence of clinically apparent disease. In this case, uncertainty in the values related to CoPs as well as uncertainty in the methods used to model the relationship between CoPs and clinical disease need to be accounted for.

When calibration is used to estimate model parameters, uncertainty in the estimates derived from the calibration process should be explored.^{Footnote 164} This is to understand whether calibration methods could have introduced bias in the cost-effectiveness results beyond accounting for variability in the calibrated parameters (e.g., goodness-of-fit statistic, parameter search algorithm). As Taylor et al. have demonstrated in their cost-effectiveness analysis of human papillomavirus (HPV) vaccine, failing to account for uncertainty related to calibrated parameters in the model underestimates the true extent of uncertainty in the cost-effectiveness estimates.^{Footnote 72}

Structural uncertainty

Structural uncertainty relates to the choice of model structure. When constructing models for economic evaluation of vaccination programs, researchers need to ensure that the model structure accounts for relevant factors related to transmission of infection between individuals, including the role of population subgroups that may be at high risk for transmitting or acquiring infection, the natural history of the disease being modeled, as well as the direct and indirect effects of the vaccination program.^{Footnote 39}^{Footnote 57}

Structural uncertainty related to transmission of infection can be related to any of the following factors: 1) mode of transmission; 2) the relationship between severity of symptoms and transmissibility (i.e., whether asymptomatic or minimally symptomatic individuals can transmit infection); 3) mixing and contact patterns of individuals within populations; and 4) behavioural changes in response to disease outbreaks.^{Footnote 39}^{Footnote 173}^{Footnote 174} Researchers should test alternate assumptions related to these factors in all applicable situations to ensure that uncertainty related to transmission has been adequately examined.

For certain infectious diseases, population subgroups may be epidemiologically important to disease transmission, or to risk of infection in the overall population. These subgroups may intersect with equity-relevant subgroups, which are discussed in chapter 14 on Equity. As an example, although anyone can be infected with hepatitis A, men who have sex with men, and people who use injection or non-injection drugs are at higher risk of infection.^{Footnote 175} Model assumptions of the role of high-risk groups with respect to transmission should be tested to understand the degree of uncertainty they contribute.

Uncertainty about the natural history of an infectious disease often centers on whether it leads to latent infection or natural immunity. This is the case with certain high-risk strains of HPV. Decision analytic models, where different assumptions on latency and natural immunity to these HPV strains were employed, demonstrated that results are highly sensitive to these assumptions.^{Footnote 176} Accordingly, researchers should account for such uncertainties in the structure of a model. In this HPV example, both susceptible-infectious-susceptible (SIS) and susceptible-infectious-removed (immune) (SIR) frameworks could be examined to assess how results vary with alternate model structures.^{Footnote 39}

Important aspects of structural uncertainty related to vaccination programs include the timing of vaccine doses, the duration of protection provided by the vaccination, and any indirect effects that may result from the vaccine.^{Footnote 39}^{Footnote 98} When applicable, the use of vaccine booster doses should also be evaluated.

In situations where there is uncertainty about whether the protection provided by a vaccine wanes, researchers should test different plausible assumptions related to duration of protection. These assumptions, where possible, should be based on immunologic evidence on the relationship between immune correlates of long-term protection and occurrence of clinical disease in the post-vaccination period.^{Footnote 98}^{Footnote 177} Epidemiologic data related to disease outbreaks, when available, might also be useful in modelling the duration of protection conferred by vaccines as has been demonstrated with the examples of childhood mumps vaccination,^{Footnote 178} and whole-cell and acellular pertussis vaccination.^{Footnote 20} Examples of some methods used to predict duration of effect include linear functions, logarithmic functions, and exponential functions. Constant functions are used in models that assume no waning of protection.^{Footnote 42}^{Footnote 98}

One of the critical decisions faced by researchers conducting economic evaluations of vaccination programs is whether to model the indirect effects of vaccines using dynamic models. See chapter 8 on Modelling for situations where static models may be acceptable. In cases where there are indirect effects associated with a vaccine such as age-shifting of disease or serotype replacement, the decision to use a static model instead of a dynamic model could be misrepresented. With the use of dynamic models, researchers should consider various scenarios related to changing epidemiology of disease after the introduction of a vaccination program to assess the effects of age-shifting, community immunity, and/or serotype replacement on the results of an economic evaluation.

Decisions on how to address uncertainty related to the different dimensions of vaccine effectiveness (prevention of clinical disease, severity of clinical disease, infection, and infectiousness) may be complicated in cases where these effects are not well understood. For example, it is thought that meningococcal B vaccines do not provide community immunity by preventing transmission of the bacteria between individuals, but data on the true extent of the effectiveness of this vaccine is limited.^{Footnote 179} Researchers should use different model structures to explore this type of uncertainty where relevant.

Structural uncertainty influences results of economic evaluations at least to the same extent as parameter uncertainty, and often to a greater extent.^{Footnote 57}^{Footnote 164} It is particularly important to explore structural uncertainty in dynamic models with uncertainty analysis because of their non-linear effects, which can lead to variable model behaviour.^{Footnote 39} Scenario analysis should be used to explore structural uncertainties in models. This technique involves testing alternate model scenarios that are underpinned by different plausible structural assumptions. Results from each scenario analysis should be presented individually. Model averaging can then be used to summarize results from all alternate model scenarios that were tested. Guidance on averaging results from the scenarios analyses can be found elsewhere, such as using weightings for each model based on the model's predictive ability according to available data (e.g., measures of fit).^{Footnote 180}^{Footnote 181} When weightings cannot be derived from data, researchers' judgment and expert opinion may be required.

Parameterizing sources of uncertainty (i.e., to include additional "uncertain" parameters) is an emerging method for addressing structural uncertainty.^{Footnote 180} It involves adding parameters to a model that are assumed to be the sources of structural uncertainty and which can be specified using distributions, depending on the available prior information.^{Footnote 180} This method allows structural uncertainty to be represented directly in the model, and could also be used to inform any decision about future research to resolve these uncertainties.^{Footnote 164}^{Footnote 180} Justification should be provided for any structural uncertainties that have not been addressed.^{Footnote 164}

Methodological uncertainty

Methodological uncertainty relates to the different methods that can be used to conduct an economic evaluation. When conducting economic evaluations of vaccination programs, important methodological choices that researchers must consider include the type of analysis, perspective, discounting approach and rate, and time horizon.^{Footnote 57}^{Footnote 163}

Because vaccination programs often prevent diseases that could result in catastrophic consequences (e.g., meningococcal B vaccination could prevent death or permanent neurological sequelae resulting from meningitis due to meningococcal type B bacteria), they produce health-related benefits as well as non-health-related benefits such as improvements in education or lifetime productivity. Cost-benefit analyses (CBAs) can account for their full range of impacts, and as such, their use as non-reference case analyses can be compared to the cost-utility analysis (CUA) reference cases to explore the impact of methodological differences.^{Footnote 12} In practice, however, uncertainty related to the type of evaluation conducted is rarely examined.^{Footnote 57} In principle, the non-health costs of vaccination programs could be captured in a CUA if a broader perspective (e.g., societal perspective) for the analysis is adopted, but capturing non-health benefits may be more challenging.^{Footnote 12}^{Footnote 163} Discrete choice experiments are an increasingly popular option for capturing relevant trade-offs for non-health benefits of interventions for either CBA or CUA.

Economic evaluations of vaccination programs are particularly sensitive to the discounting approach and time horizon chosen for the analysis. Costs related to the initiation of a vaccination program are incurred when the program is introduced while the full impacts of the program often take a much longer period of time, sometimes many years or decades, to realize.^{Footnote 13}^{Footnote 57} As such, researchers should examine how varying the discounting approach, as well as time horizon of the analysis affects the results. When the time horizon of the analysis is very long (e.g., multiple decades), results of the economic evaluation should be reported for a range of time horizons to ensure that decision-makers are considering costs and outcomes that are relevant to the decision problem they are addressing.

To thoroughly explore many aspects of methodological uncertainty, multiple different models are ideally required, which is often practically difficult. As such, researchers should transparently collaborate with other groups addressing similar decision problems, whenever possible, so that the extent of uncertainty related to methodological choices can be explored.

Value-of-information

If a value-of-information analysis is undertaken, researchers should summarize the value of additional information using the expected value of perfect parameter information and the population expected value of perfect parameter information. See CADTH Guidelines for further guidance. Researchers should apply a range of commonly used cost-effectiveness thresholds for their value-of-information analysis.

14. Equity

Guideline Statements

14.1 Researchers and decision-makers should work together to establish which equity dimensions and goals should be included in the economic evaluation of the vaccination program considered. Equity dimensions should be considered in the context of NACI's Ethics, Equity, Feasibility, and Acceptability (EEFA) framework.
14.2 Analyses that incorporate relevant equity concerns should accompany the reference case analyses (e.g., distributional cost-effectiveness analysis, extended cost-effectiveness analysis, or other emerging methods), and presented alongside the reference cases.

The traditional emphasis of economic evaluations of healthcare interventions has been the assessment of efficiency. This exercise sits within the larger decision-making framework of health technology assessment (HTA), which synthesizes and appraises primarily clinical and economic evidence related to a new health technology. However, there is growing recognition that ethical and moral questions related to how a technology is appraised and used should be addressed as part of decisions on the adoption of new health technologies.^{Footnote 182}^{Footnote 183}

NACI has established the Ethics, Equity, Feasibility and Acceptability (EEFA) Framework to systematically consider these factors as part of a multi-criteria approach to vaccine recommendations. In this framework, ethics and equity are considered with the feasibility and acceptability of a recommendation, alongside a vaccine's clinical effectiveness, immunogenicity, safety, and cost-effectiveness.^{Footnote 6} Public health ethics is the domain of applied ethics relevant to vaccination. It is primarily concerned with the following core ethical dimensions: 1) respect for persons and communities; 2) non-maleficence and beneficence; 3) trust; and 4) justice.^{Footnote 184}

Equity is considered within the core ethical dimension of justice, and is defined as "the absence of avoidable, unfair, or remedial differences among groups of people, whether those groups are defined socially, economically, demographically, geographically, or by other [means of stratification]."^{Footnote 6}^{Footnote 185} Equity in economic evaluations is an approach to distributive justice that concerns judgments about the fairness in distribution of health outcomes and experiences in a population, and it relates to the fair allocation of resources and achievement of health improvements between individuals or groups.^{Footnote 186} There has been substantial recent activity and methodological development related to considering equity in the economic evaluation of health technologies.^{Footnote 183}^{Footnote 187}^{Footnote 188}^{Footnote 189}

The distributional consequences related to the adoption of a new health technology is particularly important in situations where decision-makers must make trade-offs between attributes of health technologies, including trade-offs between cost-effectiveness and health equity. These situations arise when a technology is cost-effective but increases inequity between groups in a population because some segments of society may benefit from the technology more than others. Alternatively, they arise when a technology is not cost-effective but its adoption would improve equity between groups by reducing disparities in health gains, or when a technology would increase equity between some groups (e.g., income strata) but decrease equity between others (e.g., geographical).^{Footnote 190}

Qualitative research in the Canadian Public Health context suggests that decision-makers perceive health equity and social justice to be different although both are situated in the theoretical space of justice.^{Footnote 191} Health equity is described as having a neutral or objective quality and focuses on proximal issues such as access to Public Health resources and services. Hence, the concept is described as "comfortable" and "quantifiable". In contrast, social justice is perceived as being subject to political sensitivities, requiring a personal confrontation of biases and privilege, and being "structural" and hence "not actionable" given a lack of institutional mechanisms or appetite. Perceptions of health equity and social justice are especially relevant in discussions of fairness with respect to transfer costs. Where goods and services are not related to the consumption or production of resources and services, transfer costs are functionally monetary transfers from one person or group of people in society to another. They apply in both the publicly funded health system perspective and societal perspective. See chapter 11 on Resource use and costs for the example on vaccine price and its cost components. Given the publicly funded nature of the Canadian health system, there are questions of fairness and sustainability of vaccine pricing when public funds are being transferred to the private sector such as multinational corporations.^{Footnote 192} Healthcare payers, as well as the public, are interested in ways to reduce marginal production costs and to ensure that the attribution of profits is aligned with value creation. In other words, there is interest in achieving fairness to both buyers and sellers. Profit or perceptions of profit may influence decision-making on reimbursement and procurement, and may relate to equity goals of the vaccination program. As such, researchers should consider the pathway of monetary transfers within society and the parties involved including private and public sectors, where known.

Researchers and decision-makers should work collaboratively to establish which of the following equity goals the vaccination program is aiming to address: 1) improving equity in access to the vaccination program for eligible individuals; 2) improving equity in uptake of the vaccination program in eligible individuals; 3) improving equity in health benefit related to health conditions addressed by the vaccination program; 4) reducing lifetime health inequities between groups by means of the vaccination program; or 5) reducing overall (i.e., health and non-health-related) inequities between groups by means of the vaccination program. Different approaches may be used to conceptualize equity goals (e.g., proportionate universalism, egalitarianism). Researchers may find it helpful to refer to published literature on this topic when conceptualizing equity objectives for inclusion in an economic evaluation.^{Footnote 193}^{Footnote 194}^{Footnote 195}

When establishing equity goals researchers should consider whether there are key groups of individuals experiencing health inequities and barriers to health that could be reduced or addressed by the vaccination program. Examples of groups that may experience health inequity in Canada include Indigenous Peoples (i.e., First Nations, Inuit, and Métis Peoples for the purposes of these Guidelines), individuals of low socioeconomic status, people who are part of ethnic, sexual, or gender minority groups, populations living in certain geographic locations (urban vs. rural vs. remote and isolated), individuals with disabilities, and high-risk groups such as children, seniors or institutionalized persons.^{Footnote 6}^{Footnote 130}^{Footnote 189}^{Footnote 196}^{Footnote 197}

Researchers should also consider factors that could lead to differences in health benefits resulting from the vaccination program between groups experiencing health inequities. Factors include underlying health conditions, potential for lifetime benefit, health-seeking behaviours, uptake of the vaccine and the role of community immunity in reducing or increasing inequities between groups, risk-taking behaviours, different mixing or contact patterns within groups, and access to culturally safe healthcare.^{Footnote 10}^{Footnote 198}

Researchers should be aware that some groups may benefit disproportionally from the vaccination program, thus potentially increasing inequities. For example, differential access to a human papillomavirus (HPV) vaccination program can worsen inequity by reducing the rate of cervical cancer in a population who was already at lower risk, thereby increasing the difference in outcomes between groups. Interventions that appear to reduce inequities should be examined to investigate how relevant barriers to access faced by the population would be overcome.

Once researchers have established the equity-relevant outcomes of interest, features of the vaccination program intended to achieve these outcomes should be considered. For instance, if the goal of the program is to improve equity in access to the vaccine for all eligible individuals, then a program that decreases barriers to access should be considered. An example of such a program would be a school-based HPV vaccination program that eliminates barriers for individuals such as the need for transportation to a clinic or physician's office.^{Footnote 130} If the goal of the program, however, is to improve equity in uptake of the vaccine, researchers could consider scenarios in which vaccines are mandatory or that address misinformation about the vaccine. An example of such a program would be a legally mandated school-based program for HPV vaccination, with a provision for active opt-out.^{Footnote 199} If the goal is to reduce lifetime health inequity between groups with the vaccination program, a program that is consistent with the principle of vertical equity, which entails treating individuals with different ethically-relevant characteristics differently, should be considered.^{Footnote 200} An example would be a vaccination program that achieves high levels of vaccination coverage among Indigenous Peoples. Indigenous Peoples experience a greater burden of vaccine-preventable diseases than non-Indigenous People in Canada (e.g., cervical cancer, hepatitis A) due to systematic inequities such as poverty, crowded housing conditions, lack of running water, and poor underlying health status, which increase the risk for acquiring these infections.^{Footnote 201}^{Footnote 202} Additionally, Indigenous Peoples living on reserves and in remote communities may also experience inequities in access to treatment when they become ill, increasing their risk of infection-related morbidity and mortality.^{Footnote 203} Researchers must be aware, however, that vaccination programs restricted to certain groups that are at high risk or marginalized may further stigmatize those groups. Alternative approaches, such as universal programs, should be considered. Finally, if the goal is to improve health and non-health equity between groups, researchers could consider vaccination programs that contribute towards improving health as well as economic productivity. Examples of such programs are childhood vaccination programs, which enable children to participate in education, in turn enabling them to become healthy and economically productive adults.^{Footnote 204} When consideration of equity-relevant outcomes relates to selection and definition of comparator(s) to be included in the analysis, researchers should refer to chapter 3 on Comparators of these guidelines.

When presenting results of economic evaluations by equity-relevant subgroups, researchers should ensure that the criteria for establishing these subgroups have been transparently delineated and justified. A recent review of equity-informative cost-effectiveness analyses (CEAs) identified eleven different criteria that have been used to explicitly incorporate equity in a cost-effectiveness framework, with socioeconomic status and race/ethnicity used most frequently.^{Footnote 205} Distributed CEA and extended CEA frameworks provide guidance and methods for conducting equity-informative CEAs.^{Footnote 187}^{Footnote 206}

In addition to considering equity-related outcomes associated with vaccination programs, researchers should also consider the distribution of opportunity costs related to the implementation of these programs.^{Footnote 190} This redistribution of resources could, for example, result in decreased expenditures on screening programs or non-vaccine preventive measures related to the infection being targeted by vaccination program. Opportunity costs could also fall outside of the health sector, for example, through decreased funding of educational or social programs.^{Footnote 190} Although in many cases, it may be difficult to explicitly identify opportunity costs related to implementing vaccination programs, where possible, researchers should quantify opportunity costs in a manner that is relevant to decision-makers. In some cases, interventions to improve equity may not carry a net opportunity cost, since it may be efficient to allocate resources to groups with higher health burden.

When relevant, researchers should consider the implications of vaccination programs on intergenerational equity. Vaccination programs that result in externalities have effects on cohorts of individuals other than the cohort that is vaccinated.^{Footnote 45} For example, a childhood varicella vaccination program may result in increased cases of herpes zoster in older adults;^{Footnote 207} conversely, an HPV vaccination program may lead to disease eradication for future generations.^{Footnote 208} In both examples, the indirect effects on cohorts of individuals not intended for the vaccination program should be accounted for with dynamic models used to generate estimates of cost-effectiveness. Researchers should then explicitly consider the equity implications of these results. In the first example, researchers need to qualitatively identify the trade-offs between improved child health and negative health outcomes that may be experienced by older individuals. Quantitatively, researchers should report the summary costs and outcomes estimated in the analysis which capture the trade-offs. In the second example, researchers need to consider how health outcomes that accrue to cohorts far into the future should be valued in present day terms. At present, there is ongoing debate on whose future lives should be taken into account, as well as how they should be taken into account, especially in the case of disease eradication programs or programs with large intergenerational effects (e.g., programs that impact antimicrobial resistance or fertility).^{Footnote 209} Future lives refer to future people who do not yet exist, and may be 1) future generations that will be born regardless of the program's existence; or 2) future generations that will be born due to the intervention's impact on increased fertility, or that may live due to its impact on reduced mortality.

Such valuation of outcomes in cohorts far into the future is contingent on the discounting strategy employed in the economic evaluation. An equal discount rate results in greater value placed on health outcomes for the present cohort and cohorts close in time to the present, while lesser value is placed on health outcomes for cohorts in the distant future, which some authors argue is an unfair feature of this strategy.^{Footnote 44} Use of lower discount rates, however, could result in giving greater weight to health outcomes in cohorts that are likely to have greater income, and access to more health interventions, and as such, more potential to improve health. Given the variable outcomes related to intergenerational effects of vaccination programs that can result from different discounting strategies, researchers should consider and report the intergenerational equity implications of vaccination programs that result in health benefits for cohorts in the distant future.^{Footnote 44}^{Footnote 45}^{Footnote 210}

15. Reporting

Guideline Statements

15.1 "The economic evaluation should be reported in a transparent and detailed manner with enough information to enable the reader or user (e.g., decision-maker) to critically assess the evaluation." [CADTH Guideline Statement with amendment]
15.2 "A summary of the evaluation written in non-technical language should be included." [CADTH Guideline Statement]
15.3 "Results of the economic evaluation should be presented in graphical or visual form, in addition to tabular presentation." [CADTH Guideline Statement]
15.4 "Details and/ or documents describing quality assurance processes and results for the economic evaluation should be provided. An electronic copy of the model should be made available for review with accompanying documentation in adequate detail to facilitate understanding of the model, what it does, and how it works." [CADTH Guideline Statement]
15.5 "Funding and reporting relationships for the evaluation should be described, and any conflicts of interest disclosed." [CADTH Guideline Statement]
15.6 Researchers should use NACI's Guidelines for Reporting Economic Evaluations of Vaccination Programs in Canada (supplemental document) and complete the Impact inventory table for economic evaluations of vaccination strategies (Appendix 1).

The reporting of findings of economic evaluations should provide decision-makers with transparent and credible information that enables them to address the decision problem of interest and make an optimal funding decision related to the vaccination program being considered.

Vaccination-specific reporting considerations include the time horizon of the evaluation, and the mechanisms through which vaccines exert their effects. In cases where the model time horizon of an economic evaluation spans a long period of time, results from various time points over the model time horizon should be reported to ensure that findings of the analysis are relevant to the time horizon being considered by decision-makers. Since vaccines may exert their effects through various mechanisms (e.g., preventing transmission of infection, preventing infection, preventing disease or decreasing its severity), researchers should report outcomes of vaccination programs not only in terms of quality-adjusted life-years (QALYs), but also in terms of the number of cases prevented, the number of relevant healthcare utilization units (e.g., hospitalizations) averted, the number of deaths averted, and the number of individuals needed to vaccinate, where applicable. Reporting these metrics in addition to QALYs and costs increases the credibility and transparency of the analysis for decision-makers.

Resource use, costs, and outcomes should be reported in a disaggregated fashion for each comparator considered in the analysis and for both reference case analyses. If analyses are conducted from multiple public payer perspectives (e.g., for multiple provinces/ territories), each should be reported separately.

All assumptions and decision rules used in the analyses should be transparently reported. Researchers should provide details related to quality assurance processes, and results undertaken as part of the model verification process. A fully executable electronic copy of the model should be made available along with details related to the model's functionality to enable the decision-maker to verify results of the analysis or conduct additional analyses if required.

Researchers should disclose all sources of funding for the economic evaluation and state the role of the funder(s) in the identification, design, conduct, and reporting of the analysis. Non-monetary (e.g., in-kind) sources of support should also be disclosed.^{Footnote 211}

Researchers should disclose all potential conflicts of interest, both financial and non-financial. Types of affiliations and interests to disclose include: participation in research, equity ownership, intellectual property, and any other interest that readers may perceive as a competing interest.^{Footnote 7}^{Footnote 212}

The "NACI Guidelines for Reporting Economic Evaluations of Vaccination Programs in Canada" on the NACI website provides a standard format for reporting the results of economic evaluations of vaccination programs.^{Footnote 7} The template provided aligns with existing reporting guidelines including the Consolidated Health Economic Evaluation Reporting Standards II (CHEERS II).^{Footnote 211} Researchers should follow the structure outlined in this document when presenting their results.

Appendix 1: Impact inventory table

The table was adapted from the impact inventory published by the 2^nd Panel on Cost-Effectiveness^{Footnote 1} to further include broader impacts not previously captured. Many of these impacts relate to health technologies in general, and some are specific to vaccines or infectious diseases (e.g., age-shifting of disease, disease eradication).^{Footnote 1}^{Footnote 2}^{Footnote 3}^{Footnote 4}^{Footnote 5} The table provides a list of health and non-health impacts that could result from vaccination programs. The intent is to allow researchers to systematically consider all impacts when planning and conducting economic evaluations of vaccination programs. Specific guidance on quantifying these impacts and their associated costs is found in chapter 11 on Resource use and costs.

**Table 1: Impact inventory table for economic evaluations of vaccination strategies**
Area of impact	Definitions/Examples	Included in reference case^{Footnote b}?		Comments
Area of impact	Definitions/Examples	Publicly funded health system perspective	Societal perspective	Comments
Health
Health outcomes	Individual health outcomes for persons intended for vaccination
	Mortality	Y/N	Y/N	no data
	Health-related quality of life	Y/N	Y/N
	Safety (i.e., adverse events)	Y/N	Y/N
	Health impacts not captured by QALYs	Y/N	Y/N
	Individual health outcomes for informal caregivers
	Health-related quality of life	Y/N	Y/N	no data
	Population health outcomes
	Incidence of infection and disease in vaccinated and unvaccinated individuals	Y/N	Y/N	no data
	Changes in age distribution of individuals who develop infection and disease	Y/N	Y/N
	Emergence of new diseases related to variations of the pathogen (i.e., serotypes, serogroups, strains) or unrelated pathogens that may replace the one(s) targeted by the vaccine	Y/N	Y/N
	Disease eradication	Y/N	Y/N
Publicly funded health system costs	Healthcare costs
	Publicly funded healthcare services (e.g., physician visits, diagnostic tests, drug treatment where applicable, hospitalization, formal caregiving,^{Footnote a} rehabilitation in ^{Footnote a} facility or at home,^{Footnote a} home care,^{Footnote a} long-term care in nursing homes^{Footnote a})	Y/N	Y/N	no data
	Future related and unrelated healthcare costs	Y/N	Y/N	no data
	Public Health costs
	Program-related costs (e.g., implementation, delivery and recurrent costs including Public Health campaigns and health promotion activities; transaction costs related to introduction of new vaccines or switching between vaccines; costs related to screening, diagnosis, and treatment of disease; epidemiological surveillance, contact tracing, investigation and management of outbreaks)	Y/N	Y/N	no data
	Intervention-related costs (e.g., cost of vaccine doses, distribution such as transportation and cold storage, administration including personnel, wastage and ancillary supplies)	Y/N	Y/N	no data
Healthcare costs NOT funded by the health system	Prescription medications (in some cases)	N/A	Y/N	no data
	Formal caregiver services,a rehabilitation in a facility or at home,a home care,a long-term care in nursing homesa (in some cases)	N/A	Y/N
	Miscellaneous out-of-pocket costs (e.g., non-prescription medications)	N/A	Y/N
	Ancillary costs (e.g., private insurance copayments, dental care, vision care, assistive devices, physiotherapy, etc.)	N/A	Y/N
Non-health
Direct out-of-pocket costs	Transportation costs	N/A	Y/N	no data
Direct out-of-pocket costs	Accommodation costs	N/A	Y/N	no data
Losses in productivity	Paid work
	Time off work resulting from vaccine administration, treatment, illness, disability, or death	N/A	Y/N	no data
	Presenteeism	N/A	Y/N
	Lifetime productivity consequences of childhood disease	N/A	Y/N
	Unpaid work
	Time off work in informal labour market (e.g., volunteering, helping, mentoring) resulting from vaccine administration, treatment, illness, disability, or death	N/A	Y/N	no data
	Uncompensated household production (e.g., cooking, cleaning, shopping, raising children, other tasks related to household management)	N/A	Y/N	no data
	Informal caregiver productivity
	Time off work resulting from caring for sick individuals, accompanying individuals to vaccine appointments	N/A	Y/N	no data
	Caregiver presenteeism	N/A	Y/N	no data
	Macroeconomic consequences
	Labour supply shocks, widespread business closures	N/A	Y/N	no data
Consumption	Future individual non-medical consumption	N/A	Y/N
	Changes in household consumption	N/A	Y/N
	Health impacts of consumption (e.g., associated with job loss)	N/A	Y/N
Education	Level of educational achievement as a result of physical health, mental health, and cognition	N/A	Y/N	no data
	Costs of special education needs as a result of illness/disability	N/A	Y/N
	Disruptions to learning outcomes (e.g., as a result of school-based vaccine delivery, pediatric disease and disability, or death/disability of a close family member)	N/A	Y/N
Social services and community services	Social services and community services (e.g., disability support, programs to improve access to vaccination programs for adults)	N/A	Y/N	no data
Social services and community services	Child and Youth Services (e.g., awareness programs, family respite, programs to improve access to vaccination programs for children and youth)	N/A	Y/N	no data
Environment	Environmental impact of vaccination programs and comparators from manufacturing, distribution, and implementation (e.g., antibiotic use)	N/A	Y/N	no data
	Food and non-food waste	N/A	Y/N
	Carbon consumption	N/A	Y/N
Other areas	Consider areas such as housing when applicable	N/A	Y/N
Footnote 1 Some of these costs may or may not be incurred by the publicly funded health system, depending on the precise nature of these costs and the relevant jurisdiction Abbreviation: QALY, quality-adjusted life-year Return to footnote a referrer Footnote 2 Y indicates impact included. N indicates impact excluded Return to footnote b referrer

Appendix 2: Reference case specifications

Table 2 presents recommendations for the reference case analyses. In situations where the analyses do not follow the recommendations presented below, researchers should identify any deviations and provide justification based on the decision problem.

**Table 2: Recommendations for reference case analyses**
Section	Guidance
Decision problem	Specify the details of 1) the interventions to be compared; 2) the setting(s) in which they are to be compared; 3) the type of evaluation being conducted; 4) the perspectives from which the analysis is being carried out; 5) the costs and outcomes to be quantified in the analysis; 6) the time horizon over which the analysis is to be carried out; 7) and the population(s) for the evaluation.
Types of evaluations	Conduct a cost-utility analysis (CUA) capturing health outcomes in terms of quality-adjusted life-years (QALYs).
Study populations	Identify the population(s) in which the vaccination program will be used, and, when applicable, any populations that might experience externalities resulting from the vaccination program. Stratify analyses and report by subgroups when heterogeneities between groups of individuals may affect the results of the economic evaluation.
Comparators	Compare all relevant interventions, including other vaccination programs, screening interventions, medical and non-medical preventive interventions, and treatment-based approaches presently used in a Canadian context.
Perspective	Conduct two reference case analyses, one from the publicly funded health system perspective and one from the societal perspective.
Time horizon	Select a time horizon that is long enough to capture all relevant differences in the future costs and outcomes associated with the interventions being compared.
Discounting	Discount costs and outcomes at a rate of 1.5% per year.
Measurement and valuation of health	Identify, measure, and value all relevant health outcomes based on the perspectives of the publicly funded health system and society. Use health preferences that reflect the general Canadian population. Obtain health preferences from an indirect method of measurement that is based on a generic classification system.
Resource use and costs	Identify, measure, and value all relevant resources and costs based on the perspective of the i) publicly funded health system, and ii) society. Estimate Canadian resources and costs using data that reflect the jurisdiction(s) of interest.
Analysis	Derive expected values of costs and outcomes for both the publicly funded health system perspective and the societal perspective for each intervention through probabilistic analysis, incorporating potential correlation among parameters, whenever possible. Where distinct subgroups are identified within the study population, stratify analyses and report by subgroups. Calculate incremental costs, incremental effectiveness, and incremental cost-effectiveness ratios (ICERs) for both the publicly funded health system and societal perspective analyses. For evaluations with more than two comparators, calculate ICERs sequentially.
Uncertainty	Address methodological uncertainty by comparing the reference case results to those from a non-reference case analysis. Summarize decision uncertainty, using cost-effectiveness acceptability curves (CEACs) and cost-effectiveness acceptability frontiers (CEAFs), where possible. Use scenario analysis to address structural uncertainty. If a value-of-information analysis is undertaken, summarize the value of additional information using the expected value of perfect parameter information and the population expected value of perfect parameter information.
Equity	Consider whether there are inequities experienced by specific groups that could be improved by the vaccination program. Equity should be explored using methods such as distributional cost-effectiveness analysis and extended cost-effectiveness analysis. Any additional analyses should accompany the references case analyses when applicable.

References for foreword

Footnote 1

Erickson L.J., De Wals P., Farand L. An analytical framework for immunization programs in Canada. Vaccine. 2005; 23(19): 2470-6. doi:10.1016/j.vaccine.2004.10.029

Return to footnote a referrer

Footnote 2

Kristensen F.B., Lampe K., Wild C., Cerbo M., Goettsch W., Becla L. The HTA Core Model(®)-10 Years of Developing an International Framework to Share Multidimensional Value Assessment. Value Health. 2017; 20(2): 244-50. doi:10.1016/j.jval.2016.12.010

Return to footnote b referrer

Footnote 3

First Nations Health Authority. First Nations perspective on health and wellness. n.d. Available from: https://www.fnha.ca/wellness/wellness-for-first-nations/first-nations-perspective-on-health-and-wellness.

Return to footnote c referrer

Footnote 4

Gallagher J. Indigenous approaches to health and wellness leadership: A British Columbia First Nations perspective. Healthcare Management Forum. 2019; 32(1): 5-10. doi:10.1177/0840470418788090

Return to footnote d referrer

Footnote 5

Government of Canada. Indigenous health care in Canada. 2021. Available from: https://www.sac-isc.gc.ca/eng/1626810177053/1626810219482.

Return to footnote e referrer

Footnote 6

Landscapes of First Nations, Inuit, and Metis Health: An Envrionemntal Scan of Organizations, Literature and Research, 3rd Edition 2014. Available from: https://www.nccih.ca/495/Landscapes_of_First_Nations,_Inuit,_and_M%C3%A9tis_Health__An_Environmental
_Scan_of_Organizations,_Literature_and_Research,_3rd_Edition.nccih?id=134

Return to footnote f referrer

Footnote 7

Greenwood M. Determinants of health and Indigenous Peoples. 2017. Available from: https://cps.ca/uploads/imich/Plenary_Friday_PM_SDOH_Greenwood.pdf.

Return to footnote g referrer

Footnote 8

Benkhalti M., Espinoza M., Cookson R., Welch V., Tugwell P., Dagenais P. Development of a checklist to guide equity considerations in health technology assessment. Int J Technol Assess Health Care. 2021; 37:e17. doi: 10.1017/S0266462320002275

Return to footnote h referrer

Footnote 9

Ismail S.J., Hardy K., Tunis M.C., Young K., Sicard N., Quach C. A framework for the systematic consideration of ethics, equity, feasibility, and acceptability in vaccine program recommendations. Vaccine. 2020; 38(36): 5861-76. doi:10.1016/j.vaccine.2020.05.051

Return to footnote i referrer

Footnote 10

Tagalik S. Inuit Qaujimajatuqangit: the role of Indigenous knowledge in supporting wellness in INuit communities in Nunavut. National Collaborating Centre for Indigenous Health (NCCIH), 2012.

Return to footnote j referrer

Footnote 11

Feeny D., Torrance G., Furlong W. Health utilities index. Quality of Life and Pharmacoeconomics in Clinical Trials. 2nd ed: Lippincourt-Raven Publishers; 1996. p. 239-52.

Return to footnote k referrer

Footnote 12

Kopec J.A., Williams J.I., To T., Austin P.C. Cross-cultural comparisons of health status in Canada using the Health Utilities Index. Ethn Health. 2001; 6(1): 41-50. doi:10.1080/13557850125061

Return to footnote l referrer

Footnote 13

Mittmann N., Trakas K., Risebrough N., Liu B.A. Utility scores for chronic conditions in a community-dwelling population. Pharmacoeconomics. 1999; 15(4): 369-76. doi:10.2165/00019053-199915040-00004

Return to footnote m referrer

Footnote 14

Torrance G.W., Furlong W., Feeny D., Boyle M. Multi-attribute preference functions. Health Utilities Index. Pharmacoeconomics. 1995; 7(6): 503-20. doi:10.2165/00019053-199507060-00005

Return to footnote n referrer

Footnote 15

World Health Organization (WHO). Basic Documents: 49th Edition. 2020.

Return to footnote o referrer

Footnote 16

Bartlett C., Marshall M., Marshall A. Two-Eyed Seeing and other lessons learned within a co-learning journey of bringing together indigenous and mainstream knowledges and ways of knowing. J Environ Sci Stud. 2012; 2(4): 331-40. doi:10.1007/s13412-012-0086-8

Return to footnote p referrer

References

Footnote 1

CADTH. Guidelines for the Economic Evaluation of Health Technologies: Canada 4^th edition. 2017.

Return to footnote 1 referrer

Footnote 2

Baron J. A theory of social decisions. J Theory Soc Behav. 1995; 25.

Return to footnote 2 referrer

Footnote 3

Claxton K., Paulden M., Gravelle H., Brouwer W., Culyer A.J. Discounting and decision making in the economic evaluation of health-care technologies. Health Econ. 2011; 20(1): 2-15. doi:10.1002/hec.1612

Return to footnote 3 referrer

Footnote 4

Paulden M., Galvanni V., Ghakraborty S., Kudinga B., McCabe C. Discounting and the evaluation of health care programs. n.d.

Return to footnote 4 referrer

Footnote 5

World Health Organization (WHO). Basic Documents: 49th Edition. 2020.

Return to footnote 5 referrer

Footnote 6

Return to footnote 6 referrer

Footnote 7

National Advisory Committee on Immunizations (NACI). Process for Incorporating Economic Evidence into Federal Vaccine Recommendations. Public Health Agency of Canada; 2022.

Return to footnote 7 referrer

Footnote 8

Jit M., Hutubessy R., Png M.E., Sundaram N., Audimulam J., Salim S., et al. The broader economic impact of vaccination: reviewing and appraising the strength of evidence. BMC Med. 2015; 13: 209. doi:10.1186/s12916-015-0446-9

Return to footnote 8 referrer

Footnote 9

Gessner B.D., Kaslow D., Louis J., Neuzil K., O'Brien K.L., Picot V., et al. Estimating the full public health value of vaccination. Vaccine. 2017; 35(46): 6255-63. doi:https://doi.org/10.1016/j.vaccine.2017.09.048

Return to footnote 9 referrer

Footnote 10

Brisson M., Drolet M., Malagón T. Inequalities in Human Papillomavirus (HPV)-associated cancers: implications for the success of HPV vaccination. J Natl Cancer Inst. 2013; 105(3): 158-61. doi:10.1093/jnci/djs638

Return to footnote 10 referrer

Footnote 11

Bloom D.E., Brenzel L., Cadarette D., Sullivan J. Moving beyond traditional valuation of vaccination: Needs and opportunities. Vaccine. 2017; 35 Suppl 1: A29-a35. doi:10.1016/j.vaccine.2016.12.001

Return to footnote 11 referrer

Footnote 12

Park M., Jit M., Wu J.T. Cost-benefit analysis of vaccination: a comparative analysis of eight approaches for valuing changes to mortality and morbidity risks. BMC Med. 2018; 16(1): 139. doi:10.1186/s12916-018-1130-7

Return to footnote 12 referrer

Footnote 13

Walker D.G., Hutubessy R., Beutels P. World Health Organization (WHO) Guide for standardisation of economic evaluations of immunization programmes. Vaccine. 2019; 28(11): 2356-9. doi:10.1016/j.vaccine.2009.06.035

Return to footnote 13 referrer

Footnote 14

Robinson L., Hammitt J., Cecchini M., Chalkidou K., Claxton K., Cropper M., et al. Reference Case Guidelines for Benefit-Cost Analysis in Global Health and Development. 2019.

Return to footnote 14 referrer

Footnote 15

HM Treasury. The Green Book: Central Government Guidance on Appraisal and Evaluation. 2022.

Return to footnote 15 referrer

Footnote 16

US Department of Health and Human Services. Guidelines for Regulatory Impact Analysis. 2016.

Return to footnote 16 referrer

Footnote 17

US Environmental Protection Agency. Guidelines for Preparing Economic Analyses. 2010.

Return to footnote 17 referrer

Footnote 18

Brisson M., Edmunds W.J. The cost-effectiveness of varicella vaccination in Canada. Vaccine. 2002; 20(7-8): 1113-25. doi:10.1016/s0264-410x(01)00437-6

Return to footnote 18 referrer

Footnote 19

Christensen H., Hickman M., Edmunds W.J., Trotter C.L. Introducing vaccination against serogroup B meningococcal disease: an economic and mathematical modelling study of potential impact. Vaccine. 2013; 31(23): 2638-46. doi:10.1016/j.vaccine.2013.03.034

Return to footnote 19 referrer

Footnote 20

Gambhir M., Clark T.A., Cauchemez S., Tartof S.Y., Swerdlow D.L., Ferguson N.M. A change in vaccine efficacy and duration of protection explains recent rises in pertussis incidence in the United States. PLoS Comput Biol. 2015; 11(4): e1004138. doi:10.1371/journal.pcbi.1004138

Return to footnote 20 referrer

Footnote 21

Markowitz L.E., Drolet M., Perez N., Jit M., Brisson M. Human papillomavirus vaccine effectiveness by number of doses: Systematic review of data from national immunization programs. Vaccine. 2018; 36(32 Pt A): 4806-15. doi:10.1016/j.vaccine.2018.01.057

Return to footnote 21 referrer

Footnote 22

Public Health Agency of Canada (PHAC). Chapter 2: The Chief Public Health Officer's Report on the State of Public Health in Canada 2008 - What is Public Health? 2008.

Return to footnote 22 referrer

Footnote 23

Bärnighausen T., Bloom D.E., Cafiero-Fonseca E.T., O'Brien J.C. Valuing vaccination. Proc Natl Acad Sci U S A. 2014; 111(34): 12313-9. doi:10.1073/pnas.1400475111

Return to footnote 23 referrer

Footnote 24

Chandran A., Herbert H., Misurski D., Santosham M. Long-term sequelae of childhood bacterial meningitis: an underappreciated problem. Pediatr Infect Dis J. 2011; 30(1): 3-6. doi:10.1097/INF.0b013e3181ef25f7

Return to footnote 24 referrer

Footnote 25

Grimwood K., Anderson P., Anderson V., Tan L., Nolan T. Twelve year outcomes following bacterial meningitis: further evidence for persisting effects. Arch Dis Child. 2000; 83(2): 111-6. doi:10.1136/adc.83.2.111

Return to footnote 25 referrer

Footnote 26

Soriano-Gabarró M., Mrukowicz J., Vesikari T., Verstraeten T. Burden of rotavirus disease in European Union countries. Pediatr Infect Dis J. 2006; 25(1 Suppl): S7-s11. doi:10.1097/01.inf.0000197622.98559.01

Return to footnote 26 referrer

Footnote 27

Standaert B., Van de Mieroop E., Nelen V. Exploring the potential impact of rotavirus vaccination on work absenteeism among female administrative personnel of the City of Antwerp through a retrospective database analysis. BMJ Open. 2015; 5(6): e007453. doi:10.1136/bmjopen-2014-007453

Return to footnote 27 referrer

Footnote 28

Kim D.D., Neumann P.J. Analyzing the Cost Effectiveness of Policy Responses for COVID-19: The Importance of Capturing Social Consequences. Med Decis Making. 2020; 40(3): 251-3. doi:10.1177/0272989x20922987

Return to footnote 28 referrer

Footnote 29

Folegatti P.M., Ewer K.J., Aley P.K., Angus B., Becker S., Belij-Rammerstorfer S., et al. Safety and immunogenicity of the ChAdOx1 nCoV-19 vaccine against SARS-CoV-2: a preliminary report of a phase 1/2, single-blind, randomised controlled trial. Lancet. 2020; 396(10249): 467-78. doi:10.1016/s0140-6736(20)31604-4

Return to footnote 29 referrer

Footnote 30

Zhu F.-C., Guan X.-H., Li Y.-H., Huang J.-Y., Jiang T., Hou L.-H., et al. Immunogenicity and safety of a recombinant adenovirus type-5-vectored COVID-19 vaccine in healthy adults aged 18 years or older: a randomised, double-blind, placebo-controlled, phase 2 trial. The Lancet. 2020; 396(10249): 479-88. doi:10.1016/S0140-6736(20)31605-6

Return to footnote 30 referrer

Footnote 31

Sanders G.D., Neumann P.J., Basu A., Brock D.W., Feeny D., Krahn M., et al. Recommendations for Conduct, Methodological Practices, and Reporting of Cost-effectiveness Analyses: Second Panel on Cost-Effectiveness in Health and Medicine. JAMA. 2016; 316(10): 1093-103. doi:10.1001/jama.2016.12195

Return to footnote 31 referrer

Footnote 32

Bärnighausen T., Bloom D.E., Canning D., O'Brien J. Accounting for the full benefits of childhood vaccination in South Africa. S Afr Med J. 2008; 98(11): 842, 4-6.

Return to footnote 32 referrer

Footnote 33

Bloom B.S., Bruno D.J., Maman D.Y., Jayadevappa R. Usefulness of US cost-of-illness studies in healthcare decision making. Pharmacoeconomics. 2001; 19(2): 207-13. doi:10.2165/00019053-200119020-00007

Return to footnote 33 referrer

Footnote 34

Beck E., Biundo E., Devlin N., Doherty T.M., Garcia-Ruiz A.J., Postma M., et al. Capturing the value of vaccination within health technology assessment and health economics: Literature review and novel conceptual framework. Vaccine. 2022; 40(30): 4008-16. doi:10.1016/j.vaccine.2022.04.050

Return to footnote 34 referrer

Footnote 35

Postma M., Biundo E., Chicoye A., Devlin N., Mark Doherty T., Garcia-Ruiz A.J., et al. Capturing the value of vaccination within health technology assessment and health economics: Country analysis and priority value concepts. Vaccine. 2022; 40(30): 3999-4007. doi:https://doi.org/10.1016/j.vaccine.2022.04.026

Return to footnote 35 referrer

Footnote 36

Brassel S., Neri M., O'Neill P., Steuten L. Realising the Broader Value of Vaccines in the UK. OHE Consulting Report. 2020.

Return to footnote 36 referrer

Footnote 37

Kim S.Y., Goldie S.J. Cost-effectiveness analyses of vaccination programmes : a focused review of modelling approaches. Pharmacoeconomics. 2008; 26(3): 191-215. doi:10.2165/00019053-200826030-00004

Return to footnote 37 referrer

Footnote 38

CDC. Measles (Rubeola). 2020. Available from: https://www.cdc.gov/measles/hcp/index.html.

Return to footnote 38 referrer

Footnote 39

Pitman R., Fisman D., Zaric G.S., Postma M., Kretzschmar M., Edmunds J., et al. Dynamic transmission modeling: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force--5. Value Health. 2012; 15(6): 828-34. doi:10.1016/j.jval.2012.06.011

Return to footnote 39 referrer

Footnote 40

Ultsch B., Damm O., Beutels P., Bilcke J., Brüggenjürgen B., Gerber-Grote A., et al. Methods for Health Economic Evaluation of Vaccines and Immunization Decision Frameworks: A Consensus Framework from a European Vaccine Economics Community. Pharmacoeconomics. 2016; 34(3): 227-44. doi:10.1007/s40273-015-0335-2

Return to footnote 40 referrer

Footnote 41

Mauskopf J., Talbird S., Standaert B. Categorization of methods used in cost-effectiveness analyses of vaccination programs based on outcomes from dynamic transmission models. Expert Rev Pharmacoecon Outcomes Res. 2012; 12(3): 357-71. doi:10.1586/erp.12.11

Return to footnote 41 referrer

Footnote 42

Bilcke J., Marais C., Ogunjimi B., Willem L., Hens N., Beutels P. Cost-effectiveness of vaccination against herpes zoster in adults aged over 60 years in Belgium. Vaccine. 2012; 30(3): 675-84. doi:10.1016/j.vaccine.2011.10.036

Return to footnote 42 referrer

Footnote 43

Zechmeister I., Blasio B.F., Garnett G., Neilson A.R., Siebert U. Cost-effectiveness analysis of human papillomavirus-vaccination programs to prevent cervical cancer in Austria. Vaccine. 2009; 27(37): 5133-41. doi:10.1016/j.vaccine.2009.06.039

Return to footnote 43 referrer

Footnote 44

Attema A.E., Brouwer W.B.F., Claxton K. Discounting in Economic Evaluations. Pharmacoeconomics. 2018; 36(7): 745-58. doi:10.1007/s40273-018-0672-z

Return to footnote 44 referrer

Footnote 45

Jit M., Mibei W. Discounting in the evaluation of the cost-effectiveness of a vaccination programme: A critical review. Vaccine. 2015; 33(32): 3788-94. doi:10.1016/j.vaccine.2015.06.084

Return to footnote 45 referrer

Footnote 46

Westra T.A., Parouty M., Brouwer W.B., Beutels P.H., Rogoza R.M., Rozenbaum M.H., et al. On discounting of health gains from human papillomavirus vaccination: effects of different approaches. Value Health. 2012; 15(3): 562-7. doi:10.1016/j.jval.2012.01.005

Return to footnote 46 referrer

Footnote 47

OECD. Economic Valuation of Environmental Health Risks to Children: OECD Publishing; 2006.

Return to footnote 47 referrer

Footnote 48

National Institute for Health and Care Excellence (NICE). Discounting of health benefits in special circumstances. 2011.

Return to footnote 48 referrer

Footnote 49

STIKO. Modelling methods for predicting epidemiological and health economic effects of vaccinations - guidance for analyses to be presented to the German Standing Committee on Vaccination (STIKO). 2016.

Return to footnote 49 referrer

Footnote 50

Weinstein M.C., Stason W.B. Foundations of cost-effectiveness analysis for health and medical practices. N Engl J Med. 1977; 296(13): 716-21. doi:10.1056/nejm197703312961304

Return to footnote 50 referrer

Footnote 51

Gravelle H., Smith D. Discounting for health effects in cost-benefit and cost-effectiveness analysis. Health Econ. 2001; 10(7): 587-99. doi:10.1002/hec.618

Return to footnote 51 referrer

Footnote 52

O'Mahoney J.F., Postma M.J., Boersma C., Van Rosmalen J., Jit M., editors. Correcting for Multiple Future Cohorts When Applying Differential Discounting of Costs and Health Effects. Society for Medical Decision Making 14th Bienniel European Meeting; 2012.

Return to footnote 52 referrer

Footnote 53

O'Mahony J.F., de Kok I.M., van Rosmalen J., Habbema J.D., Brouwer W., van Ballegooijen M. Practical implications of differential discounting in cost-effectiveness analyses with varying numbers of cohorts. Value Health. 2011; 14(4): 438-42. doi:10.1016/j.jval.2010.09.009

Return to footnote 53 referrer

Footnote 54

Brennan A., Chick S.E., Davies R. A taxonomy of model structures for economic evaluation of health technologies. Health Econ. 2006; 15(12): 1295-310. doi:10.1002/hec.1148

Return to footnote 54 referrer

Footnote 55

Stahl J.E. Modelling methods for pharmacoeconomics and health technology assessment: an overview and guide. Pharmacoeconomics. 2008; 26(2): 131-48. doi:10.2165/00019053-200826020-00004

Return to footnote 55 referrer

Footnote 56

Mac S., Mishra S., Ximenes R., Barrett K., Khan Y.A., Naimark D.M.J., et al. Modeling the coronavirus disease 2019 pandemic: A comprehensive guide of infectious disease and decision-analytic models. J Clin Epidemiol. 2021; 132: 133-41. doi:10.1016/j.jclinepi.2020.12.002

Return to footnote 56 referrer

Footnote 57

Brisson M., Edmunds W.J. Impact of model, methodological, and parameter uncertainty in the economic analysis of vaccination programs. Med Decis Making. 2006; 26(5): 434-46. doi:10.1177/0272989x06290485

Return to footnote 57 referrer

Footnote 58

World Health Organization (WHO). Guide for Standardization of Economic Evaluations of Immunization Programs: 2nd Edition. 2019.

Return to footnote 58 referrer

Footnote 59

Jit M., Brisson M. Modelling the epidemiology of infectious diseases for decision analysis: a primer. Pharmacoeconomics. 2011; 29(5): 371-86. doi:10.2165/11539960-000000000-00000

Return to footnote 59 referrer

Footnote 60

Lipsitch M. Bacterial vaccines and serotype replacement: lessons from Haemophilus influenzae and prospects for Streptococcus pneumoniae. Emerg Infect Dis. 1999; 5(3): 336-45. doi:10.3201/eid0503.990304

Return to footnote 60 referrer

Footnote 61

Weinberger D.M., Malley R., Lipsitch M. Serotype replacement in disease after pneumococcal vaccination. Lancet (London, England). 2011; 378(9807): 1962-73. doi:10.1016/S0140-6736(10)62225-8

Return to footnote 61 referrer

Footnote 62

Berge J.J., Drennan D.P., Jacobs R.J., Jakins A., Meyerhoff A.S., Stubblefield W., et al. The cost of hepatitis A infections in American adolescents and adults in 1997. Hepatology. 2000; 31(2): 469-73. doi:10.1002/hep.510310229

Return to footnote 62 referrer

Footnote 63

Edmunds W.J., Brisson M. The effect of vaccination on the epidemiology of varicella zoster virus. J Infect. 2002; 44(4): 211-9. doi:10.1053/jinf.2002.0988

Return to footnote 63 referrer

Footnote 64

Chhatwal J., He T. Economic evaluations with agent-based modelling: an introduction. Pharmacoeconomics. 2015; 33(5): 423-33. doi:10.1007/s40273-015-0254-2

Return to footnote 64 referrer

Footnote 65

Marshall D.A., Burgos-Liz L., MJ I.J., Crown W., Padula W.V., Wong P.K., et al. Selecting a dynamic simulation modeling method for health care delivery research-part 2: report of the ISPOR Dynamic Simulation Modeling Emerging Good Practices Task Force. Value Health. 2015; 18(2): 147-60. doi:10.1016/j.jval.2015.01.006

Return to footnote 65 referrer

Footnote 66

Grefenstette J.J., Brown S.T., Rosenfeld R., DePasse J., Stone N.T.B., Cooley P.C., et al. FRED (A Framework for Reconstructing Epidemic Dynamics): an open-source software system for modeling infectious diseases and control strategies using census-based populations. BMC Public Health. 2013; 13(1): 940. doi:10.1186/1471-2458-13-940

Return to footnote 66 referrer

Footnote 67

Wolff E., Storsaeter J., Örtqvist Å., Naucler P., Larsson S., Lepp T., et al. Cost-effectiveness of pneumococcal vaccination for elderly in Sweden. Vaccine. 2020; 38(32): 4988-95. doi:10.1016/j.vaccine.2020.05.072

Return to footnote 67 referrer

Footnote 68

Ethgen O., Standaert B. Population- versus cohort-based modelling approaches. Pharmacoeconomics. 2012; 30(3): 171-81. doi:10.2165/11593050-000000000-00000

Return to footnote 68 referrer

Footnote 69

Vanni T., Karnon J., Madan J., White R.G., Edmunds W.J., Foss A.M., et al. Calibrating models in economic evaluation: a seven-step approach. Pharmacoeconomics. 2011; 29(1): 35-49. doi:10.2165/11584600-000000000-00000

Return to footnote 69 referrer

Footnote 70

Van de Velde N., Boily M.C., Drolet M., Franco E.L., Mayrand M.H., Kliewer E.V., et al. Population-level impact of the bivalent, quadrivalent, and nonavalent human papillomavirus vaccines: a model-based analysis. J Natl Cancer Inst. 2012; 104(22): 1712-23. doi:10.1093/jnci/djs395

Return to footnote 70 referrer

Footnote 71

Zang X., Krebs E., Min J.E., Pandya A., Marshall B.D.L., Schackman B.R., et al. Development and Calibration of a Dynamic HIV Transmission Model for 6 US Cities. Med Decis Making. 2020; 40(1): 3-16. doi:10.1177/0272989x19889356

Return to footnote 71 referrer

Footnote 72

Taylor D.C., Pawar V., Kruzikas D.T., Gilmore K.E., Sanon M., Weinstein M.C. Incorporating calibrated model parameters into sensitivity analyses: deterministic and probabilistic approaches. Pharmacoeconomics. 2012; 30(2): 119-26. doi:10.2165/11593360-000000000-00000

Return to footnote 72 referrer

Footnote 73

van der Steen A., van Rosmalen J., Kroep S., van Hees F., Steyerberg E.W., de Koning H.J., et al. Calibrating Parameters for Microsimulation Disease Models: A Review and Comparison of Different Goodness-of-Fit Criteria. Med Decis Making. 2016; 36(5): 652-65. doi:10.1177/0272989x16636851

Return to footnote 73 referrer

Footnote 74

Eddy D.M., Hollingworth W., Caro J.J., Tsevat J., McDonald K.M., Wong J.B. Model transparency and validation: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force--7. Value Health. 2012; 15(6): 843-50. doi:10.1016/j.jval.2012.04.012

Return to footnote 74 referrer

Footnote 75

Van de Velde N., Brisson M., Boily M.C. Modeling human papillomavirus vaccine effectiveness: quantifying the impact of parameter uncertainty. Am J Epidemiol. 2007; 165(7): 762-75. doi:10.1093/aje/kwk059

Return to footnote 75 referrer

Footnote 76

Ghabri S., Stevenson M., Möller J., Caro J.J. Trusting the Results of Model-Based Economic Analyses: Is there a Pragmatic Validation Solution? Pharmacoeconomics. 2019; 37(1): 1-6. doi:10.1007/s40273-018-0711-9

Return to footnote 76 referrer

Footnote 77

Government of Canada. Recommended immunization schedules: Canadian Immunization Guide. 2020. Available from: https://www.canada.ca/en/public-health/services/publications/healthy-living/canadian-immunization-guide-part-1-key-immunization-information/page-13-recommended-immunization-schedules.html.

Return to footnote 77 referrer

Footnote 78

Fine P., Eames K., Heymann D.L. "Herd immunity": a rough guide. Clin Infect Dis. 2011; 52(7): 911-6. doi:10.1093/cid/cir007

Return to footnote 78 referrer

Footnote 79

Czerkinsky C., Holmgren J. Vaccines against enteric infections for the developing world. Philos Trans R Soc Lond B Biol Sci. 2015; 370(1671). doi:10.1098/rstb.2015.0142

Return to footnote 79 referrer

Footnote 80

Gjini E. Geographic variation in pneumococcal vaccine efficacy estimated from dynamic modeling of epidemiological data post-PCV7. Sci Rep. 2017; 7(1): 3049. doi:10.1038/s41598-017-02955-y

Return to footnote 80 referrer

Footnote 81

Plotkin S.A. Complex correlates of protection after vaccination. Clin Infect Dis. 2013; 56(10): 1458-65. doi:10.1093/cid/cit048

Return to footnote 81 referrer

Footnote 82

Government of Canada. Vaccine Coverage in Canadian Children: Results from the 2017 Childhood National Immunization Coverage Survey (cNICS). 2020.

Return to footnote 82 referrer

Footnote 83

Krebs E., Nosyk B. Cost-Effectiveness Analysis in Implementation Science: a Research Agenda and Call for Wider Application. Curr HIV/AIDS Rep. 2021; 18(3): 176-85. doi:10.1007/s11904-021-00550-5

Return to footnote 83 referrer

Footnote 84

Andrews N.J., Waight P.A., Burbidge P., Pearce E., Roalfe L., Zancolli M., et al. Serotype-specific effectiveness and correlates of protection for the 13-valent pneumococcal conjugate vaccine: a postlicensure indirect cohort study. Lancet Infect Dis. 2014; 14(9): 839-46. doi:10.1016/s1473-3099(14)70822-9

Return to footnote 84 referrer

Footnote 85

Daniels C.C., Rogers P.D., Shelton C.M. A Review of Pneumococcal Vaccines: Current Polysaccharide Vaccine Recommendations and Future Protein Antigens. J Pediatr Pharmacol Ther. 2016; 21(1): 27-35. doi:10.5863/1551-6776-21.1.27

Return to footnote 85 referrer

Footnote 86

Malagón T., Drolet M., Boily M.C., Franco E.L., Jit M., Brisson J., et al. Cross-protective efficacy of two human papillomavirus vaccines: a systematic review and meta-analysis. Lancet Infect Dis. 2012; 12(10): 781-9. doi:10.1016/s1473-3099(12)70187-1

Return to footnote 86 referrer

Footnote 87

Plotkin S.A. Updates on immunologic correlates of vaccine-induced protection. Vaccine. 2020; 38(9): 2250-7. doi:10.1016/j.vaccine.2019.10.046

Return to footnote 87 referrer

Footnote 88

World Health Organization (WHO). Guidelines on clinical evaluation of vaccines: regulatory expectations. 2017.

Return to footnote 88 referrer

Footnote 89

CDC. Immunogenicity, Efficacy, and Effectiveness of Influenza Vaccines. 2019. Available from: https://www.cdc.gov/flu/professionals/acip/immunogenicity.htm.

Return to footnote 89 referrer

Footnote 90

Plotkin S.A., Gilbert P.B. Nomenclature for immune correlates of protection after vaccination. Clin Infect Dis. 2012; 54(11): 1615-7. doi:10.1093/cid/cis238

Return to footnote 90 referrer

Footnote 91

World Health Organization (WHO). Correlates of vaccine-induced protection: methods and implications. 2013.

Return to footnote 91 referrer

Footnote 92

Cowling B.J., Lim W.W., Perera R., Fang V.J., Leung G.M., Peiris J.S.M., et al. Influenza Hemagglutination-inhibition Antibody Titer as a Mediator of Vaccine-induced Protection for Influenza B. Clin Infect Dis. 2019; 68(10): 1713-17. doi:10.1093/cid/ciy759

Return to footnote 92 referrer

Footnote 93

Gilbert P.B., Luedtke A.R. Statistical Learning Methods to Determine Immune Correlates of Herpes Zoster in Vaccine Efficacy Trials. J Infect Dis. 2018; 218(suppl_2): S99-s101. doi:10.1093/infdis/jiy421

Return to footnote 93 referrer

Footnote 94

Belshe R.B., Gruber W.C., Mendelman P.M., Mehta H.B., Mahmood K., Reisinger K., et al. Correlates of immune protection induced by live, attenuated, cold-adapted, trivalent, intranasal influenza virus vaccine. J Infect Dis. 2000; 181(3): 1133-7. doi:10.1086/315323

Return to footnote 94 referrer

Footnote 95

Bentebibel S.E., Lopez S., Obermoser G., Schmitt N., Mueller C., Harrod C., et al. Induction of ICOS+CXCR3+CXCR5+ TH cells correlates with antibody responses to influenza vaccination. Sci Transl Med. 2013; 5(176): 176ra32. doi:10.1126/scitranslmed.3005191

Return to footnote 95 referrer

Footnote 96

Cooper L.V., Boukary R.M., Aseffa A., Mihret W., Collard J.M., Daugla D., et al. Investigation of correlates of protection against pharyngeal carriage of Neisseria meningitidis genogroups W and Y in the African meningitis belt. PLoS One. 2017; 12(8): e0182575. doi:10.1371/journal.pone.0182575

Return to footnote 96 referrer

Footnote 97

Plotkin S.A. Correlates of protection induced by vaccination. Clin Vaccine Immunol. 2010; 17(7): 1055-65. doi:10.1128/cvi.00131-10

Return to footnote 97 referrer

Footnote 98

Bilcke J., Ogunjimi B., Hulstaert F., Van Damme P., Hens N., Beutels P. Estimating the age-specific duration of herpes zoster vaccine protection: a matter of model choice? Vaccine. 2012; 30(17): 2795-800. doi:10.1016/j.vaccine.2011.09.079

Return to footnote 98 referrer

Footnote 99

Li X., Zhang J.H., Betts R.F., Morrison V.A., Xu R., Itzler R.F., et al. Modeling the durability of ZOSTAVAX® vaccine efficacy in people ≥60 years of age. Vaccine. 2015; 33(12): 1499-505. doi:10.1016/j.vaccine.2014.10.039

Return to footnote 99 referrer

Footnote 100

Zoratti M.J., Pickard A.S., Stalmeier P.F.M., Ollendorf D., Lloyd A., Chan K.K.W., et al. Evaluating the conduct and application of health utility studies: a review of critical appraisal tools and reporting checklists. Eur J Health Econ. 2021; 22(5): 723-33. doi:10.1007/s10198-021-01286-0

Return to footnote 100 referrer

Footnote 101

Ungar W.J. Challenges in Health State Valuation in Paediatric Economic Evaluation. PharmacoEconomics. 2011; 29(8): 641-52. doi:10.2165/11591570-000000000-00000

Return to footnote 101 referrer

Footnote 102

Oliveira C., de Silva N.T., Ungar W.J., Bayoumi A.M., Avitzur Y., Hoch J.S., et al. Health-related quality of life in neonates and infants: a conceptual framework. Quality of Life Research. 2020; 29(5): 1159-68. doi:10.1007/s11136-020-02432-6

Return to footnote 102 referrer

Footnote 103

Ungar W.J., editor. Economic evaluation in child health: Oxford University Press; 2019.

Return to footnote 103 referrer

Footnote 104

Brazier J., Ara R., Azzabi I., Busschbach J., Chevrou-Séverac H., Crawford B., et al. Identification, Review, and Use of Health State Utilities in Cost-Effectiveness Models: An ISPOR Good Practices for Outcomes Research Task Force Report. Value Health. 2019; 22(3): 267-75. doi:10.1016/j.jval.2019.01.004

Return to footnote 104 referrer

Footnote 105

Peasgood T., Brazier J. Is Meta-Analysis for Utility Values Appropriate Given the Potential Impact Different Elicitation Methods Have on Values? PharmacoEconomics. 2015; 33(11): 1101-5. doi:10.1007/s40273-015-0310-y

Return to footnote 105 referrer

Footnote 106

Wittenberg E., James L.P., Prosser L.A. Spillover Effects on Caregivers' and Family Members' Utility: A Systematic Review of the Literature. Pharmacoeconomics. 2019; 37(4): 475-99. doi:10.1007/s40273-019-00768-7

Return to footnote 106 referrer

Footnote 107

Cookson R., Skarda I., Cotton-Barratt O., Adler M., Asaria M., Ord T. Quality adjusted life years based on health and consumption: A summary wellbeing measure for cross-sectoral economic evaluation. Health Econ. 2021; 30(1): 70-85. doi:10.1002/hec.4177

Return to footnote 107 referrer

Footnote 108

de Vries L.M., van Baal P.H.M., Brouwer W.B.F. Future Costs in Cost-Effectiveness Analyses: Past, Present, Future. Pharmacoeconomics. 2019; 37(2): 119-30. doi:10.1007/s40273-018-0749-8

Return to footnote 108 referrer

Footnote 109

Feeny D., Krahn M., Prosser L.A., Salomon J.A. Valuing Health Outcomes. Cost-Effectiveness in Health and Medicine. 2 ed. New York: Oxford University Press; 2016.

Return to footnote 109 referrer

Footnote 110

Nyman J.A. Should the consumption of survivors be included as a cost in cost-utility analysis? Health Econ. 2004; 13(5): 417-27. doi:10.1002/hec.850

Return to footnote 110 referrer

Footnote 111

Shiroiwa T., Fukuda T., Ikeda S., Shimozuma K. QALY and productivity loss: empirical evidence for "double counting". Value Health. 2013; 16(4): 581-7. doi:10.1016/j.jval.2013.02.009

Return to footnote 111 referrer

Footnote 112

Tilling C., Krol M., Tsuchiya A., Brazier J., Brouwer W. In or out? Income losses in health state valuations: a review. Value Health. 2010; 13(2): 298-305. doi:10.1111/j.1524-4733.2009.00614.x

Return to footnote 112 referrer

Footnote 113

Adarkwah C.C., Sadoghi A., Gandjour A. Should Cost-Effectiveness Analysis Include the Cost of Consumption Activities? AN Empirical Investigation. Health Econ. 2016; 25(2): 249-56. doi:10.1002/hec.3162

Return to footnote 113 referrer

Footnote 114

Basu A. Estimating Costs and Valuations of Non-Health Benefits in Cost-Effectiveness Analysis. Cost-Effectiveness in Health and Medicine. 2 ed. New York: Oxford University Press; 2016.

Return to footnote 114 referrer

Footnote 115

Salomon J.A., Trikalinos T.A., Sanders G.D., Mandelblatt J.S. Identifying and Quantifying the Consequences of Interventions. Cost-Effectiveness in Health and Medicine. 2 ed. New York: Oxford University Press; 2016.

Return to footnote 115 referrer

Footnote 116

Jacobs P., Budden A., Lee K.M. Guidance document fo rthe costing of health care resources in the Canadian setting. 2nd edition. 2016.

Return to footnote 116 referrer

Footnote 117

Feenstra T.L., van Baal P.H., Gandjour A., Brouwer W.B. Future costs in economic evaluation. A comment on Lee. J Health Econ. 2008; 27(6): 1645-9; discussion 50-1. doi:10.1016/j.jhealeco.2008.07.007

Return to footnote 117 referrer

Footnote 118

Morton A., Adler A.I., Bell D., Briggs A., Brouwer W., Claxton K., et al. Unrelated Future Costs and Unrelated Future Benefits: Reflections on NICE Guide to the Methods of Technology Appraisal. Health Econ. 2016; 25(8): 933-8. doi:10.1002/hec.3366

Return to footnote 118 referrer

Footnote 119

Meltzer D. Accounting for future costs in medical cost-effectiveness analysis. J Health Econ. 1997; 16(1): 33-64. doi:10.1016/s0167-6296(96)00507-3

Return to footnote 119 referrer

Footnote 120

Manns B., Meltzer D., Taub K., Donaldson C. Illustrating the impact of including future costs in economic evaluations: an application to end-stage renal disease care. Health Econ. 2003; 12(11): 949-58. doi:10.1002/hec.790

Return to footnote 120 referrer

Footnote 121

CIHI. National Health Expenditure Trends. 2021. Available from: https://www.cihi.ca/en/national-health-expenditure-trends.

Return to footnote 121 referrer

Footnote 122

Levy A.R., Sobolev B., James D., Barrable W., Clarke-Richardson P., Sullivan S.D., et al. The costs of change: direct medical costs of solid organ transplantation in British Columbia, Canada, 1995-2003. Value Health. 2009; 12(2): 282-92. doi:10.1111/j.1524-4733.2008.00445.x

Return to footnote 122 referrer

Footnote 123

Manns B.J., Mendelssohn D.C., Taub K.J. The economics of end-stage renal disease care in Canada: incentives and impact on delivery of care. Int J Health Care Finance Econ. 2007; 7(2-3): 149-69. doi:10.1007/s10754-007-9022-y

Return to footnote 123 referrer

Footnote 124

CDC. Guidelines for Evaluating Surveillance Systems. 1988. Available from: https://www.cdc.gov/mmwr/preview/mmwrhtml/00001769.htm.

Return to footnote 124 referrer

Footnote 125

Coleman M.S., Garbat-Welch L., Burke H., Weinberg M., Humbaugh K., Tindall A., et al. Direct costs of a single case of refugee-imported measles in Kentucky. Vaccine. 2012; 30(2): 317-21. doi:10.1016/j.vaccine.2011.10.091

Return to footnote 125 referrer

Footnote 126

Dalton C.B., Haddix A., Hoffman R.E., Mast E.E. The cost of a food-borne outbreak of hepatitis A in Denver, Colo. Arch Intern Med. 1996; 156(9): 1013-6.

Return to footnote 126 referrer

Footnote 127

Dayan G.H., Ortega-Sánchez I.R., LeBaron C.W., Quinlisk M.P. The cost of containing one case of measles: the economic impact on the public health infrastructure--Iowa, 2004. Pediatrics. 2005; 116(1): e1-4. doi:10.1542/peds.2004-2512

Return to footnote 127 referrer

Footnote 128

Parker A.A., Staggs W., Dayan G.H., Ortega-Sánchez I.R., Rota P.A., Lowe L., et al. Implications of a 2005 measles outbreak in Indiana for sustained elimination of measles in the United States. N Engl J Med. 2006; 355(5): 447-55. doi:10.1056/NEJMoa060775

Return to footnote 128 referrer

Footnote 129

Institut national de sante publique du Quebec. Publications. n.d. Available from: https://www.inspq.qc.ca/en/publications/.

Return to footnote 129 referrer

Footnote 130

Musto R., Siever J.E., Johnston J.C., Seidel J., Rose M.S., McNeil D.A. Social equity in Human Papillomavirus vaccination: a natural experiment in Calgary Canada. BMC Public Health. 2013; 13: 640. doi:10.1186/1471-2458-13-640

Return to footnote 130 referrer

Footnote 131

Workplace Safety and Insurance Board. Operational policy manual: travel and related expenses. 2017.

Return to footnote 131 referrer

Footnote 132

Doherty M., Buchy P., Standaert B., Giaquinto C., Prado-Cohrs D. Vaccine impact: Benefits for human health. Vaccine. 2016; 34(52): 6707-14. doi:10.1016/j.vaccine.2016.10.025

Return to footnote 132 referrer

Footnote 133

Koopmanschap M.A., Rutten F.F., van Ineveld B.M., van Roijen L. The friction cost method for measuring indirect costs of disease. J Health Econ. 1995; 14(2): 171-89. doi:10.1016/0167-6296(94)00044-5

Return to footnote 133 referrer

Footnote 134

Krol M., Brouwer W. How to estimate productivity costs in economic evaluations. Pharmacoeconomics. 2014; 32(4): 335-44. doi:10.1007/s40273-014-0132-3

Return to footnote 134 referrer

Footnote 135

Pike J., Grosse S.D. Friction Cost Estimates of Productivity Costs in Cost-of-Illness Studies in Comparison with Human Capital Estimates: A Review. Appl Health Econ Health Policy. 2018; 16(6): 765-78. doi:10.1007/s40258-018-0416-4

Return to footnote 135 referrer

Footnote 136

Goeree R., O'Brien B.J., Blackhouse G., Agro K., Goering P. The valuation of productivity costs due to premature mortality: a comparison of the human-capital and friction-cost methods for schizophrenia. Can J Psychiatry. 1999; 44(5): 455-63. doi:10.1177/070674379904400505

Return to footnote 136 referrer

Footnote 137

Hank K., Stuck S. Volunteer work, informal help, and care among the 50+ in Europe: further evidence for 'linked' productive activities at older ages. Soc Sci Res. 2008; 37(4): 1280-91. doi:10.1016/j.ssresearch.2008.03.001

Return to footnote 137 referrer

Footnote 138

Bouwmans C., Krol M., Severens H., Koopmanschap M., Brouwer W., Hakkaart-van Roijen L. The iMTA Productivity Cost Questionnaire: A Standardized Instrument for Measuring and Valuing Health-Related Productivity Losses. Value Health. 2015; 18(6): 753-8. doi:10.1016/j.jval.2015.05.009

Return to footnote 138 referrer

Footnote 139

Pearce A. Productivity losses and how they are calculated. In: (CHERE) C.f.H.E.R.a.E., editor. 2016.

Return to footnote 139 referrer

Footnote 140

Statistics Canada. Average usual and actual hours worked in a reference week by type of work (full- and part-time), annual. 2021. Available from: https://www150.statcan.gc.ca/t1/tbl1/en/tv.action?pid=1410004301.

Return to footnote 140 referrer

Footnote 141

Statistics Canada. Income of individuals by age group, sex and income source, Canada, provinces and selected census metropolitan areas. 2021. Available from: https://www150.statcan.gc.ca/t1/tbl1/en/tv.action?pid=1110023901.

Return to footnote 141 referrer

Footnote 142

Prosser L.A. SMDM President's Column: Gender bias and cost-effectiveness analysis. 2020. Available from: https://myemail.constantcontact.com/SMDM-Newsletter---Issue-4--September-2020.html?soid=1116971938232&aid=ywAYbDuCkUY.

Return to footnote 142 referrer

Footnote 143

Zhang W., Bansback N., Boonen A., Severens J.L., Anis A.H. Development of a composite questionnaire, the valuation of lost productivity, to value productivity losses: application in rheumatoid arthritis. Value Health. 2012; 15(1): 46-54. doi:10.1016/j.jval.2011.07.009

Return to footnote 143 referrer

Footnote 144

Grosse S.D., Pike J., Soelaeman R., Tilford J.M. Quantifying Family Spillover Effects in Economic Evaluations: Measurement and Valuation of Informal Care Time. Pharmacoeconomics. 2019; 37(4): 461-73. doi:10.1007/s40273-019-00782-9

Return to footnote 144 referrer

Footnote 145

Hoefman R.J., van Exel J., Brouwer W. How to include informal care in economic evaluations. Pharmacoeconomics. 2013; 31(12): 1105-19. doi:10.1007/s40273-013-0104-z

Return to footnote 145 referrer

Footnote 146

McKibbin W.J., Fernando R. Global macroeconomic scenarios of the COVID-19 pandemic. CAMA Working Papers. 2020.

Return to footnote 146 referrer

Footnote 147

Statistics Canada. Household spending, Canada, regions and provinces. 2021. Available from: https://www150.statcan.gc.ca/t1/tbl1/en/tv.action?pid=1110022201.

Return to footnote 147 referrer

Footnote 148

Statistics Canada. Unit of analysis: the household. 2020. Available from: https://www150.statcan.gc.ca/n1/pub/13-607-x/2016001/952-eng.htm.

Return to footnote 148 referrer

Footnote 149

OECD. OECD Framework for Statistics on the Distribution of Household Income, Consumption and Wealth2013.

Return to footnote 149 referrer

Footnote 150

Roed C., Omland L.H., Skinhoj P., Rothman K.J., Sorensen H.T., Obel N. Educational achievement and economic self-sufficiency in adults after childhood bacterial meningitis. Jama. 2013; 309(16): 1714-21. doi:10.1001/jama.2013.3792

Return to footnote 150 referrer

Footnote 151

Federal Reserve Bank of Cleveland. Educational attainment and employment. 2011. Available from: https://www.clevelandfed.org/en/newsroom-and-events/publications/economic-trends/2011-economic-trends/et-20110302-educational-attainment-and-employment.aspx.

Return to footnote 151 referrer

Footnote 152

US Bureau of Labor Statistics. Earnings and unemployment rates by educational attainment, 2020. 2021.

Return to footnote 152 referrer

Footnote 153

Gallagher-Mackay K., Srivastava P., Underwood K., Dhuey E., McCready L., Born K.B., et al. COVID-19 and education disruption in Ontario: emerging evidence on impacts. 2021.

Return to footnote 153 referrer

Footnote 154

Grimwood K., Anderson V.A., Bond L., Catroppa C., Hore R.L., Keir E.H., et al. Adverse outcomes of bacterial meningitis in school-age survivors. Pediatrics. 1995; 95(5): 646-56.

Return to footnote 154 referrer

Footnote 155

Klein E.Y., Schueller E., Tseng K.K., Morgan D.J., Laxminarayan R., Nandi A. The Impact of Influenza Vaccination on Antibiotic Use in the United States, 2010-2017. Open Forum Infect Dis. 2020; 7(7): ofaa223. doi:10.1093/ofid/ofaa223

Return to footnote 155 referrer

Footnote 156

Klugman K.P., Black S. Impact of existing vaccines in reducing antibiotic resistance: Primary and secondary effects. Proc Natl Acad Sci U S A. 2018; 115(51): 12896-901. doi:10.1073/pnas.1721095115

Return to footnote 156 referrer

Footnote 157

Pruden A., Larsson D.G.J., Amézquita A., Collignon P., Brandt K.K., Graham D.W., et al. Management options for reducing the release of antibiotics and antibiotic resistance genes to the environment. Environ Health Perspect. 2013; 121(8): 878-85. doi:10.1289/ehp.1206446

Return to footnote 157 referrer

Footnote 158

Akenji L., Bengtsson V.T., Lettenmeier M., Fawcett T., YParag Y., Saheb Y., et al. 1.5-degree lifestyles: twoards a cair consumption space for all. Berlin: Hot or Cool Institute, 2021.

Return to footnote 158 referrer

Footnote 159

Government of Canada. Global greenhouse gas emissions. 2022. Available from: https://www.canada.ca/en/environment-climate-change/services/environmental-indicators/global-greenhouse-gas-emissions.html.

Return to footnote 159 referrer

Footnote 160

Community Research Connections. Waste. n.d. Available from: https://www.crcresearch.org/solutions-agenda/waste.

Return to footnote 160 referrer

Footnote 161

Government of Canada. Municipal Solid Waste: Statistics and Trends. 2020. Available from: https://www.canada.ca/en/environment-climate-change/services/managing-reducing-waste/municipal-solid/statistics-trends.html.

Return to footnote 161 referrer

Footnote 162

Briggs A.H., Claxton K., Sculpher M.J. Decision Modelling for Health Economic Evaluation: Oxford University Press; 2006.

Return to footnote 162 referrer

Footnote 163

Bilcke J., Beutels P., Brisson M., Jit M. Accounting for methodological, structural, and parameter uncertainty in decision-analytic models: a practical guide. Med Decis Making. 2011; 31(4): 675-92. doi:10.1177/0272989x11409240

Return to footnote 163 referrer

Footnote 164

Briggs A.H., Weinstein M.C., Fenwick E.A., Karnon J., Sculpher M.J., Paltiel A.D. Model parameter estimation and uncertainty: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force--6. Value Health. 2012; 15(6): 835-42. doi:10.1016/j.jval.2012.04.014

Return to footnote 164 referrer

Footnote 165

Mahsin M.D., Deardon R., Brown P. Geographically dependent individual-level models for infectious diseases transmission. Biostatistics. 2020. doi:10.1093/biostatistics/kxaa009

Return to footnote 165 referrer

Footnote 166

Popinga A., Vaughan T., Stadler T., Drummond A.J. Inferring epidemiological dynamics with Bayesian coalescent inference: the merits of deterministic and stochastic models. Genetics. 2015; 199(2): 595-607. doi:10.1534/genetics.114.172791

Return to footnote 166 referrer

Footnote 167

Vreman R.A., Geenen J.W., Knies S., Mantel-Teeuwisse A.K., Leufkens H.G.M., Goettsch W.G. The Application and Implications of Novel Deterministic Sensitivity Analysis Methods. Pharmacoeconomics. 2021; 39(1): 1-17. doi:10.1007/s40273-020-00979-3

Return to footnote 167 referrer

Footnote 168

Sickbert-Bennett E.E., Weber D.J., Poole C., MacDonald P.D., Maillard J.M. Completeness of communicable disease reporting, North Carolina, USA, 1995-1997 and 2000-2006. Emerg Infect Dis. 2011; 17(1): 23-9. doi:10.3201/eid1701.100660

Return to footnote 168 referrer

Footnote 169

Mattoo S., Cherry J.D. Molecular pathogenesis, epidemiology, and clinical manifestations of respiratory infections due to Bordetella pertussis and other Bordetella subspecies. Clin Microbiol Rev. 2005; 18(2): 326-82. doi:10.1128/cmr.18.2.326-382.2005

Return to footnote 169 referrer

Footnote 170

Lipsitch M., Jha A., Simonsen L. Observational studies and the difficult quest for causality: lessons from vaccine effectiveness and impact studies. Int J Epidemiol. 2016; 45(6): 2060-74. doi:10.1093/ije/dyw124

Return to footnote 170 referrer

Footnote 171

Berger M.L., Sox H., Willke R.J., Brixner D.L., Eichler H.G., Goettsch W., et al. Good practices for real-world data studies of treatment and/or comparative effectiveness: Recommendations from the joint ISPOR-ISPE Special Task Force on real-world evidence in health care decision making. Pharmacoepidemiol Drug Saf. 2017; 26(9): 1033-9. doi:10.1002/pds.4297

Return to footnote 171 referrer

Footnote 172

Sterne J.A., Hernán M.A., Reeves B.C., Savović J., Berkman N.D., Viswanathan M., et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016; 355: i4919. doi:10.1136/bmj.i4919

Return to footnote 172 referrer

Footnote 173

Del Valle S.Y., Hyman J.M., Chitnis N. Mathematical models of contact patterns between age groups for predicting the spread of infectious diseases. Math Biosci Eng. 2013; 10(5-6): 1475-97. doi:10.3934/mbe.2013.10.1475

Return to footnote 173 referrer

Footnote 174

Funk S., Bansal S., Bauch C.T., Eames K.T., Edmunds W.J., Galvani A.P., et al. Nine challenges in incorporating the dynamics of behaviour in infectious diseases models. Epidemics. 2015; 10: 21-5. doi:10.1016/j.epidem.2014.09.005

Return to footnote 174 referrer

Footnote 175

CDC. Hepatitis A Questions and Answers for Health Professionals. 2020. Available from: https://www.cdc.gov/hepatitis/hav/havfaq.htm.

Return to footnote 175 referrer

Footnote 176

Gravitt P.E. The known unknowns of HPV natural history. J Clin Invest. 2011; 121(12): 4593-9. doi:10.1172/jci57149

Return to footnote 176 referrer

Footnote 177

Campbell H., Andrews N., Borrow R., Trotter C., Miller E. Updated postlicensure surveillance of the meningococcal C conjugate vaccine in England and Wales: effectiveness, validation of serological correlates of protection, and modeling predictions of the duration of herd immunity. Clin Vaccine Immunol. 2010; 17(5): 840-7. doi:10.1128/cvi.00529-09

Return to footnote 177 referrer

Footnote 178

Lewnard J.A., Grad Y.H. Vaccine waning and mumps re-emergence in the United States. Sci Transl Med. 2018; 10(433). doi:10.1126/scitranslmed.aao5945

Return to footnote 178 referrer

Footnote 179

CDC. About Meningococcal Vaccines. 2021. Available from: https://www.cdc.gov/vaccines/vpd/mening/hcp/about-vaccine.html.

Return to footnote 179 referrer

Footnote 180

Bojke L., Claxton K., Sculpher M., Palmer S. Characterizing structural uncertainty in decision analytic models: a review and application of methods. Value Health. 2009; 12(5): 739-49. doi:10.1111/j.1524-4733.2008.00502.x

Return to footnote 180 referrer

Footnote 181

Jackson C.H., Thompson S.G., Sharples L.D. Accounting for uncertainty in health economic decision models by using model averaging. J R Stat Soc Ser A Stat Soc. 2009; 172(2): 383-404. doi:10.1111/j.1467-985X.2008.00573.x

Return to footnote 181 referrer

Footnote 182

Hofmann B. Toward a procedure for integrating moral issues in health technology assessment. Int J Technol Assess Health Care. 2005; 21(3): 312-8. doi:10.1017/s0266462305050415

Return to footnote 182 referrer

Footnote 183

Krahn M.D., Bielecki J.M., Bremner K.E., de Oliveira C., Almeida N., Clement F., et al. Picturing ELSI+: a visual representation of ethical, legal, and social issues, and patient experiences in Health Technology Assessment in Canada. Int J Technol Assess Health Care. 2020; 36(1): 40-9. doi:10.1017/s0266462319000722

Return to footnote 183 referrer

Footnote 184

Public Health Agency of Canada (PHAC). Framework for ethical deliberation and decision-making in public health: a tool for public health practitioners, policy makers and decision-makers. Ottawa, Canada2017.

Return to footnote 184 referrer

Footnote 185

World Health Organization (WHO). Health equity. N.d. Available from: https://www.who.int/health-topics/health-equity#tab=tab_1.

Return to footnote 185 referrer

Footnote 186

Rawlins M., Barnett D., Stevens A. Pharmacoeconomics: NICE's approach to decision-making. Br J Clin Pharmacol. 2010; 70(3): 346-9. doi:10.1111/j.1365-2125.2009.03589.x

Return to footnote 186 referrer

Footnote 187

Cookson R., Griffin S., Norheim O.F., Culyer A.J. Distributional Cost-Effectiveness Analysis: Quantifying Health Equity Impacts and Trade-Offs: Quantifying Health Equity Impacts and Trade-Offs. Oxford, UK: Oxford University Press; 2020 2020-10.

Return to footnote 187 referrer

Footnote 188

Johri M., Norheim O.F. Can cost-effectiveness analysis integrate concerns for equity? Systematic review. Int J Technol Assess Health Care. 2012; 28(2): 125-32. doi:10.1017/s0266462312000050

Return to footnote 188 referrer

Footnote 189

Norheim O.F., Baltussen R., Johri M., Chisholm D., Nord E., Brock D., et al. Guidance on priority setting in health care (GPS-Health): the inclusion of equity criteria not captured by cost-effectiveness analysis. Cost Eff Resour Alloc. 2014; 12: 18. doi:10.1186/1478-7547-12-18

Return to footnote 189 referrer

Footnote 190

Cookson R., Mirelman A.J., Griffin S., Asaria M., Dawkins B., Norheim O.F., et al. Using Cost-Effectiveness Analysis to Address Health Equity Concerns. Value Health. 2017; 20(2): 206-12. doi:10.1016/j.jval.2016.11.027

Return to footnote 190 referrer

Footnote 191

Smith M.J., Thompson A., Upshur R.E.G. Is 'health equity' bad for our health? A qualitative empirical ethics study of public health policy-makers' perspectives. Canadian Journal of Public Health. 2018; 109(5): 633-42. Doi:10.17269/s41997-018-0128-4

Return to footnote 191 referrer

Footnote 192

Al M.J., Feenstra T., Brouwer W.B. Decision makers' views on health care objectives and budget constraints: results from a pilot study. Health Policy. 2004; 70(1): 33-48. doi:10.1016/j.healthpol.2004.01.009

Return to footnote 192 referrer

Footnote 193

Converge3. Equity, values, and health decisions in Ontario: a Converge3 methods report. 2019.

Return to footnote 193 referrer

Footnote 194

O'Neill J., Tabish H., Welch V., Petticrew M., Pottie K., Clarke M., et al. Applying an equity lens to interventions: using PROGRESS ensures consideration of socially stratifying factors to illuminate inequities in health. J Clin Epidemiol. 2014; 67(1): 56-64. doi:10.1016/j.jclinepi.2013.08.005

Return to footnote 194 referrer

Footnote 195

Powers M., Powers A.P.P.M., Faden R.R., Faden R.R. Social Justice: The Moral Foundations of Public Health and Health Policy: Oxford University Press; 2006.

Return to footnote 195 referrer

Footnote 196

Flannery B., Schrag S., Bennett N.M., Lynfield R., Harrison L.H., Reingold A., et al. Impact of childhood vaccination on racial disparities in invasive Streptococcus pneumoniae infections. Jama. 2004; 291(18): 2197-203. doi:10.1001/jama.291.18.2197

Return to footnote 196 referrer

Footnote 197

Public Health Agency of Canada (PHAC). Key health inequalities in Canada: a national portrait. 2018.

Return to footnote 197 referrer

Footnote 198

Castle P.E., Zhao F.-H. Population Effectiveness, Not Efficacy, Should Decide Who Gets Vaccinated Against Human Papillomavirus via Publicly Funded Programs. J Infect Dis. 2011; 204(3): 335-7. doi:10.1093/infdis/jir287

Return to footnote 198 referrer

Footnote 199

Blakely T., Kvizhinadze G., Karvonen T., Pearson A.L., Smith M., Wilson N. Cost-effectiveness and equity impacts of three HPV vaccination programmes for school-aged girls in New Zealand. Vaccine. 2014; 32(22): 2645-56. doi:10.1016/j.vaccine.2014.02.071

Return to footnote 199 referrer

Footnote 200

Culyer A.J. Equity of what in healthcare? Why the traditional answers don't help policy--and what to do in the future. Healthc Pap. 2007; 8 Spec No: 12-26. doi:10.12927/hcpap.2007.19216

Return to footnote 200 referrer

Footnote 201

Menzies R.I., Singleton R.J. Vaccine preventable diseases and vaccination policy for indigenous populations. Pediatr Clin North Am. 2009; 56(6): 1263-83. doi:10.1016/j.pcl.2009.09.006

Return to footnote 201 referrer

Footnote 202

Veras M., Zakus D. Equity Aspects of Canadian Immunization Programs: Differences within and between countries. 2011. 2011: 22. doi:10.5195/hcs.2011.45

Return to footnote 202 referrer

Footnote 203

Brearley L., Eggers R., Steinglass R., Vandelaer J. Applying an equity lens in the Decade of Vaccines. Vaccine. 2013; 31 Suppl 2: B103-7. doi:10.1016/j.vaccine.2012.11.088

Return to footnote 203 referrer

Footnote 204

Quilici S., Smith R., Signorelli C. Role of vaccination in economic growth. J Mark Access Health Policy. 2015; 3: 10.3402/jmahp.v3.27044. doi:10.3402/jmahp.v3.27044

Return to footnote 204 referrer

Footnote 205

Avanceña A.L.V., Prosser L.A. Examining Equity Effects of Health Interventions in Cost-Effectiveness Analysis: A Systematic Review. Value Health. 2021; 24(1): 136-43. doi:10.1016/j.jval.2020.10.010

Return to footnote 205 referrer

Footnote 206

Verguet S., Kim J.J., Jamison D.T. Extended Cost-Effectiveness Analysis for Health Policy Assessment: A Tutorial. Pharmacoeconomics. 2016; 34(9): 913-23. doi:10.1007/s40273-016-0414-z

Return to footnote 206 referrer

Footnote 207

van Hoek A.J., Melegaro A., Gay N., Bilcke J., Edmunds W.J. The cost-effectiveness of varicella and combined varicella and herpes zoster vaccination programmes in the United Kingdom. Vaccine. 2012; 30(6): 1225-34. doi:10.1016/j.vaccine.2011.11.026

Return to footnote 207 referrer

Footnote 208

Crosignani P., De Stefani A., Fara G.M., Isidori A.M., Lenzi A., Liverani C.A., et al. Towards the eradication of HPV infection through universal specific vaccination. BMC Public Health. 2013; 13: 642. doi:10.1186/1471-2458-13-642

Return to footnote 208 referrer

Footnote 209

Luyten J., Verbeke E., Schokkaert E. To be or not to be: Future lives in economic evaluation. Health Econ. 2022; 31(1): 258-65. doi:https://doi.org/10.1002/hec.4454

Return to footnote 209 referrer

Footnote 210

Tinghög G. Discounting, preferences, and paternalism in cost-effectiveness analysis. Health Care Anal. 2012; 20(3): 297-318. Doi:10.1007/s10728-011-0188-6

Return to footnote 210 referrer

Footnote 211

Husereau D., Drummond M., Augustovski F., de Bekker-Grob E., Briggs A.H., Carswell C., et al. Consolidated Health Economic Evaluation Reporting Standards (CHEERS) 2022 Explanation and Elaboration: A Report of the ISPOR CHEERS II Good Practices Task Force. Value Health. 2022; 25(1): 10-31. doi:10.1016/j.jval.2021.10.008

Return to footnote 211 referrer

Footnote 212

Government of Canada. National Advisory Committee on Immunization (NACI): membership and representation. 2021. Available from: https://www.canada.ca/en/public-health/services/immunization/national-advisory-committee-on-immunization-naci/naci-membership-representation.html

Return to footnote 212 referrer

Page details

Date modified:: 2024-02-15

National Advisory Committee on Immunization (NACI): Guidelines for the economic evaluation of vaccination programs in Canada

On this page

Acknowledgements

NACI Economic guidelines task group members and liaisons

Reviewers

Contributors

In memory of Murray Krahn

Conflicts of interest declaration

Abbreviations

Glossary

Foreword: How First Nations, Inuit, and Métis perspectives can be considered in economic evaluation of vaccination programs

Economic evaluations and health technology assessments of vaccination programs

Preliminary thoughts on accounting for and incorporating Indigenous perspectives into the process of technology assessment

1) Select frameworks on social determinants of health

2) Inuit perspectives on health and well-being

Implications for Canadian economic guidelines of vaccination programs

Introduction

Development process

Overview and structure of the guidelines

Motivation for two reference cases

A general note to decision-makers in consideration of two perspectives

Guideline statements

Guidelines in detail

1. Decision problem

2. Types of evaluations

3. Study populations

4. Comparators

5. Perspectives

6. Time horizon

7. Discounting

8. Modelling

Model structure and attributes

Dynamic versus static models

Other attributes

Deterministic versus stochastic

Aggregate versus individual-based

Open versus closed

Discrete versus continuous time

Model calibration

Model validation

9. Effectiveness

Assessing estimates of vaccine effectiveness

Data synthesis

Surrogate outcomes

Extrapolation

10. Measurement and valuation of health

Health utility data

Quality-adjusted life years in societal perspective economic evaluations

11. Resource use and costs

Health system costs

Healthcare costs

Formal caregiving

Future healthcare costs

Public health costs

Healthcare costs not funded by the health system

Non-healthcare system costs

Direct out-of-pocket costs

Losses in productivity

Paid and unpaid work

Macroeconomic consequences

Non-medical consumption

Education

Social services and community services

Environment

Other areas

Transfer costs

12. Analysis

13. Uncertainty

Parameter uncertainty

Structural uncertainty

Methodological uncertainty

Value-of-information

14. Equity

15. Reporting

Appendix 1: Impact inventory table

Appendix 2: Reference case specifications

References for foreword

References

Page details