Guidelines for reporting model-based economic evaluations of vaccination programs in Canada: National Advisory Committee on Immunization (NACI)

February 2022

• Previous
• Next

See in PDF format the Guidelines for reporting model-based economic evaluations of vaccination programs in Canada.

On this page

• Guidelines for reporting model-based economic evaluations of vaccination programs in Canada
  • Purpose
  • Template
• Abbreviations
• References
• Version history

Guidelines for reporting model-based economic evaluations of vaccination programs in Canada

Purpose

A structured reporting format for the preparation of economic evaluations ensures that studies are thoroughly presented and organized consistently to facilitate review and comparison by decision-makers.

Model-based economic evaluations of vaccination programs in Canada prepared for the National Advisory Committee on Immunization (NACI) should follow this format as much as possible. In some instances, deviation from the format may be appropriate. For example, the report sections could be reordered or certain sections excluded if they are irrelevant to the evaluation. The study should be presented in a clear and transparent manner, with enough information provided to enable the audience to critically evaluate the validity of the analysis. The Executive Summary and Conclusion sections should be written so they can be understood by a non-technical reader.

This template is based an environmental scan of guidance on how to report economic evaluations. The template aligns with the many existing guidelines, including the Consolidated Health Economic Evaluation Reporting Standards II (CHEERS II),^Footnote 1 which is primarily for reporting on studies for peer-reviewed journals. Here, the template lists elements required for reports submitted to NACI.

Template

Title

• Identify the study as an economic evaluation (specifying the analytical technique used, e.g., cost-utility, cost-benefit analysis)
• Identify the interventions being compared

Preface

• List of authors, affiliations, and a description of contributions
• Acknowledgements
• Disclosure of potential conflicts of interest over the last five years for each author. These include: participation in research, equity ownership, intellectual property, and any other interest that readers may perceive as a competing interest (e.g., public statements about the topic). If there are no potential conflicts of interest, a statement to that effect must be included.
•  Disclosure of how the study was funded and the role of the funder in the identification, design, conduct, and reporting of the analysis. Describe other non-monetary sources.

Executive summary

The Executive Summary should be no more than three pages long and written in non-technical language. It should include the following sections:

• Issue: a statement about the policy or economic issue, or reason for evaluating the vaccination program
• Objectives and Decision Problem
• Methods
• Results: a numerical and narrative summary of the findings
• Discussion: study limitations, relevance of findings, health services impact
• Conclusions: state the bottom-line findings of the evaluation, uncertainty about the results, and caveats

Table of contents

Abbreviations

Glossary

Objectives

• Description of issue(s) addressed in the model-based economic evaluation. Set the scene for the reader, and include reasons for the analysis (e.g., policy issues, funding or costs implications, issues of competing interventions).

Statement of decision problem

• Define the decision problem, state it in an answerable form, and make it relevant for the target audience. For example, "What is the cost-effectiveness of routinely vaccinating population X against Disease Y, using vaccine A compared to the current approach of B?"
• Define the study population(s) for the vaccination program and comparators. Also define the population at risk for the disease of interest and any populations that may be indirectly affected by the vaccination program, either through externalities or spillover effects.
• Specify the interventions to be compared, the setting(s) in which they are to be compared, and the time horizon.
• State the perspectives of the reference cases and any non-reference cases, if applicable.

Background

Condition:

• State the condition and population group(s) being studied.
• List the etiology, pathology, diagnosis, risk factors, and prognosis (if relevant).
• Describe the epidemiological (i.e., incidence or prevalence) burden of the infection/ disease in Canada.
• Describe the economic impact and burden of the infection/ disease in Canada.
• Describe the current clinical practice for prevention and treatment of the disease in Canada. Refer to clinical practice guidelines (if relevant). Include a description of the selected comparators.

Vaccination program:

• State the brand and generic names of the vaccine, dosage of vaccine, the number of doses required, dose schedule, whether any booster doses are required, expected dose completion, handling of vaccine wastage, assumptions on waning, and coverage estimates.
• State the adverse events, contraindications, cautions, and warnings.
• Describe potential setting(s) for delivery of the vaccine (e.g., schools, public health clinics, physicians' offices, pharmacies)
• Give the unit costs of the vaccination doses and comparator(s). Provide other intervention-related costs including distribution costs such as for transportation and cold storage, and administration costs such as for personnel time, vaccine wastage, and ancillary supplies.

Regulatory status:

• List the approved indication(s) in Canada that are the topic of the study, including applicable population and subgroups, and date of approval.
• List any additional approved indication(s) in Canada.
• Include the regulatory status and approved indications in other countries.

Review of economic evidence

• Provide a narrative account of existing economic studies that evaluate the same vaccination program and similar decision problem. (A systematic review is not required in this report, as NACI conducts separate systematic reviews on economic evaluations).
• Where validated economic models have been developed that address similar decision problems, indicate whether the approach can be used to address the current decision problem.

Methods

As outlined in these Guidelines, report how each element of the economic evaluation has been handled.

Health economic analysis plan:

• Indicate whether a health economic plan was developed and where available.

Type of economic evaluation:

• Describe the cost-utility analyses (CUAs) used in the reference cases.
• If applicable, describe other analytic techniques (e.g., cost-benefit analysis) used alongside the reference cases.

Study populations:

• Describe the intended population(s) for the vaccination program, the population at risk for the disease of interest, and any populations that may be indirectly affected by the vaccination program, either through externalities or spillover effects
• Characterize heterogeneity. State whether there are a priori identifiable subgroups for which differential results might be expected (e.g., demographic factors, behavioural factors, disease-related factors, and effectiveness of the vaccine or comparator(s)). Describe and justify the subgroups. Describe any methods used for estimating how the results of the study vary for subgroups.
• If no subgroup analyses were conducted, provide justification for why they were not required.

Comparators:

• Describe and justify comparators that were selected or not selected. Relate choice of comparators to the study population, and the local epidemiologic data, disease patterns, and current approaches to the prevention and treatment of the infection/ disease of interest.

Perspective:

• State the perspectives used in the analysis (publicly funded health system and societal at minimum).
• Describe how other types of variability (e.g., epidemiologic variation) were analyzed, and provide justification.
• Complete the NACI Impact inventory table for economic evaluations of vaccination strategies and include in the appendix.

Time horizon:

• Indicate the time horizon(s) used in the analysis, and provide justification. Where applicable, state the different points during the model time horizon from which results are reported.

Discount rate:

• Indicate the discount rates used for costs and outcomes, and provide justification.

Modelling:

1. Modelling considerations
   • Describe the model structure: description of the scope, structure, and assumptions made (with justification). Provide a schematic of the model. Inclusion of the model code to NACI is recommended. Report if the model is publicly available and where it can be accessed.
   • Specify the study design (e.g., decision tree, Markov cohort model).
   • Describe how the model was validated. This can involve different methods (e.g., face validity, internal validity, cross-validation, external validity, predictively validity). Results from validation exercises can be attached as appendices.
   • Justify the choice of a static model versus a dynamic model.
   • If a dynamic model is used, report details of contact patterns, differential equations if applicable, rules underpinning agent behaviours if applicable, and transmission dynamics over the model time horizon. Provide a schematic diagram with suitable annotation in the appendix, without the use of mathematical notation.

2. Data considerations
   • List data sources and justify assumptions. This may include details about epidemiological factors, such as prevalence or incidence of the infection/ disease.
   • Describe any statistical analyses.
   • Describe calibration methods used to generate parameter estimates, where applicable.

Effectiveness:

1. Evidence of efficacy and effectiveness
   • Give details about the evidence on efficacy and effectiveness used in the analysis, and the relationship between these estimates (if lengthy, place in an appendix).
   • For clinical studies, report on PICO(TS): population, intervention, comparator(s), outcome(s) of interest; and where relevant, timing, type of study, and setting.
   • Describe adverse events, where important and relevant.
   • Report the sources of information (e.g., trials, a meta-analysis or network meta-analysis, observational studies, surveillance data), assumptions, and justification.

2. Modelling effectiveness
   • Identify factors that are likely to have an impact on effectiveness (e.g., vaccine series completion, community immunity, host factors, coverage), and describe how these were factored into the analysis. Explain causal relationships and techniques used to model or extrapolate parameter estimates (e.g., waning of vaccine protection, surrogate to final outcomes). Describe the strength of the evidence for the relationships and links.

Measurement and valuation of outcomes:

• Identify, measure, and value all relevant outcomes, including important adverse events, for each intervention and for each study population (e.g., population intended for vaccination, population at risk of the disease, and any populations affected through externalities or spillover effects).
• Indicate the health-related quality-of-life measurement used (e.g., name of instrument, version), and include justification. A copy of the instrument may be included in an appendix. Describe the methods for eliciting preferences (e.g., type of stated preference survey, format and frequency of administration of instrument), and the population measured (e.g., size, demographic characteristics, proxies used).
• Include other outcomes that were considered but rejected (with rationale).
• Report the sources of information, assumptions, and justification.

Resource use and costs:

• Identify, measure, and value all resources included in the analysis.
• Report the costing methods used (e.g., micro-costing versus gross-costing, top-down versus bottom-up).
• Classify resources into categories relevant to the perspective (e.g., sectors considered in the societal perspective analysis).
• Report resource quantities and unit costs separately.
• For the societal perspective analysis, report the method used for valuing productivity losses, consumption, and costs and outcomes of non-health sectors. Identify, measure, and value these impacts.
• Report the sources of information, assumptions, and justification.
• Report the currency, price date, method of price adjustment, and method of currency conversion.

Uncertainty:

• Identify sources of uncertainty in the analysis.
• Clearly delineate the reference case analysis from non-reference case analyses.
• Provide sources and justification for the probability distributions used in probabilistic analyses. State the number of Monte Carlo iterations.
• Justify ranges of values used in deterministic uncertainty analyses.
• For scenario analyses, state the values and assumptions tested. Provide sources and justification for each.

Equity:

• State equity assumptions (e.g., all quality-adjusted life-years [QALYs] are equal).
• State any equity objectives that were considered in the analysis (e.g., increasing vaccine uptake in specific under-immunized groups).
• Characterize distributional effects (e.g., how impacts are distributed across different subgroups). Identify the equity-relevant characteristics of the main subgroups that may benefit, or be adversely affected by, the vaccination program, and describe how they were analyzed.

The input values from above (e.g., Effectiveness, Measurement and Valuation of Outcomes, Resource Use and Costs) can be summarized as a table. The table should present the point estimates, range, probability distributions, references to the sources of information, and key assumptions. A table with key parameters should appear in the main body of the report and a complete table should appear in the appendix.

Results

Analysis and results:

• Present all analyses in a step-by-step fashion so the calculations can be replicated, if desired. This includes outcomes and costs by comparator.
• Show the goodness-of-fit of model-projected outcomes (e.g. infections, symptomatic cases) to data.
• Report transmission dynamics over the model time horizon, including data on infection, disease incidence and prevalence, and when applicable, changes in other relevant outcomes such serotype or strain replacement and development of antimicrobial resistance.
• Present the analysis first in a disaggregated fashion, by types of outcomes and costs. If relevant, show separately the analysis of different time horizons and types of economic evaluations performed.
• For outcomes, express in terms of infections, healthcare utilization units (e.g. hospitalizations), and deaths first, then translate into alternative units such as QALYs or monetary benefits. Provide any standardized case definitions for clinical outcomes.
• Show undiscounted totals (gross and net) before aggregation and discounting.
• Report the overall analysis that includes all affected populations over the entire time horizon. Report the incremental cost-effectiveness ratios (ICERs), the numerator (mean costs of each intervention), and denominator (mean outcomes of each intervention) for each perspective. Report the 95% confidence intervals or credible intervals for the incremental estimates and ICERs (or other indicators of precision if the distribution of uncertain outcomes is not approximately Gaussian). Avoid presenting negative ICERs when an intervention is either dominant or dominated.
• For estimates of net health/ net monetary benefit, provide the cost-effectiveness threshold and its source.
• Present results stratified by subgroup in the same manner.
• Present results graphically such as through health production functions, cost-effectiveness efficacy frontiers (CEEFs).

Results of uncertainty analyses:

• Present uncertainty of results graphically such as through cost-effectiveness plane scatter plots, tornado diagrams, cost-effectiveness acceptability curves (CEACs), and cost-effectiveness acceptability frontiers (CEAFs).
• Report the results for sensitivity and scenario analyses.
• Report any variables that are influential to the results of economic evaluations.
• Describe the interpretation of the results in relation to the reference cases.
• Indicate the results of analyses for types of variability (e.g., variation in costs or epidemiologic patterns).

Model validation:

• Provide details on the process for validating the model.
• Where details of the exercise are relevant for inclusion, consider including this as an appendix to the economic evaluation.
• Where other economic studies have been reviewed, compare the methods and results of these studies with the present study.

Discussion

Summary of results:

• Critically appraise and interpret the main findings of the analysis in the context of all reasonable interventions.
• Address the intervention's place in practice, based on the evidence.
• Discuss the uncertainty of the results and the key drivers of results.
• Discuss the trade-off between outcomes and costs.
• Compare results to other studies, including a fulsome discussion on similarities, differences, and significance.

Study limitations:

• Discuss key limitations and issues concerning the analysis, including methodological limitations and issues, validity of assumptions, strength of the data, and relationships or links used in the model.
• Describe whether the data and methods used may bias the analysis in favour of any intervention.

Generalizability:

• Comment on the generalizability or relevance of results, and the validity of the data and model for the relevant jurisdictions and populations.
• Comment on regional differences in terms of disease epidemiology, population characteristics, clinical practice patterns, resource use patterns, unit costs, and other factors of relevance. Where differences exist, discuss the impact on the results (expected direction and magnitude), and the conclusions.

Equity considerations:

• Indicate the distributional considerations (e.g., intergenerational effects of the vaccination program).
• List other ethical and equity implications or issues; for example, are there likely to be variations in access to the vaccination program due to geographic location or population characteristics? Does the vaccination program address the unmet needs of certain disadvantaged groups (e.g., racialized groups)? Is the vaccination program responsive to those with greatest need?

Future research:

• Identify knowledge gaps and areas for further research that are relevant to Canada.

Conclusions

• Address the decision problem(s).
• Summarize the main findings of the study: aggregate impact, uncertainty about the results, appropriate uses for the vaccination program (e.g., population subgroups at higher risk of infection), and any caveats.

References

Appendices

• Depending on practical considerations and amount of material, include the following in the appendices: completed impact inventory table; a table of data sources; data collection forms, questionnaires, and instruments; step-by-step details of analyses, including intermediate results; results of model validation; comparisons of model-projected outcomes to data for goodness-of-fit; tables of results; and visual presentations of results (e.g., figures, graphs).

Abbreviations

CBA

Cost-benefit analysis

CEAC

Cost-effectiveness acceptability curve

CEAF

Cost-effectiveness acceptability frontier

CEEF

Cost-effectiveness efficacy frontier

CHEERS II

Consolidated Health Economic Evaluation Reporting Standards II

CUA

Cost-utility analysis

ICER

Incremental cost-effectiveness ratio

NACI

National Advisory Committee on Immunization

PICO(TS)

Population, intervention, comparator(s), outcome(s) of interest; and where relevant, timing, type of study, and setting

QALY

Quality-adjusted life year

• Previous
• Next