Process for incorporating economic evidence into federal vaccine recommendations: National Advisory Committee on Immunization (NACI)

February 2022

• Previous
• Next

See in PDF format the Process for incorporating economic evidence into federal vaccine recommendations.

On this page

• Process for incorporating economic evidence into federal vaccine recommendations
  • Preamble
• Types of economic evidence
  • Figure 1: Types of economic evidence commonly used in decision making
• NACI workplan
  • Figure 2: Triggers that may determine the NACI workplan
  • Figure 3: NACI decision framework
  • Figure 4:Overview of the NACI economic process
• Economic process in detail
  • Step 1: Assessing need
  • Step 2: Determining the approach and knowledge gathering
  • Step 3: Evidence  synthesis and guidance development
• Timelines
  • Figure 5: Two-year timeline of NACI economic process
• Closing remarks
• Abbreviations
• References
• Version history

Process for incorporating economic evidence into federal vaccine recommendations

Preamble

The National Advisory Committee on Immunization (NACI) provides the Public Health Agency of Canada (PHAC) with ongoing and timely medical, scientific, and public health advice related to immunization. Through the work of working groups and with the support of a secretariat at the Public Health Agency of Canada, NACI makes recommendations on the use of human vaccines that are currently or newly approved in Canada.

Traditionally, NACI reviewed safety, efficacy, immunogenicity, effectiveness and burden of illness. PHAC has recently expanded this mandate to include consideration of programmatic factors—economics, ethics, equity, feasibility and vaccine acceptability—in developing evidence-based recommendations. This expanded mandate is to facilitate timely decision-making for publicly funded vaccine programs at provincial and territorial levels. NACI is continuing to refine methodological approaches to include these programmatic factors. NACI statements will include varying degrees of programmatic analyses for public health programs.

The NACI Economic Process is a document that outlines when and how NACI incorporates economic evidence for vaccine recommendation. This work was informed by processes and information from Canadian and international health technology assessment agencies, the Vaccine Industry Committee (VIC), and other national immunization technical advisory groups (NITAGs) from several countries. From March to May 2021, there was a public consultation on the NACI Economic Process and its supporting tools to obtain feedback from stakeholders, including industry, patient advocacy groups, the Canadian Immunization Committee (CIC), NITAGs, among others. From September to December 2021, there was a targeted consultation with select provinces and territories to obtain feedback on the changes to the NACI Economic Process based on the public consultation. The final version here represents the input from both consultations.

Types of economic evidence

Three common types of economic evidence are (full) economic evaluations, budget impact analyses, and systematic reviews (of economic evaluations) (Figure 1).

Figure 1 - Text Description

This figure presents two types of economic evidence commonly used in decision-making.

1. Economic Evaluation - Is this cost-effective?
   • Includes cost-utility, cost-effectiveness, cost-benefit analyses, etc.
   • Analysis of health outcomes and costs of alternative healthcare interventions
   • Examines value for money
2. Examines value for money
   • Analysis of the likely change in expenditure to a specific budget holder resulting from a decision to reimburse a new healthcare intervention (or some other change in policy) at an aggregate population level
   • Examines budget

Based on feedback from the provinces and territories (P/Ts) the NACI will bring considerations of cost-effectiveness into NACI guidance. Discussions about budget impact analyses will be re-visited by NACI at a later date. Therefore, in the remainder of this document, the term "economic evidence" will not include budget impact analyses.

NACI workplan

The NACI workplan consists of projects that have been prioritized for a given fiscal year. These projects will not necessarily be completed within the fiscal year. The workplan may be reassessed based on emerging public health needs. Figure 2 outlines possible triggers that may drive activation of Working Groups (WGs) and the development of new NACI guidance documents, or the adoption of new priorities.

Figure 2 - Text Description

This figure outlines possible triggers that may drive activation of Working Groups (WGs) and the development of new NACI guidance documents, or the adoption of new priorities. Possible triggers include: a new vaccine or new indication, new recommendation by authoritative Canadian body (government or professional organization), P/T programmatic concerns, recommendation adoption by reputable international organization (ACIP, WHO), new safety signal detected, vaccine supply issues, anticipated public health emergency (e.g. EVD imported case), changes in disease epidemiology (including outbreaks), CIG chapter review, or key publication (pivotal trial). Abbreviations: ACIP, Advisory Committee on Immunization Practices (United States Centers for Disease Control and Prevention); CIG, Canadian Immunization Guide; EVD, Ebola virus disease; P/T, provinces/territories; WHO, World Health Organization

There are several expert WGs that draft products for NACI consideration and voting. WGs are comprised of NACI members and external experts. WGs may require the following evidence to inform NACI guidance:

1. Vaccine characteristics and burden (e.g., safety, efficacy, immunogenicity, effectiveness, burden of illness)
2. Ethics, Equity, Feasibility, Acceptability (EEFA)
3. Economics

See Figure 3 for the NACI Decision Framework. Note that these analyses will often be done in parallel with staggered start times. They are interconnected and inform one another. For instance, economic analyses may require information from syntheses of clinical data and EEFA analyses.

Figure 3 - Text Description

This figure depicts the key considerations that form the foundation for NACI recommendations. Key considerations include: acceptability of the immunization program, feasibility of program implementation given existing resources, cost-effectiveness of the program relative to other options, equity of access to the vaccine for all target groups who would benefit from immunization, ethics of the immunization program, safety of the vaccine in question, nature of the immune response following vaccination, vaccine effectiveness, vaccine efficacy, and burden of disease in both the general population as well as in high-risk groups.

Figure 4 outlines the process of determining when and how NACI incorporates economic evidence into its guidance. Please see each step in the "Economic process in detail" section.

Abbreviation: SR, systematic review; MMC, multi-model comparison

Figure 4 - Text Description

This figure presents a flow chart depicting the general NACI Economic Process. The first step queries whether there is a need for an economic evaluation. If there is no need for an economic evaluation, there are no further steps in the process. The NACI Priortization Guide can be used to help guide decision-making in this step. If a need is identified, the next step is to determine the approach for knowledge gathering or otherwise generating economic evidence. The approach may take the form of a systematic review of economic evaluations, a systematic review as well as a model, or a systematic review accompanied by a multi-model comparison. The approach may be reassessed based on working group (WG) feedback. The last step is an iterative process of evidence synthesis and guidance development. Resources relevant to this step of the process include NACI's Submission Criteria for Economic Evaluations, Economic Evaluation Guidelines, Systematic Review Guidelines, Reporting Guidelines for Economic Evaluations, Presentation Template for Economic Evaluations, and Presentation Template for Systematic Reviews. Abbreviations: WG, working group

Economic process in detail

The Economic Process consists of three basic steps:

• Assessing need
• Determining the approach and knowledge gathering
• Evidence synthesis and guidance development

Step 1. Assessing need

The first step of the Economic Process is to determine if economic evidence is needed for NACI to make a vaccine recommendation.

In theory, economic evidence should support every policy question. Nonetheless, during this period as NACI transitions into its expanded mandate, there are capacity constraints on how many systematic reviews, de novo/ adapted models, and multi-model comparisons can be realistically produced each year. Therefore, in practice, prioritization of policy questions is necessary to determine which policy questions require economic evidence in the immediate future and which can be deferred. Prioritization for economic evidence will support timeliness and quality of vaccine recommendations by ensuring the appropriate resources are allocated to policy questions where cost-effectiveness will be a major decision determinant in the recommendation.

Step 1a is to determine stakeholder need for economic evidence. This is done via the annual workplan prioritization with NACI and CIC members. As part of the survey, NACI and CIC members rank the need for economic evidence for each workplan item as high, moderate, or low need. CIC is well-positioned to relay demands from other stakeholders including the public, clinicians, and industry.

For workplan items where the need for economic evidence is ranked as high or moderate, proceed to Step 1b. Step 1b is to complete the Prioritization guide for assessing the need for economic evidence. The purpose of the guide is to structure deliberative discussions. The criteria are: (i) burden of disease, and (ii) proposed benefit of the vaccine/ vaccine program. There is no quantitative rating scale or weighting of the criteria assessed. The qualitative nature allows for context-specific assessments. Given the variation across vaccine-preventable diseases (VPDs), it is not feasible to have quantitative thresholds for each criterion (e.g., definitions of "high" prevalence differ across VPDs) or to have quantitative thresholds for an overall score. Please refer to the accompanying user guide that explains the criteria and assessments. The guide was adapted from Ontario Health's Health Technology Assessments Topic Prioritization Guide^Footnote 1 and revised to meet the specific needs of vaccine evaluations. Step 1b is completed through consensus by the Working Group Chair/ Vice Chair, the NACI Secretariat (including at least one epidemiologist/ medical specialist, and one health economist), and, if needed, additional Working Group subject matter experts. In cases where a WG has not been established, the deliberation will include NACI member(s) with subject matter expertise. Upon completion of the table, users will (i) recommend the policy question to be prioritized for economic evidence, (ii) recommend the policy question to be deferred for economic evidence, or (iii) not recommend the policy question to be prioritized for economic evidence. (Note that the Secretariat will reassess this process of prioritizing policy questions for economic evidence. The Secretariat will collect feedback on an ongoing basis from NACI, CIC, and other users, and will conduct a formal reassessment after several cycles of prioritization.)

Once the shortlist of workplan items requiring economic evidence has been decided upon, the Secretariat will present to NACI for approval. The Presentation template: Policy questions prioritized for economic evidence is available for use and outlines the rationale. Stakeholders can be informed of the finalized workplan by visiting the NACI website.

Step 2. Determining the approach and knowledge gathering

The second step of the Economic Process is to determine the approach for generating economic evidence and to gather knowledge for the workplan topic. Step 2a is for the NACI Secretariat to conduct an environmental scan, which will incorporate the peer-reviewed literature, grey literature, and expert/ stakeholder input. The NACI Secretariat will reach out to industry to inquire about their planned or existing economic evaluations. Other stakeholders include national immunization technical advisory groups, and authors of relevant studies. Step 2b is for the NACI Secretariat to develop the project plan in consultation with the WG and any potential contractors involved. This will be an iterative process.

Below are three options of "information packages" to make up the economic evidence base for NACI decision-making. The decision will be informed by the environmental scan as well as by operational considerations (i.e., timeliness of issuing NACI guidance).

• Systematic review
• Systematic review and a de novo (i.e., purpose-built) or adapted model-based economic evaluation
• Systematic review and a multi-model comparison

Note that all three options include a systematic review. In the second and third options, an additional de novo (or adapted) model, or multi-model comparison will be included as part of the evidence base, respectively. These are described below:

• Systematic review (of economic evaluations): This will include economic evaluations from the peer-reviewed literature and grey literature. Economic evaluations under development or that have yet to be published may be included in the review depending on the stage of development (e.g., studies without results will not be included). Each systematic review includes a quality appraisal of included studies as well as narrative and graphical syntheses.
• De Novo (or adapted) model-based economic evaluation: This refers to either a de novo (i.e., purpose-built) economic evaluation, or to an existing economic evaluation that will be updated or adapted for NACI's purposes.
  • Based on separate consultations with NACI, P/Ts, and the PHAC Public Health Ethics Consultative Group, an adapted model will not be based on a model developed by industry (i.e., developed by industry employees or by consultants sponsored by industry).
  • When determining which existing economic evaluation to update or adapt, many factors will be considered including: (i) ability to engage with the authors (i.e., willingness of authors to update/ adapt their work for NACI's needs); (ii) relevance of the economic evaluation to NACI needs in terms of the population, intervention, comparator, outcome, applicability (i.e., jurisdiction, recency, data sources); (iii) study quality; (iv) source(s) of study funding and the role of funders.
• Multi-model comparison: This refers to the comparison of two or more model-based economic evaluations. Model structures, inputs, assumptions and results will be assessed and compared.
  • One of the economic evaluations in a multi-model comparison will be a de novo/ adapted model.
  • The other economic evaluation(s) included may be developed/ funded by others such as academia, government, a recognized funding agency, and industry.

Step 3. Evidence synthesis and guidance development

The third step of the Economic Process is to synthesize the evidence and develop NACI guidance. In Step 3a, the economic evidence will be generated or synthesized by the Secretariat or any contractors involved, in consultation with the WG. This will be an iterative process. Step 3b is the development of the economic report for inclusion into NACI guidance. This again will be completed by the Secretariat or any contractors involved, in consultation with the WG, in an iterative manner.

The following tools have been developed by NACI for conducting, reporting and presenting both economic evaluations and systematic reviews:

• Guidelines for the economic evaluation of vaccination programs in Canada: The document informs best practices for conducting and reporting economic evaluations of vaccines in Canada. This is to ensure the economic information is standardized, credible, and relevant for decision-makers in Canada's publicly funded health care system. As of early 2022, the draft guidelines are available for public consultation.
• Guidelines for systematic reviews of economic evaluations of vaccination programs: This document provides guidance on conducting and reporting a systematic review of economic evaluations for NACI.
• Guidelines for reporting model-based economic evaluations of vaccination programs in Canada: This document provides a template for reporting economic evaluations of vaccines in Canada for NACI.
• Presentation template for presenting economic evaluations: The template is to ensure economic evaluations presented to WGs and NACI are standardized, understandable, and of the highest quality.
• Presentation template for presenting systematic reviews: The template is to ensure systematic reviews presented to WGs and NACI are standardized, understandable, and of the highest quality.

The Secretariat will engage with any stakeholders where needed. For instance, relevant manufacturers will be invited to submit their economic evaluations in the event that NACI seeks to conduct a multi-model comparison. If stakeholders choose to submit to NACI, please refer to the Submission criteria for model-based economic evaluations. This document outlines the mandatory standards, including how to conduct and report the economic evaluation, software and run time requirements, as well as the submission of the model code and instructions on how to run the model.

Timelines

Figure 5 shows the timeline of the steps outlined above. The NACI Secretariat projects a 2-year time horizon when planning and conducting economic analyses.

December: Prioritization of NACI policy questions for economic evidence (steps 1a and 1b) should begin every December. This coincides with NACI's biennial Vaccine Pipeline Meeting, where industry presents vaccine products under development, prior to Health Canada submission. The timing of this exercise approximately translates to having more than 1-2 years prior to Health Canada's Notice of Compliance to plan and conduct any economic analyses for new products and new indications. For other policy questions (i.e., not new products or new indications), the amount of time to plan and conduct analyses will depend on when the policy question arises.

February: NACI approves its workplan every February. The Secretariat will provide to NACI the subset of workplan items prioritized for economic evidence for their approval.

Spring: The approved NACI workplan, along with the subset of workplan items that will include economic evidence, will be publicly available and posted on the NACI website.

Year 1: The Secretariat will conduct environmental scans, draft contracts where needed, connect with stakeholders (including industry, other national immunization technical advisory groups, and authors of relevant studies), and develop the project plan (steps 2a and 2b).

Year 2: The work will be executed if priorities do not change (i.e., due to public health emergencies). An economic report will be generated for inclusion into NACI guidance (steps 3a and 3b).

Abbreviations: NACI, National Advisory Committee on Immunization; CIC, Canadian Immunization Committee; WG, working group; SR, systematic review; MMC, multi-model comparison

Figure 5 - Text Description

This figure presents a two-year timeline of the general NACI Economic Process. The first step is to determine whether there is a need for an economic evaluation, and this deliberation begins in the December prior to Year 1 at which time NACI and the CIC complete an annual workplan prioritization and deliberation via the NACI Priortization Guide. NACI meets in February of Year 1 to approve workplan items prioritized for economic evidence, and the finalized workplan is made available to the public the following spring. In Year 1, the NACI Secretariat and Working Group (WG) conduct an environmental scan and decide on the most appropriate approach for knowledge gathering or generation of economic evidence. Year 2 will be spent generating economic evidence as per the chosen approach (systematic review, systematic review plus model, or systematic review plus multi-model comparison). The NACI Secretariat and WG, in consultation with stakeholders where needed, will generate an economic report for inclusion in NACI guidance.

Closing remarks

In summary, economic evidence may be one component of NACI guidance. Other decision determinants include but are not limited to safety, efficacy, immunogenicity, effectiveness, burden of illness, as well as EEFA considerations. NACI guidance will be published on NACI's website with notifications to stakeholders, web subscribers, and typically as a notification in the Canada Communicable Disease Report (CCDR).

Abbreviations

CCDR

Canada Communicable Disease Report

CIC

Canadian Immunization Committee

EFFA

Ethics, Equity, Feasibility, Acceptability

NACI

National Advisory Committee on Immunization

NITAG

National Immunization Technical Advisory Groups

PHAC

Public Health Agency of Canada

P/Ts

Provinces and Territories

VIC

Vaccine Industry Committee

VPD

Vaccine-preventable diseases

WG

Working Group

• Previous
• Next