Process for incorporating economic evidence into federal vaccine recommendations: National Advisory Committee on Immunization (NACI)
February 2022
This document describes the process for incorporating economic evidence into federal vaccine recommendations. The supporting tools that accompany this process can be accessed through the hyperlinks found under the Economic process in detail section of this document, as well as through the navigation bar.
The following items referred to in the third step of the Economic Process are available online:
On this page
- Process for incorporating economic evidence into federal vaccine recommendations
- Types of economic evidence
- NACI workplan
- Economic process in detail
- Timelines
- Closing remarks
- Abbreviations
- References
- Version history
Process for incorporating economic evidence into federal vaccine recommendations
Preamble
The National Advisory Committee on Immunization (NACI) provides the Public Health Agency of Canada (PHAC) with ongoing and timely medical, scientific, and public health advice related to immunization. Through the work of working groups and with the support of a secretariat at the Public Health Agency of Canada, NACI makes recommendations on the use of human vaccines that are currently or newly approved in Canada.
Traditionally, NACI reviewed safety, efficacy, immunogenicity, effectiveness and burden of illness. PHAC has recently expanded this mandate to include consideration of programmatic factors—economics, ethics, equity, feasibility and vaccine acceptability—in developing evidence-based recommendations. This expanded mandate is to facilitate timely decision-making for publicly funded vaccine programs at provincial and territorial levels. NACI is continuing to refine methodological approaches to include these programmatic factors. NACI statements will include varying degrees of programmatic analyses for public health programs.
The NACI Economic Process is a document that outlines when and how NACI incorporates economic evidence for vaccine recommendation. This work was informed by processes and information from Canadian and international health technology assessment agencies, the Vaccine Industry Committee (VIC), and other national immunization technical advisory groups (NITAGs) from several countries. From March to May 2021, there was a public consultation on the NACI Economic Process and its supporting tools to obtain feedback from stakeholders, including industry, patient advocacy groups, the Canadian Immunization Committee (CIC), NITAGs, among others. From September to December 2021, there was a targeted consultation with select provinces and territories to obtain feedback on the changes to the NACI Economic Process based on the public consultation. The final version here represents the input from both consultations.
Types of economic evidence
Three common types of economic evidence are (full) economic evaluations, budget impact analyses, and systematic reviews (of economic evaluations) (Figure 1).
Based on feedback from the provinces and territories (P/Ts) the NACI will bring considerations of cost-effectiveness into NACI guidance. Discussions about budget impact analyses will be re-visited by NACI at a later date. Therefore, in the remainder of this document, the term "economic evidence" will not include budget impact analyses.
NACI workplan
The NACI workplan consists of projects that have been prioritized for a given fiscal year. These projects will not necessarily be completed within the fiscal year. The workplan may be reassessed based on emerging public health needs. Figure 2 outlines possible triggers that may drive activation of Working Groups (WGs) and the development of new NACI guidance documents, or the adoption of new priorities.
There are several expert WGs that draft products for NACI consideration and voting. WGs are comprised of NACI members and external experts. WGs may require the following evidence to inform NACI guidance:
- Vaccine characteristics and burden (e.g., safety, efficacy, immunogenicity, effectiveness, burden of illness)
- Ethics, Equity, Feasibility, Acceptability (EEFA)
- Economics
See Figure 3 for the NACI Decision Framework. Note that these analyses will often be done in parallel with staggered start times. They are interconnected and inform one another. For instance, economic analyses may require information from syntheses of clinical data and EEFA analyses.
Figure 4 outlines the process of determining when and how NACI incorporates economic evidence into its guidance. Please see each step in the "Economic process in detail" section.
Economic process in detail
The Economic Process consists of three basic steps:
- Assessing need
- Determining the approach and knowledge gathering
- Evidence synthesis and guidance development
Step 1. Assessing need
The first step of the Economic Process is to determine if economic evidence is needed for NACI to make a vaccine recommendation.
In theory, economic evidence should support every policy question. Nonetheless, during this period as NACI transitions into its expanded mandate, there are capacity constraints on how many systematic reviews, de novo/ adapted models, and multi-model comparisons can be realistically produced each year. Therefore, in practice, prioritization of policy questions is necessary to determine which policy questions require economic evidence in the immediate future and which can be deferred. Prioritization for economic evidence will support timeliness and quality of vaccine recommendations by ensuring the appropriate resources are allocated to policy questions where cost-effectiveness will be a major decision determinant in the recommendation.
Step 1a is to determine stakeholder need for economic evidence. This is done via the annual workplan prioritization with NACI and CIC members. As part of the survey, NACI and CIC members rank the need for economic evidence for each workplan item as high, moderate, or low need. CIC is well-positioned to relay demands from other stakeholders including the public, clinicians, and industry.
For workplan items where the need for economic evidence is ranked as high or moderate, proceed to Step 1b. Step 1b is to complete the Prioritization guide for assessing the need for economic evidence. The purpose of the guide is to structure deliberative discussions. The criteria are: (i) burden of disease, and (ii) proposed benefit of the vaccine/ vaccine program. There is no quantitative rating scale or weighting of the criteria assessed. The qualitative nature allows for context-specific assessments. Given the variation across vaccine-preventable diseases (VPDs), it is not feasible to have quantitative thresholds for each criterion (e.g., definitions of "high" prevalence differ across VPDs) or to have quantitative thresholds for an overall score. Please refer to the accompanying user guide that explains the criteria and assessments. The guide was adapted from Ontario Health's Health Technology Assessments Topic Prioritization GuideFootnote 1 and revised to meet the specific needs of vaccine evaluations. Step 1b is completed through consensus by the Working Group Chair/ Vice Chair, the NACI Secretariat (including at least one epidemiologist/ medical specialist, and one health economist), and, if needed, additional Working Group subject matter experts. In cases where a WG has not been established, the deliberation will include NACI member(s) with subject matter expertise. Upon completion of the table, users will (i) recommend the policy question to be prioritized for economic evidence, (ii) recommend the policy question to be deferred for economic evidence, or (iii) not recommend the policy question to be prioritized for economic evidence. (Note that the Secretariat will reassess this process of prioritizing policy questions for economic evidence. The Secretariat will collect feedback on an ongoing basis from NACI, CIC, and other users, and will conduct a formal reassessment after several cycles of prioritization.)
Once the shortlist of workplan items requiring economic evidence has been decided upon, the Secretariat will present to NACI for approval. The Presentation template: Policy questions prioritized for economic evidence is available for use and outlines the rationale. Stakeholders can be informed of the finalized workplan by visiting the NACI website.
Step 2. Determining the approach and knowledge gathering
The second step of the Economic Process is to determine the approach for generating economic evidence and to gather knowledge for the workplan topic. Step 2a is for the NACI Secretariat to conduct an environmental scan, which will incorporate the peer-reviewed literature, grey literature, and expert/ stakeholder input. The NACI Secretariat will reach out to industry to inquire about their planned or existing economic evaluations. Other stakeholders include national immunization technical advisory groups, and authors of relevant studies. Step 2b is for the NACI Secretariat to develop the project plan in consultation with the WG and any potential contractors involved. This will be an iterative process.
Below are three options of "information packages" to make up the economic evidence base for NACI decision-making. The decision will be informed by the environmental scan as well as by operational considerations (i.e., timeliness of issuing NACI guidance).
- Systematic review
- Systematic review and a de novo (i.e., purpose-built) or adapted model-based economic evaluation
- Systematic review and a multi-model comparison
Note that all three options include a systematic review. In the second and third options, an additional de novo (or adapted) model, or multi-model comparison will be included as part of the evidence base, respectively. These are described below:
- Systematic review (of economic evaluations): This will include economic evaluations from the peer-reviewed literature and grey literature. Economic evaluations under development or that have yet to be published may be included in the review depending on the stage of development (e.g., studies without results will not be included). Each systematic review includes a quality appraisal of included studies as well as narrative and graphical syntheses.
- De Novo (or adapted) model-based economic evaluation: This refers to either a de novo (i.e., purpose-built) economic evaluation, or to an existing economic evaluation that will be updated or adapted for NACI's purposes.
- Based on separate consultations with NACI, P/Ts, and the PHAC Public Health Ethics Consultative Group, an adapted model will not be based on a model developed by industry (i.e., developed by industry employees or by consultants sponsored by industry).
- When determining which existing economic evaluation to update or adapt, many factors will be considered including: (i) ability to engage with the authors (i.e., willingness of authors to update/ adapt their work for NACI's needs); (ii) relevance of the economic evaluation to NACI needs in terms of the population, intervention, comparator, outcome, applicability (i.e., jurisdiction, recency, data sources); (iii) study quality; (iv) source(s) of study funding and the role of funders.
- Multi-model comparison: This refers to the comparison of two or more model-based economic evaluations. Model structures, inputs, assumptions and results will be assessed and compared.
- One of the economic evaluations in a multi-model comparison will be a de novo/ adapted model.
- The other economic evaluation(s) included may be developed/ funded by others such as academia, government, a recognized funding agency, and industry.
Step 3. Evidence synthesis and guidance development
The third step of the Economic Process is to synthesize the evidence and develop NACI guidance. In Step 3a, the economic evidence will be generated or synthesized by the Secretariat or any contractors involved, in consultation with the WG. This will be an iterative process. Step 3b is the development of the economic report for inclusion into NACI guidance. This again will be completed by the Secretariat or any contractors involved, in consultation with the WG, in an iterative manner.
The following tools have been developed by NACI for conducting, reporting and presenting both economic evaluations and systematic reviews:
- Guidelines for the economic evaluation of vaccination programs in Canada: The document informs best practices for conducting and reporting economic evaluations of vaccines in Canada. This is to ensure the economic information is standardized, credible, and relevant for decision-makers in Canada's publicly funded health care system. As of early 2022, the draft guidelines are available for public consultation.
- Guidelines for systematic reviews of economic evaluations of vaccination programs: This document provides guidance on conducting and reporting a systematic review of economic evaluations for NACI.
- Guidelines for reporting model-based economic evaluations of vaccination programs in Canada: This document provides a template for reporting economic evaluations of vaccines in Canada for NACI.
- Presentation template for presenting economic evaluations: The template is to ensure economic evaluations presented to WGs and NACI are standardized, understandable, and of the highest quality.
- Presentation template for presenting systematic reviews: The template is to ensure systematic reviews presented to WGs and NACI are standardized, understandable, and of the highest quality.
The Secretariat will engage with any stakeholders where needed. For instance, relevant manufacturers will be invited to submit their economic evaluations in the event that NACI seeks to conduct a multi-model comparison. If stakeholders choose to submit to NACI, please refer to the Submission criteria for model-based economic evaluations. This document outlines the mandatory standards, including how to conduct and report the economic evaluation, software and run time requirements, as well as the submission of the model code and instructions on how to run the model.
Timelines
Figure 5 shows the timeline of the steps outlined above. The NACI Secretariat projects a 2-year time horizon when planning and conducting economic analyses.
December: Prioritization of NACI policy questions for economic evidence (steps 1a and 1b) should begin every December. This coincides with NACI's biennial Vaccine Pipeline Meeting, where industry presents vaccine products under development, prior to Health Canada submission. The timing of this exercise approximately translates to having more than 1-2 years prior to Health Canada's Notice of Compliance to plan and conduct any economic analyses for new products and new indications. For other policy questions (i.e., not new products or new indications), the amount of time to plan and conduct analyses will depend on when the policy question arises.
February: NACI approves its workplan every February. The Secretariat will provide to NACI the subset of workplan items prioritized for economic evidence for their approval.
Spring: The approved NACI workplan, along with the subset of workplan items that will include economic evidence, will be publicly available and posted on the NACI website.
Year 1: The Secretariat will conduct environmental scans, draft contracts where needed, connect with stakeholders (including industry, other national immunization technical advisory groups, and authors of relevant studies), and develop the project plan (steps 2a and 2b).
Year 2: The work will be executed if priorities do not change (i.e., due to public health emergencies). An economic report will be generated for inclusion into NACI guidance (steps 3a and 3b).
Closing remarks
In summary, economic evidence may be one component of NACI guidance. Other decision determinants include but are not limited to safety, efficacy, immunogenicity, effectiveness, burden of illness, as well as EEFA considerations. NACI guidance will be published on NACI's website with notifications to stakeholders, web subscribers, and typically as a notification in the Canada Communicable Disease Report (CCDR).
Abbreviations
- CCDR
- Canada Communicable Disease Report
- CIC
- Canadian Immunization Committee
- EFFA
- Ethics, Equity, Feasibility, Acceptability
- NACI
- National Advisory Committee on Immunization
- NITAG
- National Immunization Technical Advisory Groups
- PHAC
- Public Health Agency of Canada
- P/Ts
- Provinces and Territories
- VIC
- Vaccine Industry Committee
- VPD
- Vaccine-preventable diseases
- WG
- Working Group
References
Version | Date | Changes made |
---|---|---|
V0.0 | Jan 2020 | Draft approved by Economics Task Group |
V0.1 | Feb 2020 | Draft approved by NACI |
V0.2 | Sep 2021 | Feedback incorporated from public consultation (March – May 2021) |
V0.3 | Dec 2021 | Feedback incorporated from second round of consultations with PTs (September – December 2021) |
V1.0 | Feb 2022 | Approved for use by NACI |
Page details
- Date modified: