ARCHIVED - Canada Communicable Disease Report

Volume 35 • ACS-1
January 2009

An Advisory Committee Statement (ACS)
National Advisory Committee on Immunization (NACI)^{Footnote †}

For readers interested in the PDF version, the document is available for download or viewing:

CCDR - January 2009 - Volume 35 - ACS-1 (PDF Document - 115 KB - 10 pages)

Evidence-based recommendations for immunization - Methods of the National Advisory Committee on Immunization

The National Advisory Committee on Immunization (NACI) provides the Public Health Agency of Canada with ongoing and timely medical, scientific and public health advice relating to immunization. The Public Health Agency of Canada acknowledges that the advice and recommendations set out in this statement are based upon the best current available scientific knowledge and is disseminating this document for information purposes. NACI members and liaison members conduct themselves within the context of the Public Health Agency of Canada's Policy on Conflict of Interest, including yearly declaration of potential conflict of interest.

Background

Since 1975 the National Advisory Committee on Immunization (NACI) has provided the federal Ministry of Health, currently through the Public Health Agency of Canada, with medical, scientific, and public-health advice relating to immunization. While NACI recommendations regarding vaccine use have always been based on review of available scientific evidence, the methods used to retrieve, assemble and evaluate evidence in order to make a recommendation about vaccine use have not always been explicitly stated. With the goal of increasing transparency of the recommendation process and further serving the information needs of stakeholders in the vaccine community, NACI has prepared this summary of the process used for knowledge collation and synthesis, and recommendation development. This process will be used by NACI in the development of recommendations for new vaccines, existing vaccines for which there are new indications or restrictions, or for recommendations on other aspects of the prevention of infectious disease that fall within NACI's mandate. These processes will be described briefly in the methods section of each NACI recommendation statement. In this document we review the overall process.

Considerations in the development of evidencebased vaccine recommendations

Guidelines have been defined as "systematically developed evidence-based statements which assist providers, recipients and other stakeholders to make informed decisions about appropriate health interventions^{Footnote 1}." Whether the products of these systematic processes for evidence collation and interpretation are called guidelines, protocols or recommendations, they share a common purpose of providing advice as to which health care interventions are likely to improve health outcomes. The justification for these recommendations should be visible to the user, an attribute known as transparency^{Footnote 2}. Improved transparency of NACI's recommendation process will permit health care providers to better explain to patients and clients the rationale for use of a vaccine, should facilitate translation of recommendations in different settings (e.g. provincial program planning, for an individual patient) and is expected to be more intellectually satisfying for readers who wish to review the primary data for themselves. Secondly, a standardized transparent approach by NACI for evidence review, synthesis and recommendation preparation should reduce variation due to process alone, and thus reduce perceived or real arbitrariness of decision making.

Implicit in any process for evaluation of evidence is an acceptance that not all evidence is of equal value. The recognition that evidence exists in a hierarchical fashion and is of varying quality was first highlighted over 30 years ago by the Canadian Task Force on the Periodic Health Examination^{Footnote 3}. NACI has used a grading scheme in previous statements, based on an earlier schema of the Task Force, which consisted of ranking the overall level of evidence for a recommendation, and assigning a letter grade for the strength of its recommendation. An essential component of critical appraisal^{Footnote 4}, assessment of the quality of the evidence, was not explicitly considered in that NACI rating scheme. Strengths and weaknesses in the design and execution of individual studies should be evaluated since, for example, a poorly executed randomized controlled trial may provide weaker evidence than an excellent quality observational study. In general, the most common study design used to support vaccine efficacy is the randomized controlled trial (RCT), and since many NACI statements accompany regulatory approval of a vaccine these trials are generally of high quality^{Footnote 5}. However, non-randomized trials may be the only evidence available for some populations (e.g. immunocompromised persons), and observational postmarketing studies of the impact of immunization on the incidence of infection or other outcomes (e.g. uncommon adverse events) may provide useful information. Assessment of the quality of a randomized controlled trial focuses on the presence of design strengths such as presence of blinding/ masking of participants and observers, sources of bias such as persons lost to follow-up, handling of confounding factors, and sample size and power to demonstrate important outcomes^{Footnote 4}

Vaccines have unique characteristics compared to other health interventions that should be considered in the process of recommendation development. Two examples will perhaps illustrate this concept: the relationship between the outcome measure used to assess efficacy or effectiveness of the vaccine, and secondly, the person or population to whom the benefits of immunization are conferred.

In the clinical development of a vaccine prior to regulatory approval, large randomized controlled phase 3 trials of a vaccine's efficacy or effectiveness in preventing a clinical outcome (e.g. Haemophilus influenzae meningitis) are often performed, which provide direct evidence of a health benefit to those who received the vaccine. However, once a successful vaccine becomes a standard of care and is integrated into practice, randomized controlled trials may no longer be possible, either because it is considered unethical to withhold the standard of care, or because a trial with a rare outcome is not feasible or is exorbitantly expensive. In this instance researchers use surrogate outcomes of vaccine protection, also called intermediate outcomes, or indicators of efficacy. In vaccine studies these surrogate outcomes are usually immunological correlates of protection, such as specific concentrations of serum antibody or other putative immunologic correlates of protection in vaccinees. When surrogate measures of vaccine efficacy are used, the evidence linking these measures to the health outcome of interest, whether individual or population-based^{Footnote 6,Footnote 7,Footnote 8} should be assembled and evaluated as part of the overall review of evidence.

A second example of the unique nature of evidence supporting vaccine efficacy is evident when one considers the direct and indirect benefits of immunization. The direct benefit of a vaccine may be in the prevention of infection (e.g. prevention of Bordetella pertussis infection of the respiratory tract), of illness associated with acute infection (e.g. exacerbation of underlying cardiopulmonary disease with influenza infection), or of a long term health outcome associated with infection (e.g. cervical cancer following chronic human papillomavirus infection). Indirect benefits of immunization are those conferred on persons other than the vaccinee such as the decreased incidence of invasive Streptococcus pneumoniae infection in older adults associated with infant heptavalent conjugate S. pneumoniae vaccination programs^{Footnote 9}. Indirect harms may also occur, such as unintended transmission of a vaccine strain of an infectious agent from a vaccinee to an uninfected person, as can occur with the use of oral poliovirus vaccine. These community, or population, health outcomes must be considered in assembling evidence for the potential benefits and harms associated with a vaccine.

The nature and magnitude of harms must be evaluated for every health intervention. In general, there is minimal tolerance for harm associated with immunization programs since they are often delivered to children as well as adults, and are usually given to healthy persons. The concept of harm in evaluation of a vaccine is considered in terms of short term reactogenicity (local and systemic adverse effects associated with vaccine) and adverse events that are more common in vaccine than placebo/control recipients. When a new vaccine is introduced, knowledge about rare harms will be limited.

The overall direction and magnitude of the benefits and harms associated with use of a vaccine must be weighed in each recommendation.

Once the level of evidence and the quality of individual studies has been ranked, the entire body of evidence can be judged in terms of the direction of the evidence and the magnitude of the observed effects. As well, other factors may need to be considered (e.g. severity of disease in a certain population in Canada) with these rankings, and the overall scientific case linking the intervention with outcomes of interest assessed.

Multiple other factors, well articulated in the Erickson and de Wals analytical framework for immunization programs in Canada^{Footnote 10}, must be considered by the individual care provider and patient or the health policy planner providing a vaccine program to a population; these parameters are not all currently within the current scope of NACI's work. The individual provider for example, should consider the patient's circumstances, preferences and values. The public health decision maker may consider the cost-effectiveness of a vaccine program and public health resources, and community and political preferences in their jurisdictions for various types of immunization strategies compared to other health programs^{Footnote 11}.

Over time many changes and refinements to the steps in critical appraisal of evidence leading to recommendations have occurred. Among the more well known methodologies are those of the Canadian Task Force on Preventive Health Care (CTFPHC)^{Footnote 12}, the US Preventive Services Task Force^{Footnote 4,Footnote 13}, the Cochrane Collaboration (www.cochrane.org ) and more recently, the Grading of Recommendations Assessment, Development and Evaluation (GRADE) system^{Footnote 14}. There are many established and accepted processes for literature evaluation and synthesis, and NACI does not wish to reinvent a process when acceptable ones are available. These systems, particularly the CTFPHC and GRADE systems, were considered with others^{Footnote 1,Footnote 2,Footnote 15} in the preparation of NACI's process for recommendation development. In brief, the broad stages in the preparation of a NACI recommendation statement are:

1. knowledge synthesis (retrieve and summarize individual studies, rank the level and quality of the evidence)
2. synthesis of the body of evidence of benefits and harms, considering the quality of the evidence and magnitude of effects observed
3. translation of evidence into a recommendation.

Process for NACI recommendation development

1) Knowledge retrieval and synthesis

A NACI Working Group for each vaccine or topic area, comprised of NACI members, liaison members and Public Health Agency of Canada medical and/or epidemiologic specialists convenes to outline the scope of the work and determine if external stakeholders should be added to the group for this project (e.g. external content expert, provincial public health representative). The Working Group defi nes the research question(s) to be answered by the knowledge synthesis including whether epidemiologic evidence is needed about the burden of illness associated with the infection to be prevented. The population(s) of interest should be defined, and the measures of efficacy/effectiveness and safety of the vaccine determined. If necessary, evidence linking surrogate outcomes (e.g. serologic measures) to the efficacy outcomes of interest should be assembled. The work plan for the knowledge synthesis, including scope and deliverables, is presented by the Working Group to NACI for input, modification and finalization. Throughout this process support is provided by the NACI secretariat of the Public Health Agency of Canada.

The literature search may be contracted out by the Public Health Agency or done internally within PHAC. The Working Group maintains contact with the team performing the literature search and syntheses in order to ensure it meets the Working Group's objectives. The completed literature search is used as the evidence base for recommendation development, and will be published separately on the NACI website. Details of the methods of the search (search terms, databases searched, restrictions on language of publication, method for extracting titles, criteria for including/excluding studies, number of reviewers, etc) are provided in the methods section of the literature search. Summary tables are prepared which organize the extracted information (this forms Table 1). Each study is assigned a level of evidence (Table 2) and its quality is assessed (Table 3). Separate tables may need to be created for safety outcomes; this will depend on the nature of the data.

2) Synthesis of the body of evidence

The body of evidence is then reviewed by the Working Group. This review considers the strengths and weaknesses of the overall set of evidence, based on the study design and quality of execution of the individual studies. The overall magnitude of benefits and harms associated with the intervention, other factors (e.g. severity of disease in a certain population in Canada) and the overall scientific case linking the intervention (vaccines, antivirals) with the outcomes of interest in Canadians are considered.

3) Translation of evidence into a recommendation with a strength rating

The Working Group develops options for recommendations for the populations of interest, assigns grades to the strength of the recommendation(s) (Table 4) for each population, and presents these to NACI for consultation and decision making.

The final recommendation will include consideration of the burden of illness associated with the disease and the possible individual and community benefits and harms based on the quality of individual studies, the direction of the effects observed and the magnitude of these benefits and harms. There is not a quantitative process for assigning the letter associated with a recommendation, but the rationale should be presented explicitly in the text of the recommendation statement. In general, stronger recommendations are possible when there is higher quality evidence supporting a greater magnitude of benefit and there is little or no known harm associated with use of the vaccine^{Footnote 12,Footnote 16,Footnote 17}.

Figure 1. Standard language template for methods section of recommendation statement.

NACI reviewed the draft work plan of the X Working Group and the key questions for the proposed literature review, including such considerations as the burden of illness of the disease to be prevented and the target population(s), safety, immunogenicity, efficacy, effectiveness of the vaccine(s), vaccine schedules, and other aspects of the overall immunization strategy. The knowledge synthesis was performed by , and supervised by the Working Group. Following critical appraisal of individual studies, summary tables with ratings of the quality of the evidence using NACI's methodological hierarchy (Table ) were prepared, and proposed recommendations for vaccine use developed. The Working Group chair and PHAC medical specialist presented the evidence and proposed recommendations to NACI on . Following thorough review of the evidence and consultation at the NACI meetings of <date> and <date>, the committee voted on specific recommendations. The description of relevant considerations, rationale for specific decisions, and knowledge gaps are described in the text. The full knowledge synthesis is maintained by the PHAC at the following url:<>. PHAC maintains documentation of these processes throughout knowledge synthesis and recommendation development.

Table 1. Evidence retrieval and synthesis - examples of variables to be extracted from primary studies of vaccine efficacy, effectiveness and safety.

i

Population studied (e.g. age, health status, setting, gender).

ii

Intervention (e.g. vaccine dose(s), route, schedule, concomitant vaccines or medications).

iii

Sample size.

iv

Outcome measures (laboratory confirmed, clinical, surrogate) and method of detection (active/passive). Where surrogate outcome measures are used evidence must be available that directly links the measure with clinical outcomes.

v

Length of follow-up/duration of protection.

vi

Results in treatment and control arm providing confidence intervals and/or statistical tests of significance as appropriate.

vii

Reactogenicity, method(s) for detecting these.

viii

Adverse events, method(s) for detecting these.

ix

Ranking of level of evidence (study design) of each individual study (e.g. randomized controlled trial).

x

Evaluation of quality of the study.

Table 2. Schema for ranking individual study design

I

Evidence from randomized controlled trial(s).

II-1

Evidence from controlled trial(s) without randomization.

II-2

Evidence from cohort or case-control analytic studies, preferably from more than one centre or research group using clinical outcome measures of vaccine efficacy.

II-3

Evidence obtained from multiple time series with or without the intervention. Dramatic results in uncontrolled experiments (such as the results of the introduction of penicillin treatment in the 1940s) could also be regarded as this type of evidence.

III

Opinions of respected authorities, based on clinical experience, descriptive studies and case reports, or reports of expert committees.

Table 3. Quality (internal validity) rating

Good

A study (including meta-analyses or systematic reviews) that meets all design- specific criteria* well.

Fair

A study (including meta-analyses or systematic reviews) that does not meet (or it is not clear that it meets) at least one design-specific criterion* but has no known "fatal flaw".

Poor

A study (including meta-analyses or systematic reviews) that has at least one design-specific* "fatal flaw", or an accumulation of lesser flaws to the extent that the results of the study are not deemed able to inform recommendations.

* General design specific criteria are outlined in Harris et al., 2001^{Footnote 4}.

Table 4. NACI Recommendations for Immunization

A

NACI concludes that there is good evidence to recommend immunization.

B

NACI concludes that there is fair evidence to recommend immunization.

C

NACI concludes that the existing evidence is conflicting and does not allow making a recommendation for or against immunization, however other factors may influence decision-making.

D

NACI concludes that there is fair evidence to recommend against immunization.

E

NACI concludes that there is good evidence to recommend against immunization.

I

NACI concludes that there is insufficient evidence (in either quantity and/or quality) to make a recommendation, however other factors may infl uence decision-making.

† Members: Dr. J. Langley (Chairperson), Dr. B. Warshawsky (Vice-Chairperson), Dr. S. Virani (Executive Secretary), Ms. A. Hanrahan, Dr. K. Laupland, Dr. A. McGeer, Dr. S. McNeil, Dr.  B. Seifert, Dr. D. Skowronski, Dr. B. Tan.

Liaison Representatives:Dr. B. Bell (CDC), Dr. P. Orr (AMMI Canada), Ms. S. Pelletier (CHICA), Ms. K. Pielak (CNCI), Dr. P. Plourde (CATMAT), Dr. S. Rechner (CFPC), Dr. M. Salvadori (CPS), Dr. D. Scheifele (CAIRE), Dr. N. Sicard (CPHA), Dr. V. Senikas (SOGC),.

Ex-Officio Representatives: Dr. S. Desai (CIRID - Vaccine Preventable Diseases), Ms. M. FarhangMehr (CIRID - Canadian Immunization Committee), Major P. Laforce (DND), Dr. B. Law (CIRID - Vaccine Safety), Dr. R. Ramsingh (FNIHB), Dr. F. Hindieh (BGTD),

^† This statement was prepared by Dr. Joanne Langley and approved by NACI.