Evidence Based Process for developing travel and tropical medicine related guidelines and recommendations
An Advisory Committee Statement (ACS)
Committee to Advise on Tropical Medicine and Travel (CATMAT)
Table of Contents
- Preamble
- Key points/Messages
- Introduction
- Objective
- Guideline Development Methods
- Overview
- Methods of Guideline Development Process
- Deciding on methodological approach for Recommendation Development
- General approach for Development of non-GRADE Recommendations
- General approach for Development of a Guideline containing GRADE Recommendations
- Developing an Analytic Framework and identifying key questions (steps 1-5)
- Assessing the quality (confidence in the estimate of effect) of the evidence (steps 6-12)
- Rating down the quality of the body of evidence
- Estimating the baseline risk of disease and quality of the body of evidence (steps 9 & 10)
- Establishing a risk threshold to guide GRADE Recommendations (step 14)
- Strength of the Recommendation (step 15)
- Discussion
- Acknowledgements
- Conflict of Interest
- References
- Appendices
- Appendix 1: Summary of the GRADE Framework for developing recommendations
- Appendix 2: Typhoid example of logic model and Analytic Framework
- Appendix 3: PICO and Key Questions for Typhoid Analytic Framework
- Appendix 4: Criteria to Judge Quality of Evidence
- Appendix 5: Evidence profile for the efficacy of typhoid vaccine
- Appendix 6: Summary of findings for the efficacy of typhoid vaccine
- Appendix 7: Evidence profile for assessment of confidence in estimates of baseline risk of typhoid by region
Preamble
The Committee to Advise on Tropical Medicine and Travel (CATMAT) provides the Public Health Agency of Canada with ongoing and timely medical, scientific, and public health advice relating to tropical infectious disease and health risks associated with international travel. The Agency acknowledges that the advice and recommendations set out in this statement are based upon the best current available scientific knowledge and medical practices, and is disseminating this document for information purposes to both travellers and the medical community caring for travellers.
Persons administering or using drugs, vaccines, or other products should also be aware of the contents of the product monograph(s) or other similarly approved standards or instructions for use. Recommendations for use and other information set out herein may differ from that set out in the product monograph(s) or other similarly approved standards or instructions for use by the licensed manufacturer(s). Manufacturers have sought approval and provided evidence as to the safety and efficacy of their products only when used in accordance with the product monographs or other similarly approved standards or instructions for use.
Key Points/Messages
- CATMAT has been developing travel and tropical medicine guidelines and recommendations using an "evidence based" approach since 1994.
- CATMAT has updated its guideline development process and will use the Grading of Recommendations, Assessment, Development and Evaluation (GRADE) method for certain recommendations. For other recommendations, a less rigorous search and synthesis of the literature will be used.
- The choice of recommendations to which GRADE will be applied will depend on a number of factors including: the anticipated burden of the disease, the seriousness of the outcomes to be prevented, the potential benefits and harms of the intervention, the quality of the evidence, and the resources available to the committee.
- These updated methods will provide a transparent process of developing travel and tropical medicine related recommendations.
Introduction
The Canadian Committee to Advise on Tropical Medicine and Travel (CATMAT) has been developing guidelines for travel health and tropical disease for more than 20 years. Over this period, evidence-based medicine (EBM) methodologies have evolved dramatically. To keep pace with these developments, CATMAT has recently updated its approach to developing guidelines. In 2013, CATMAT formally adopted the Grading of Recommendations, Assessment, Development and Evaluation (GRADE) system to develop guidelines. GRADE uses a systematic framework for assessing the quality of the body of evidence and determining the direction and strength of a recommendationFootnote 1 Footnote 2.This methodology emphasizes transparency, and clinically important outcomes. CATMAT will use GRADE to develop certain recommendations; otherwise evidence assessment and recommendation development will be developed using a less rigorous synthesis of the evidence.
The approach chosen to develop recommendations will depend on a number of factors including: the anticipated burden of the disease, the seriousness of the outcomes to be prevented, the potential benefits and harms of the intervention, the quality of the evidence, and the resources available to the committee. The first CATMAT statement to use this approach was the statement on "International Travel and Typhoid (2014)"Footnote 3. It included recommendations developed with and without GRADE. A similar approach was used to develop the CATMAT statement on Travellers' DiarrheaFootnote 4. A full list of CATMAT statements can be found at About CATMATFootnote 5.
Objective
The objective of this statement is to describe CATMAT's current guideline development process. The target population for this statement are: health care providers who use CATMAT guidelines; those who have an interest in and/or to whom the CATMAT guidelines apply; and, individuals who will be involved with developing CATMAT guidelines. The intent is that this statement will enhance the understanding of CATMAT's approach to guideline development. It should also help with the interpretation and implementation of their recommendations.
Guideline Development Methods
Overview
CATMAT is an expert panel, composed primarily of clinicians and other subject matter experts, that writes guidelines and makes recommendations related to the prevention and treatment of infectious diseases and other health hazards that may be encountered by Canadians travelling outside of the country. The work of the panel is supported by a secretariat at the Public Health Agency of Canada (PHAC). Additional information on committee functioning, reporting, membership, conduct and conflict of interest requirements are available on Government of Canada's websiteFootnote 6.
CATMAT identifies and prioritizes potential guideline topics at least annually. CATMAT members, PHAC, stakeholders or other organizations and individuals, may suggest topics. For each guideline topic, a working group (WG) is formed and is comprised of members from the committee, the CATMAT secretariat and, as necessary, external experts and/or interested parties. The WG is responsible to develop the scope of the proposal and to write an initial draft guideline. CATMAT reviews drafts and revisions are made accordingly. CATMAT votes on the recommendations and, once approved by the committee, the final version is presented to PHAC for approval and publication.
Methods of Guideline Development Process
All guidelines will include a clear description of the methods including how the literature was identified and reviewed, the approach used for recommendation development (i.e. GRADE or other method) and reasons for choosing a given approach. Guidelines may include a recommendation based on a GRADE approach, a recommendation based on a synthesis of evidence (without GRADE) or Good Practice statements or a combination (see description below). In some cases, guidelines will be purely descriptive. These reviews will synthesize the relevant literature on the topic and will highlight the epidemiology of a condition of concern for a specific population, e.g., "International Travellers who intend to Visit Friends and Relatives"Footnote 7. Descriptive guidelines are valuable and a useful resource for travel medicine providers as they will serve to promote the health of particular populationFootnote 7.
Deciding on Methodological Approach for Recommendation Development
A CATMAT guideline document may include recommendations based on GRADE, on a synthesis of evidence (without GRADE) or a combination of both. The use of GRADE for selected recommendations will be based on the judgement of the full committee and the WG.
Criteria that enhance the likelihood that a recommendation(s) will be developed with GRADE:
- A high prevalence (common) travel-related condition for which an effective intervention is available.
- A rare, low, or moderate prevalence (less common/rare) travel-related condition associated with significant morbidity or mortality for which an effective intervention is available.
- A travel-related condition for which the potential magnitude of benefits or harms of an intervention is high.
- An intervention for which there is a narrow gradient between the balance of harms and benefits.
Criteria that reduce the likelihood that a recommendation(s) will be developed with GRADE:
- A travel-related condition for which urgent guidance is required (e.g., Zika virus in 2016)Footnote 8.
- A travel-related condition for which the effectiveness of the intervention is unknown or poorly defined.
- Interventions that are not central to the key question(s) addressed in the guideline but may provide important contextual or background information (e.g., hand washing).
- Resource constraints.
General Approach for Development of Non-GRADE Recommendations
Recommendation Based on a Literature Review
Non-GRADE recommendations may be based on a literature review. Where possible, recommendations will be informed by recent systematic reviews, relevant randomized controlled trials (RCTs) or other literature. Non-GRADE recommendations are not inherently of lower value; indeed, they might be relatively more important than a GRADE recommendation. For example, a GRADE recommendation for a vaccine that protects against a relatively low-probability threat might have less impact than a non-GRADE recommendation for an intervention that provides significant protection (e.g., washing hands, using insect repellents) against an array of threats.
Good Practice Recommendations
Good practice recommendations are suited to interventions or other preventive approaches that entail minimal or no risk, inconvenience, or cost, and for which there is indirect or generally accepted evidence of benefit in terms of reducing the outcome or harm. This type of recommendation does not require formal literature to substantiate the recommendation. Examples include avoiding high-risk foods or suggesting that a physician think of and test for malaria in any patient with a febrile illness who has recently returned from a malaria endemic area. Good practice recommendations may address issues related to equity in health care.
Good practice statements should have the following characteristicsFootnote 9:
- Clear, actionable and feasible
- Have an important message
- Have an unequivocal net benefit with negligible harms and costs.
Descriptive Statements
Descriptive statements that highlight the epidemiology and baseline risk of a specific population such as the CATMAT statement on "International Travellers who intend to Visit Friends and Relatives" are also valuable as they are a useful resource for travel medicine providers and will serve to promote the health of particular populationsFootnote 7. These reviews will synthesize the relevant literature on the topic.
General Approach for Development of a Guideline Containing GRADE Recommendations
The following steps outline the CATMAT approach to developing guidelines that contain GRADE-based recommendations (Appendix 1). For examples of how this approach has been applied see the CATMAT statements on "International Travel and Typhoid (2014)"Footnote 3 and "Travellers' Diarrhea"Footnote 4.
Determine the scope and specific objective(s) of the guideline.
- Develop an analytic framework (Appendix 2).
- Determine what, if any, specific questions will be assessed with a GRADE approach.
- For interventions to be assessed with GRADE, develop a "PICO" (population of interest, intervention, comparison and outcome) question(s) (Appendix 3). Assign each outcome a level of importance i.e. critical, important, not-important (see below).
- Identify other key and contextual questions that will inform the development of recommendations.
- Conduct a literature search to identify relevant evidence.
- If a published systematic review(s) is/are identified, assess its/their relevance and qualityFootnote 9. If relevant and of satisfactory quality, use the systematic review (with a literature review to identify relevant evidence since its publication) as the basis for the GRADE analyses.
- Assess the quality of the body of evidence for each individual outcome for every intervention/comparator pair by assessing risk of bias, publication bias, imprecision, inconsistency, and indirectness.
- Assign a degree of confidence in the estimate of effect for each outcome for each intervention assessed, i.e. very low, low, moderate, or high (Appendix 4)Footnote 10.
- Estimate the baseline likelihood for the outcome(s) of interest in the population(s) of interest. This may highlight specific risk populations (e.g. adults/children/infants) or travel destinations (e.g. Africa, southeast Asia, South America) as appropriate.
- Assess the quality of the body of the evidence for baseline likelihood using the same criteria as are applied to interventions i.e. risk of bias, publication bias, imprecision, inconsistency, and indirectness.
- Summarize the evidence by outcome(s) for each intervention(s)/comparator in Evidence Profiles (EP) and Summary of Findings tables (SoF) (Appendix 5 & 6) as per the GRADE methodology.
- Summarize the evidence by outcome(s) for the baseline risk overall and by risk groups in Evidence Profiles (EP) (Appendix 7) and Summary of Findings tables (SoF) (not included) as per the GRADE methodology.
- Calculate the relative effect and confidence interval and, using the baseline risk, the absolute effect and confidence interval.
- Establish a risk threshold for use (or non-use) of the intervention(s) based on the balance of benefits and harms combined with evidence for patient values and preferences. In the absence of published evidence for patient values and preferences, the committee will use their experience in shared decision making to estimate an appropriate decision threshold.
- Develop recommendations taking into consideration:
- Magnitude of benefits and harms of the intervention.
- Confidence in the estimates of the effect for intervention benefits and harms.
- The values and preferences of travellers including considerations related to cost and the resulting benefit/harm threshold.
Developing an Analytic Framework and Identifying Key Questions (steps 1-5)
An analytic framework should identify clinical preventive actions (interventions) and relevant factors that impact on outcomes including important subgroups at risk, availability or uptake of diagnostic test(s) or intervention(s), and differences in how the outcome is measured (or if surrogate endpoints will be used)Footnote 11. Assign a level of importance to each outcome on a 1 to 9 scale, i.e. critical (7-9), important (4-6), and limited importance (1-3). Ranking importance in this way can help to focus effort on the most relevant questions. The target audience of CATMAT statements are clinicians and patients; therefore, in most circumstances, the primary perspective will be that of the patient and the relative importance of the outcome should reflect this. Examples of an analytic framework and PICO and key question developed for the International Travel and Typhoid statementFootnote 3, are shown in Appendix 2 and 3.
Assessing the Quality (confidence in the estimate of effect) of the Evidence (steps 6-12)
The quality of the "body of evidence" by outcome is assessed by determining the study design by which the selected evidence was produced (e.g., RCT, observational studies) and then assessing for risk of bias, inconsistency across studies, indirectness, imprecision, and publication biasFootnote 12. If concerns are identified then the quality of evidence may be rated down (see below). Evidence is summarized into Evidence Profiles (EP) and Summary of Findings (SoF) tables, and each outcome/intervention combination is assigned a quality designation (very low, low, moderate, or high (see box 1 and examples of tables in Appdendix 5 & 6)Footnote 10. This quality assessment reflects confidence in how close the estimate is to the true effect. For interventions, evidence derived from RCTs starts out with high confidence whereas it starts out with low confidence for observational studies. For baseline risk estimates, the situation is different. Observational cohort studies start out as high quality evidence because they are more suited than RCTs to provide population based estimates of riskFootnote 13 and RCTs, if they include non-representative populations, will be rated down for indirectnessFootnote 13 Footnote 14 (See further discussion in section on baseline risk below). Readers are refered to a series of articles on GRADE in the Journal of Clinical Epidemiology for a thorough treatment of the quality assessment processFootnote 10 Footnote 12 Footnote 15 - Footnote 19. Other articles describe rating the quality of evidence for observational studiesFootnote 13 Footnote 14. What follows is a brief review of the quality assessment domains.
Levels | Rationale |
---|---|
High | We are very confident that the true effect lies close to that of the estimate of effect. |
Moderate | We are moderately confident in the effect estimate. The true effect is likely close to the estimate of effect, but there is a possibility that it is substantially different. |
Low | Our confidence in the effect estimate is limited. The true effect may be substantially different from the estimate of the effect. |
Very low | We have very little confidence in the effect estimate. The true effect is likely to be substantially different from the estimate of effect. |
Adapted from Balshem et al. 2011Footnote 10 |
Rating Down the Quality of the Body of Evidence
Risk of Bias
Quality of the evidence may be rated downwards by one or two levels for serious or very serious risk of bias, respectivelyFootnote 19. Examples that could result in rating down for RCTs include: inappropriate randomization; lack of allocation concealment; failure to express outcomes by 'intention-to-treat'; inadequate blinding; substantial loss to follow-up; and stopping early for benefit.
Inconsistency
Quality of the evidence may be rated downwards by one or two levels if the estimate of effect shows substantial variation across studiesFootnote 17.
Indirectness
Quality of the evidence may be rated downwards by one or two levels if there are serious or very serious issues with indirectness. Examples of indirectness include: use of surrogate rather than patient-important outcomes (e.g. use of immunogenicity as opposed to episodes of infection); making indirect comparisons between two treatments; study populations that are dissimilar to the population(s) of interest; and test inaccuraciesFootnote 16. The extent of concerns (no concerns, serious concerns, very serious concerns) is a matter of judgment. For instance, some might view immunogenicity as sufficient evidence of benefit. If studies of the extent of protection are planned after establishing immunogenicity, this suggests the need to rate down one for indirectness.
Imprecision
Quality of the evidence may be rated downwards by one or two levels if there is serious or very serious imprecision (e.g., wide confidence intervals)Footnote 15.
Publication Bias
Quality of the evidence may be rated downwards by one or two levels if publication bias is detected or strongly suspectedFootnote 18.
Rating Up the Quality of the Body of Evidence
In specific circumstances it may be appropriate to rate up the quaility of the body of evidenceFootnote 20. These are described below.
Large Magnitude of Effect/Strength of Association
Quality of the evidence may be rated upwards by one or two levels based on magnitude of effect. As a general guideline, rating up by one level may be considered if evidence from RCTs or observational studies suggest a 2 fold or greater reduction or increase (OR/RR >=2 or =< .5) with no major residual confounders. Rating upwards by two or more levels may be considered if evidence from RCTs or observational studies suggest a 5 fold or greater reduction or increase (OR/RR >=5 or =< .2) with no major residual confounders. For instance, rigorous observational studies demonstrating reduction in incidence of the target condition by over 90% in vaccinated versus unvaccinated cohorts would warrant rating up by two levels. Generally, rating upward for a large effect should not be applied if the informing evidence suffers from serious methodologic limitations. Serious limitations include use of surrogates: we would never rate up for evidence from a surrogate such as immunogenicity. Rating up of RCTs is theoretically possible but seldom done.
Dose-Response Gradient
Quality of the evidence may be rated upwards if there is evidence of a dose-response gradient. The dose response has long been recognized as an important criterion supporting a cause-effect relationship and its presence increases confidence in the findings of an observational study.
Mitigated Bias and Confounding
Observational studies begin as low-quality evidence because they are more likely to suffer from unmeasured or unknown sources of error and are therefore more likely to be affected by residual confounding. For such studies, if all residual confounders in a study would result in an underestimate of an apparent treatment effect then the quality of the evidence may be rated upward by one level.
Estimating the Baseline Risk of Disease and Quality of the Body of Evidence (steps 9 & 10)
Estimating the baseline likelihood of an outcome allows for estimation of absolute intervention benefit and related measures, e.g., number needed to treat (NNT) or number needed to vaccinate (NNV) to prevent a case. Methods to assess the quality of the body of evidence from observational, non-intervention studies of baseline likelihood have recently been publishedFootnote 13 Footnote 14. The GRADE group recommends applying the same criteria that are used for assessing the quality of intervention-based data, i.e. risk of bias, publication bias, imprecision, inconsistency, and indirectnessFootnote 13 Footnote 14. A notable difference when assessing quality for baseline risk studies is that observational studies start out as high-quality data because they are generally more suited than RCTs to provide population based estimates of riskFootnote 13. Different tools have been developed to assess the risk of bias of observational studies including the New Castle Ottawa instrument and the Cochrane ROBINS-I toolFootnote 21 Footnote 22. The criteria for assessing bias depend on the study design and should include assessment for selection, detection, and attrition bias. Key questions for this assessment include:
- Is the study sample from a well-defined population that is representative of Canadian travellers?
- Are the exposure and outcome(s) measures reliable and accurate?
- Is it likely that there is complete ascertainment of the exposure(s) and outcome(s)?
- Has follow-up been sufficiently long and complete?Footnote 13
An example of the evaluation of studies of baseline risk and their quality assessment (adapted from the statement on International Travel and Typhoid statement) is included in Appendix 7Footnote 3.
Establishing a Risk Threshold to Guide GRADE Recommendations (step 14)
A GRADE recommendation will be accompanied by a risk threshold determined by the balance between the importance (utility or disutility) of the relevant outcomes to the patients. It represents the point at which an intervention is recommended or not recommended. Thresholds must take into account patient values and preferences. Ideally, these will be derived from studies on the relevant population(s). In the absence of such data or as a second source of information, the experience of clinicians who regularly engage in shared decision-making can be used to estimate patient values and preferences. In all cases, the evidence and/or judgements made to establish thresholds will be explicitly stated.
An example of a risk threshold is the judgement made by CATMAT in the "International Travel and Typhoid Statement" that travellers would accept a likelihood of 1/10,000 for developing clinical disease before accepting the cost and inconvenience of typhoid vaccineFootnote 3. This decision balanced factors such as the relatively high effectiveness of typhoid therapy, the low absolute risk of typhoid-associated hospitalization or death (< 1%), the costs associated with vaccination, the relatively low efficacy of the vaccine (50%), and the minor harms such as fever and gastrointestinal upset associated with the vaccine. In contrast, the risk threshold for taking Japanese encephalitis vaccine to prevent Japanese encephalitis (JE) is likely to be substantially lower, perhaps by one or two orders of magnitude than for typhoid (i.e. a risk of disease of 1/100,000 or lower). This reflects fundamental differences between these diseases and vaccines. Specifically, JE is: more severe (25% of cases die and 50% of those who survive develop severe and long-term neurologic sequelae); lacks specific treatments like antibiotics (which can be used for typhoid); and, has a higher probability of being prevented (>90% efficacy) through vaccination.
Strength of the Recommendation (step 15)
The strength of a recommendation reflects the extent to which the committee is confident that the desirable effects of an intervention outweighs undesirable effects, taking into account the importance and impact of the intervention, and values and preferences of the patient. The following factors need to be considered (Box 2) when determining the strength associated with a given recommendation. The categories of, and implicit meanings for the various types of CATMAT GRADE-based recommendations are described in Box 3.
Factors | Impact on the strength of a recommendation |
---|---|
Certainty about the evidence of effects |
The higher the quality of evidence, the more likely a strong recommendation is warranted. |
Balance between desirable/undesirable effects |
The larger the difference between the desirable and undesirable effects, the more likely a strong recommendation is warranted. The smaller the net benefit and the narrower the gradient, the more likely a conditional recommendation is warranted. |
Importance of outcomes (more likely a weak recommendation) |
The more variability or more uncertainty about the relative importance of the outcome, the more likely a conditional recommendation is warranted. |
Values and Preferences |
The greater the variability in value and preferences, or uncertainty in values and preference, the more likely a conditional recommendation is warranted. |
Costs (resource allocation) |
The higher the cost of an intervention, the more likely a conditional recommendation is warranted. |
Recommendation | Rationale |
---|---|
StrongBox 3 Footnote * recommendation for |
The committee believes that all or almost all well-informed people would want the recommended course of action and only a small number would not. Implication for practitioners: The balance of risks and benefits are such that most travellers would choose the intervention. |
Strong recommendation against |
The committee believes that all or almost all well-informed people would not want the recommended course of action and only a small number would. Implication for practitioners: The balance of risks and benefits are such that most travellers would not choose the intervention. |
ConditionalBox 3 Footnote ** recommendation for |
The committee believes that the majority of well-informed people would want the recommended course of action, but a minority (perhaps a large minority) would not. Implication for practitioners: With a conditional recommendation different travellers may make different choices. Practitioners should present the risks and benefits of the intervention and help each traveller make a decision consistent with his/her values and preferences. |
Conditional recommendation against |
The committee believes that the majority of well-informed people would not want the recommended course of action, but a minority (perhaps a large minority) would. Implication for practitioners: With a conditional recommendation different travellers may make different choices. Practitioners should present the risks and benefits of the intervention and help each traveller make a decision consistent with his/her values and preferences. |
|
Discussion
Benefits of GRADE in the Travel Medicine Context
Applying the GRADE methodology to travel medicine recommendations may allow for improved harmonization of recommendations across countries. This is because GRADE provides a standardized and very widely accepted method of synthesizing and judging the quality of the evidence for interventions. Moreover, and crucial to travel medicine, GRADE-based methods for observational data such as baseline risk are well developedFootnote 13 Footnote 14. Finally, the methodology stresses transparency and openness at each step in the guideline development process. Consequently, even if consensus on recommendations cannot be reached, applying GRADE will allow guideline developers and users to better understand how and why recommendations were made.
A notable example of how GRADE could be used to enhance clarity in the travel medicine field is the divergent recommendations for chemoprophylaxis to prevent malaria. Currently, some national authorities recommend chemoprophylaxis for virtually all endemic areas, whereas others recommend it in much more limited geographic areas. If GRADE was applied by all travel medicine guideline developers, there may still not be agreement on malaria chemoprophylaxis recommendations, but the evidence on which recommendations were made including the risk threshold based on the values and preferences of patients would be transparently presented.
Another important benefit of applying GRADE to clinical aspects of travel medicine is the clear presentation of absolute risks and benefits. This should make it easier for practitioners to help patients make informed decisions related to use of preventive measures.
GRADE Challenges and Limitations
The GRADE methodology has several limitations. The systematic review and synthesis process is labour-intensive and there is a steep learning curve to understand GRADE evidence synthesis methodology and software. Specialist support (librarian, methodologist) can enhance efficiency and is required in most settings. Determining the risk threshold for a given intervention may be difficult and will be based on clinical experience (low confidence) in the absence of published or other evidence on patient values and preferences. There is often low- or very low-quality data available on the baseline risk of disease in travellers, which serves to reduce confidence in recommendations. Further, preventive interventions often have been assessed in endemic countries rather than with travellers, which also reduces confidence in the estimates (i.e. indirectness). Finally, the GRADE terminology and process may be unfamiliar to many practitioners who will need to learn the new classifications for data quality and then find ways to best translate the recommendations into clinical practice.
Conclusion and Future Directions
GRADE can be used to develop travel medicine guidelines. However, there are limitations, not least of which are resource constraints. For this and other reasons, CATMAT will use GRADE in a strategic fashion. For non-GRADE recommendations, CATMAT will use a selected evidence synthesis process. EBM and guideline development will continue to evolve. CATMAT will, as is necessary, update its approach to EBM and guideline development.
Acknowledgements
- This statement was developed by the Evidence Based Medicine Working Group: Greenaway C (Chair), Schofield S, Geduld J, Guyatt G and approved by CATMAT.
- CATMAT members: McCarthy A (Chair), Acharya A, Boggild A, Brophy J, Bui Y, Crockett M, Greenaway C, Libman M, Teitelbaum P and Vaughan S.
- Liaison members: Angelo K (United States Centers for Disease Control and Prevention), Audcent T (Canadian Paediatric Society) and Pernica J (Association of Medical Microbiology and Infectious Disease Canada).
- Ex officio members: Marion D (Canadian Forces Health Services Centre, Department of National Defence), McDonald P (Bureau of Medical Sciences, Health Canada), Rossi C (Medical Intelligence, Department of National Defence) and Schofield S (Pest Management Entomology, Department of National Defence).
Conflict of Interest
None declared.
References
- Footnote 1
What is GRADE? 2017; Available at: http://www.gradeworkinggroup.org/. Accessed July 27, 2016.
- Footnote 2
Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ 2008 Apr 26;336(7650):924-926.
- Footnote 3
Greenaway C, Schofield S, Henteleff A, Plourde P, Geduld J, Abdel-Motagally M, et al. Summary of the Statement on International Traveller's and Typhoid by the Committee to Advise on Tropical Medicine and Travel (CATMAT). Can Comm Dis Rep 2014 February 20;40(4):60.
- Footnote 4
Libman M, on behalf of CATMAT. Summary of the Committee to Advise on Tropical Medicine and Travel (CATMAT) Statement on Travellers' Diarrhea. Can Comm Dis Rep 2015 November 5;41(11):272.
- Footnote 5
Public Health Agency of Canada. About Committee to Advise on Tropical Medicine and Travel (CATMAT). 2016; Available at: http://www.phac-aspc.gc.ca/tmp-pmv/catmat-ccmtmv/index-eng.php. Accessed July 27, 2016.
- Footnote 6
Public Health Agency of Canada. Committee to Advise on Tropical Medicine and Travel (CATMAT) Terms of Reference. 2015; Available at: http://www.phac-aspc.gc.ca/tmp-pmv/catmat-ccmtmv/references-eng.php. Accessed July 29, 2016.
- Footnote 7
Brophy J, on behalf of the Committee to Advise on Tropical Medicine and Travel (CATMAT). Summary of the Statement on International Travellers Who Intend to Visit Friends and Relatives. Can Comm Dis Rep 2015 May 7;41(05):89.
- Footnote 8
Zika Working Group on behalf of the Committee to Advise on Tropical Medicine and Travel (CATMAT). Canadian Recommendations on the Prevention and Treatment of Zika virus: Update. Can Comm Dis Rep 2016 May 5;42(5):101.
- Footnote 9
Guyatt GH, Schunemann HJ, Djulbegovic B, Akl EA. Guideline panels should not GRADE good practice statements. J Clin Epidemiol 2015 May;68(5):597-600.
- Footnote 10
Balshem H, Helfand M, Schunemann HJ, Oxman AD, Kunz R, Brozek J, et al. GRADE guidelines: 3. Rating the quality of evidence. J Clin Epidemiol 2011 Apr;64(4):401-406.
- Footnote 11
Guyatt GH, Oxman AD, Kunz R, Atkins D, Brozek J, Vist G, et al. GRADE guidelines: 2. Framing the question and deciding on important outcomes. J Clin Epidemiol 2011 Apr;64(4):395-400.
- Footnote 12
Guyatt G, Oxman AD, Akl EA, Kunz R, Vist G, Brozek J, et al. GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J Clin Epidemiol 2011 Apr;64(4):383-394.
- Footnote 13
Iorio A, Spencer FA, Falavigna M, Alba C, Lang E, Burnand B, et al. Use of GRADE for assessment of evidence about prognosis: rating confidence in estimates of event rates in broad categories of patients. BMJ 2015 Mar 16;350:h870.
- Footnote 14
Spencer FA, Iorio A, You J, Murad MH, Schunemann HJ, Vandvik PO, et al. Uncertainties in baseline risk estimates and confidence in treatment effects. BMJ 2012 Nov 14;345:e7401.
- Footnote 15
Guyatt GH, Oxman AD, Kunz R, Brozek J, Alonso-Coello P, Rind D, et al. GRADE guidelines 6. Rating the quality of evidence—imprecision. J Clin Epidemiol 2011 12;64(12):1283-1293.
- Footnote 16
Guyatt GH, Oxman AD, Kunz R, Woodcock J, Brozek J, Helfand M, et al. GRADE guidelines: 8. Rating the quality of evidence—indirectness. J Clin Epidemiol 2011 12;64(12):1303-1310.
- Footnote 17
Guyatt GH, Oxman AD, Kunz R, Woodcock J, Brozek J, Helfand M, et al. GRADE guidelines: 7. Rating the quality of evidence—inconsistency. J Clin Epidemiol 2011 12;64(12):1294-1302.
- Footnote 18
Guyatt GH, Oxman AD, Montori V, Vist G, Kunz R, Brozek J, et al. GRADE guidelines: 5. Rating the quality of evidence—publication bias. J Clin Epidemiol 2011 12;64(12):1277-1282.
- Footnote 19
Guyatt GH, Oxman AD, Vist G, Kunz R, Brozek J, Alonso-Coello P, et al. GRADE guidelines: 4. Rating the quality of evidence—study limitations (risk of bias). J Clin Epidemiol 2011 4;64(4):407-415.
- Footnote 20
Guyatt GH, Oxman AD, Sultan S, Glasziou P, Akl EA, Alonso-Coello P, et al. GRADE guidelines: 9. Rating up the quality of evidence. J Clin Epidemiol 2011 12;64(12):1311-1316.
- Footnote 21
Wells G, Shea B, O'Connell D, Peterson J, Welch V, Losos M. The Ottawa Hospital Research Institute: The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses. Available at: http://www.ohri.ca/programs/clinical_epidemiology/oxford.asp. Accessed February/19, 2017.
- Footnote 22
Sterne JA, Hernan MA, Reeves BC, Savovic J, Berkman ND, Viswanathan M, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ 2016 October/12;355(i4919).
- Footnote 23
Schünemann H, Brożek J, Guyatt G, Oxman A, editors. GRADE handbook for grading quality of evidence and strength of recommendations. Updated October 2013. The GRADE Working Group, 2013. Available at : https://gdt.gradepro.org/app/handbook/handbook.html. Accessed March 2017.
Appendices
Appendix 1: Summary of the GRADE Framework for developing recommendations
Source: Handbook for grading the quality of evidence and the strength of recommendations using the GRADE approachFootnote 25.
- GDT: Guideline Development Tool
- EtD: Evidence to Decision
- PICO: population of interest, intervention, comparison and outcome
- RCTs: relevant randomized controlled trials
- SR: Systematic Review
- HTA : Health Technology Assessment
Text Equivalent
Appendix 1 shows a schematic view of the GRADE approach for synthesizing evidence and developing recommendations (a description of this process is provided on pages 6-10). The upper half of the figure (steps 1-3) describe steps in the process related to evidence synthesis and common to systematic reviews, including formulating the question and assessing the quality of the evidence for each outcome. The lower half (steps 4-6) describe steps that are specific to assessing the overall quality of evidence to make a decision or develop recommendations (based on GRADE meeting, Edinburgh 2009).
Six steps are presented:
- Formulate the question using “PICO” (population of interest, intervention, comparison and outcome). Select outcomes and rate the importance of each outcome (not important, important, critical).
- Conduct a literature review and create a summary table with an evidence profile for each study using the Guideline Development Tool (GDT), including an estimate of effect for each outcome.
- Rate the quality of evidence for each outcome (high, moderate, low, very low). Randomization raises initial quality (RCTs: high; Observational studies: low). Grade up or down considering risk of bias; inconsistency; indirectness; imprecision; publication bias; large effect; dose response; opposing bias and confounders.
- Grade the overall quality of evidence across outcomes based on the lowest quality of critical outcomes.
- Grade recommendations or the evidence to decision (for or against; strong or conditional/weak strength of evidence) using the Evidence to Decision (EtD) framework with GDT by considering a balance of the consequences. These include the quality of evidence; a balance of the benefits/harms; values and preferences; feasibility, equity and acceptability; and resource use (if applicable).
- Formulate recommendations/guidelines/decision which is/are transparent, clear and actionable.
Appendix 2: Typhoid example of logic model and Analytic Framework
Text Equivalent
Appendix 2 shows an example of an analytic framework that was developed for the “International Travel and Typhoid (2014)” CATMAT statement. This framework identifies clinical preventive actions (interventions) for typhoid and risk factors for typhoid disease.
The analytic framework depicts twelve key concepts.
The main “PICO” (population of interest, intervention, comparison and outcome) question:
- Among Canadian travellers, does the administration of typhoid vaccine versus no vaccine decrease the incidence of typhoid and the associated morbidity and mortality?
Additional concepts related to the magnitude of benefits and harms:
- What is the incidence of typhoid in unvaccinated travellers?
- What are the subgroups at a higher risk for typhoid?
- What are high risk destinations for typhoid?
- What duration of travel affects the risk for typhoid?
- What is the vaccine efficacy?
- What harms are associated with typhoid vaccination?
- What are the costs associated with vaccination?
- What is the incidence of typhoid and the associated morbidity and mortality in unvaccinated travellers that do not get treatment?
- What is the efficacy and effectiveness of typhoid treatment?
- What are the associated side effects of treatment versus the proportion cured?
- What are the associated costs of treatment versus the proportion cured?
Appendix 3: PICO and Key Questions for Typhoid Analytic Framework
- 1."PICO" (population of interest, intervention, comparison and outcome) question
Among Canadian travellers, does the administration of typhoid vaccine versus no vaccine decrease the incidence of typhoid and the associated morbidity and mortality?
Key questions from the analytic framework,
- 2-5. What are the important risk factors for typhoid among travellers (e.g., destination, duration of travel, age, VFRs, comorbidities such as infection with HIV or acid suppression/achlorhydria)?
- 6. What is the efficacy of the typhoid vaccine
- 7. What are the harms associated with typhoid vaccination?
- 8. What are the values and preferences of travellers regarding the magnitude of risk reduction in typhoid that would make use of the vaccine worthwhile given the cost and inconvenience associated with the vaccine?
Additional contextual questions that were considered included:
- 9. What is the efficacy of antimicrobial treatment to reduce typhoid mortality and morbidity among travellers.
- 10. What are the side-effects and costs of antimicrobial treatment?
- 11. Does pathogen susceptibility to antimicrobial treatment vary by destination?
- 12. Do hygiene and/or food and water precautions reduce the risk of acquiring typhoid among travellers?
Appendix 4:
Quality of evidence | Study design | Rating down if… | Rating up if… |
---|---|---|---|
High |
randomized study (RCT) |
study limitations inconsistency indirectness imprecision publication bias |
magnitude of effect dose-response gradient all plausible confounding |
Moderate |
n/a | ||
Low |
observational studyAppendix 4 Footnote * |
||
Very low |
n/a | ||
|
Appendix 5:
Quality assessment | No of patients | Effect | Quality | ||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
No of studies | Design | Risk of bias | Inconsistency | Indirectness | Imprecision | Other considerations | Typhoid vaccine | Control | Relative (95% CI) |
Absolute | |
Three year cumulative risk of typhoid fever combined | |||||||||||
4 | randomised trials | no serious risk of bias | no serious inconsistency | serious | no serious imprecision | none | 177/67767 (0.26%) |
370/48168 (0.77%) |
RR 0.51 (0.42 to 0.62) | 4 fewer per 1000 (from 3 fewer to 4 fewer) | ⊕⊕⊕⊚ MODERATE |
Two year cumulative risk of typhoid fever combined | |||||||||||
4 | randomised trials | no serious risk of biasAppendix 5 Footnote 1 | no serious inconsistency | seriousAppendix 5 Footnote 2 | no serious imprecision | none | 100/59969 (0.17%) |
237/60395 (0.39%) |
RR 0.43 (0.34 to 0.54) | 2 fewer per 1000 (from 2 fewer to 3 fewer) | ⊕⊕⊕⊚ MODERATE |
Three year cumulative risk of typhoid fever Ty21a | |||||||||||
3 | randomised trials | no serious risk of biasAppendix 5 Footnote 1 | no serious inconsistency | seriousAppendix 5 Footnote 2 | no serious imprecision | none | 147/62075 (0.24%) |
304/42476 (0.72%) |
RR 0.53 (0.43 to 0.65) | 3 fewer per 1000 (from 3 fewer to 4 fewer) | ⊕⊕⊕⊚ MODERATE |
Two year cumulative risk of typhoid fever Ty21a | |||||||||||
1 | randomised trials | no serious risk of biasAppendix 5 Footnote 3 | no serious inconsistency | seriousAppendix 5 Footnote 2 | no serious imprecision | none | 15/22170 (0.07%) |
44/21906 (0.2%) |
RR 0.34 (0.19 to 0.61) | 1 fewer per 1000 (from 1 fewer to 2 fewer) | ⊕⊕⊕⊚ MODERATE |
Three year cumulative risk of typhoid fever Vi polysaccharide | |||||||||||
1 | randomised trials | no serious risk of bias | no serious inconsistency | seriousAppendix 5 Footnote 2 | no serious imprecision | none | 30/5692 (0.53%) |
66/5692 (1.2%) |
RR 0.45 (0.30 to 0.7) | 6 fewer per 1000 (from 3 fewer to 8 fewer) | ⊕⊕⊕⊚ MODERATE |
Two year cumulative risk of typhoid fever Vi polysaccharide | |||||||||||
3 | randomised trials | no serious risk of bias | no serious inconsistency | seriousAppendix 5 Footnote 2 | no serious imprecision | none | 85/37799 (0.22%) |
193/38489 (0.5%) |
RR 0.45 (0.35 to 0.58) | 3 fewer per 1000 (from 2 fewer to 3 fewer) | ⊕⊕⊕⊚ MODERATE |
Cumulative risk of typhoid fever by age Ty21a | |||||||||||
2 | randomised trials | no serious risk of biasAppendix 5 Footnote 1 | no serious inconsistency | seriousAppendix 5 Footnote 2 | no serious imprecision | none | 86/56864 (0.15%) |
98/32206 (0.3%) |
RR 0.45 (0.33 to 0.63) | 2 fewer per 1000 (from 1 fewer to 2 fewer) | ⊕⊕⊕⊚ MODERATE |
Cumulative risk of typhoid fever by age Ty21a - Age 5-<10 | |||||||||||
2 | randomised trials | no serious risk of biasAppendix 5 Footnote 1 | no serious inconsistency | seriousAppendix 5 Footnote 2 | no serious imprecision | none | 54/28162 (0.19%) |
40/13182 (0.3%) |
RR 0.61 (0.39 to 0.97) | 1 fewer per 1000 (from 0 fewer to 2 fewer) | ⊕⊕⊕⊚ MODERATE |
Cumulative risk of typhoid fever by age Ty21a - Age ≥ 10 | |||||||||||
2 | randomised trials | no serious risk of biasAppendix 5 Footnote 1 | no serious inconsistency | seriousAppendix 5 Footnote 2 | no serious imprecision | none | 32/28702 (0.11%) |
58/19024 (0.3%) |
RR 0.34 (0.21 to 0.54) | 2 fewer per 1000 (from 1 fewer to 2 fewer) | ⊕⊕⊕⊚ MODERATE |
Cumulative risk of typhoid fever by age Vi polysaccharide | |||||||||||
2 | randomised trials | no serious risk of bias | no serious inconsistency | seriousAppendix 5 Footnote 2 | no serious imprecision | none | 64/32107 (0.2%) |
145/32797 (0.44%) |
RR 0.45 (0.34 to 0.6) | 2 fewer per 1000 (from 2 fewer to 3 fewer) | ⊕⊕⊕⊚ MODERATE |
Cumulative risk of typhoid fever by age Vi polysaccharide - Age 2-<5 | |||||||||||
2 | randomised trials | no serious risk of bias | no serious inconsistency | seriousAppendix 5 Footnote 2 | no serious imprecision | none | 21/4251 (0.49%) |
40/4419 (0.91%) |
RR 0.54 (0.32 to 0.91) | 4 fewer per 1000 (from 1 fewer to 6 fewer) | ⊕⊕⊕⊚ MODERATE |
Cumulative risk of typhoid fever by age Vi polysaccharide - Age ≥ 5 | |||||||||||
2 | randomised trials | no serious risk of bias | no serious inconsistency | seriousAppendix 5 Footnote 2 | seriousAppendix 5 Footnote 4 | none | 43/27856 (0.15%) |
105/28378 (0.37%) |
RR 0.42 (0.29 to 0.59) | 2 fewer per 1000 (from 2 fewer to 3 fewer) | ⊕⊕⊚⊚ LOW |
|
Appendix 6:
Outcomes | Illustrative comparative risksAppendix 6 Footnote * (95% CI) | Relative effect (95% CI) |
No of Participants (studies) |
Quality of the evidence (GRADE) |
|
---|---|---|---|---|---|
Assumed risk | Corresponding risk | ||||
Control | Typhoid vaccine | ||||
Three year cumulative risk of typhoid fever combined |
8 per 1000 |
4 per 1000 |
RR 0.51 |
115935 |
⊕⊕⊕⊚ |
Two year cumulative risk of typhoid fever combined |
4 per 1000 |
2 per 1000 |
RR 0.43 |
120364 |
⊕⊕⊕⊚ |
Three year cumulative risk of typhoid fever Ty21a |
7 per 1000 |
4 per 1000 |
RR 0.53 |
104551 |
⊕⊕⊕⊚ |
Two year cumulative risk of typhoid fever Ty21a |
2 per 1000 |
1 per 1000 |
RR 0.34 |
44076 |
⊕⊕⊕⊚ |
Three year cumulative risk of typhoid fever Vi polysaccharide |
12 per 1000 |
5 per 1000 |
RR 0.45 |
11384 |
⊕⊕⊕⊚ |
Two year cumulative risk of typhoid fever Vi polysaccharide |
5 per 1000 |
2 per 1000 |
RR 0.45 |
76288 |
⊕⊕⊕⊚ |
Cumulative risk of typhoid fever by age Ty21a |
3 per 1000 |
1 per 1000 |
RR 0.45 |
89070 |
⊕⊕⊕⊚ |
Cumulative risk of typhoid fever by age Ty21a - Age 5-<10 |
3 per 1000 |
2 per 1000 |
RR 0.61 |
41344 |
⊕⊕⊕⊚ |
Cumulative risk of typhoid fever by age Ty21a - Age ≥ 10 |
3 per 1000 |
1 per 1000 |
RR 0.34 |
47726 |
⊕⊕⊕⊚ |
Cumulative risk of typhoid fever by age Vi polysaccharide |
4 per 1000 |
2 per 1000 |
RR 0.45 |
64904 |
⊕⊕⊕⊚ |
Cumulative risk of typhoid fever by age Vi polysaccharide - Age 2-<5 |
9 per 1000 |
5 per 1000 |
RR 0.54 |
8670 |
⊕⊕⊕⊚ |
Cumulative risk of typhoid fever by age Vi polysaccharide - Age ≥ 5 |
4 per 1000 |
2 per 1000 |
RR 0.42 |
56234 |
|
GRADE Working Group grades of evidence
|
Appendix 7:
No of studies | DesignAppendix 7 Footnote 1 | Risk of bias | Inconsistency | Indirectness | ImprecisionAppendix 7 Footnote 2 | Other considerations | Estimate (cases/ 100,000) |
Confidence in baseline estimate |
---|---|---|---|---|---|---|---|---|
Eastern Mediterranean | ||||||||
1 | Observational cohort | intermediate risk of bias | no serious inconsistency | no serious indirectness | intermediate risk of imprecision | none | 0.09 | ⊕⊕⊕⊚ MODERATE |
North Africa/Middle East | ||||||||
3 | Observational cohort | intermediate risk of bias | no serious inconsistency | no serious indirectness | intermediate risk of imprecision | none | 1.48 | ⊕⊕⊕⊚ MODERATE |
Sub-Saharan Africa | ||||||||
3 | Observational cohort | intermediate risk of bias | no serious inconsistency | no serious indirectness | intermediate risk of imprecision | none | 1.84 | ⊕⊕⊕⊚ MODERATE |
Caribbean, Central America | ||||||||
3 | Observational cohort | intermediate risk of bias | no serious inconsistency | no serious indirectness | intermediate risk of imprecision | none | 0.33 | ⊕⊕⊕⊚ MODERATE |
South America | ||||||||
2 | Observational cohort | intermediate risk of bias | no serious inconsistency | no serious indirectness | intermediate risk of imprecision | none | 0.95 | ⊕⊕⊕⊚ MODERATE |
East Asia | ||||||||
3 | Observational cohort | intermediate risk of bias | no serious inconsistency | no serious indirectness | intermediate risk of imprecision | none | 0.25 | ⊕⊕⊕⊚ MODERATE |
South Asia | ||||||||
4 | Observational cohort | intermediate risk of bias | no serious inconsistency | no serious indirectness | intermediate risk of imprecision | none | 32.73 | ⊕⊕⊕⊚ MODERATE |
Overall Risk of Bias: Moderate Selection Bias: Low: All included studies used mandatory infectious disease surveillance data of the reported cases of typhoid in the province or country as the numerator and a population based estimate of the travelling population for the same region as the denominator. Outcome Measurement: Low: All studies only included cases of typhoid from clinical specimens that were laboratory confirmed Outcome Ascertainment: Moderate: Potential biases in surveillance data are only the most severe cases seek medical attention and have the opportunity to be diagnosed. Furthermore different populations may have different health seeking behavior and some cases may have been missed due to diagnosis and treatment during travel.
|
Page details
- Date modified: