Weight of Evidence: General Principles and Current Applications at Health Canada

Prepared by the Task Force on Scientific Risk Assessment's Weight of Evidence Working Group

T. Tao,^{Footnote 1} Y. Bhuller,^{Footnote 2} Y. Bonvalot,^{Footnote 3} M. Hill,^{Footnote 4} A. Klein,^{Footnote 5} G. Kozak,^{Footnote 6} I. Plante,^{Footnote 7} and N. Robert^{Footnote 8}

Table of Contents

• 1. Introduction
• 2. Purpose and Scope
• 3. Role in Risk Assessments
• 4. General Principles
  • 4.1 Gathering “All” Available Evidence
  • 4.2 Assessing Individual Studies
  • 4.3 Assembling Lines of Evidence
  • 4.4 Assessing Lines of Evidence
  • 4.5 Integrating Multiple Lines of Evidence
• 5. Application at Health Canada
• 6. International Context
• 7. Conclusions
• 8. References
• Annex 1
• Annex 2
• Annex 3

1. Introduction

Weight of Evidence (WoE) is frequently cited as the basis on which risk assessment conclusions are made. However, multiple interpretations and a lack of consensus about its meaning could potentially compromise communication between diverse stakeholders in the decision-making process. In response to this issue, an analysis of the WoE approach was initiated by Health Canada's Science Policy Directorate in 2010, as a project under the Task Force on Scientific Risk Assessment. By examining current interpretations and identifying potential best practices, this analysis aims to enhance the consistency and coherence of risk assessments across the Department.

2. Purpose and Scope

The current document aims to inform senior management about WoE in Health Canada risk assessments by providing an overview of the approach in terms of its:

• role in scientific risk assessments;
• main guiding principles; and
• application by various risk assessment programs at Health Canada.

In addition, this explanatory document serves as a value-added Departmental resource of high level contextual information and guiding principles to supplement program specific guidelines, procedures and/or tools.

While this document acknowledges that WoE could also be applied in the risk management decision making context, where scientific evidence is weighed against other policy considerations, it will not expand on this information as it is considered not within the scope of this document.^{Footnote 9}

3. Role in Risk Assessments

In general, scientific risk assessments encompass the following steps: identifying and characterizing the hazard, assessing the exposure, and characterizing the risk; risk assessments also play an integrated role in an evidence- informed decision making process which also involves managing and communicating the risk.

WoE in the risk assessment context is defined in Health Canada Decision-Making Framework for Identifying, Assessing, and Managing Health Risks (Health Canada, 2000) as:

“A qualitative measure that takes into account the nature and quality of scientific studies intended to examine the risk of an agent. Uncertainties that result from the incompleteness and unavailability of scientific data frequently require scientists to make inferences, assumptions, and judgements in order to characterize a risk. Making judgements about risk based on scientific information is called “evaluating the weight of evidence”.

The above description can be interpreted to implicitly include two separate concepts frequently associated with WoE terminology:

1. Totality of Evidence: what types and sources of information are to be gathered and considered for subsequent assessment; and
2. Weighing Evidence: how such individual sources of evidence are assessed and integrated into an overall conclusion or recommendation.

Totality of evidence can be influenced by varying interpretations of “all” available or relevant evidence to date. This concept provides the opportunity to make use of information/studies that may be regarded insufficient individually, but which contribute to a total “weight of evidence” case in support of conclusions during risk assessment when they are considered alongside other studies/sources of evidence. Moreover, an evaluation of evidence and of any subsequent decision can be reassessed,  at a later date, based on the availability of data that may not have been readily available at the time  of the original assessment.

The latter, methodological concept of weighing evidence is applicable to most risk assessments. While specific methodologies and tools used for assessing and integrating evidence (e.g., quantitative or qualitative) may vary and are context dependent, the general principles for the assessment and integration process remain the same.

4. General Principles

The inter-relationship of the above two concepts, and the general principles of the WoE approach outlined below, is presented in Annex 1, for illustrative purposes only. The Totality of Evidence concept includes the general principles 4.1, 4.2, and 4.3, while the methodological concept of Weighing Evidence can be subdivided into the general principles of 4.4 and 4.5. Regardless of specific interpretations of terminology, the following steps are applicable to building a “weight of evidence” case for a given risk assessment conclusion or recommendation.

4.1 Gathering “All” Available Evidence

Multiple sources and types of evidence may be gathered or submitted and considered in context of “all” available evidence to date. Depending on the regulatory data requirements, the full spectrum of sources and types of evidence may include: randomized controlled clinical trials, company and/or third party generated studies of a proprietary nature, peer-reviewed, published scientific literature, expert opinion reports, decisions and analysis reports from regulatory authorities, incident reports, adverse reactions submitted to regulatory authorities, and unpublished data.

4.2 Assessing Individual Studies

General criteria for inclusion/exclusion are useful when screening “all” evidence gathered for further consideration. While specific terminology and scope for inter-related screening criteria such as “quality”, “reliability”, “relevance”, etc. could differ across various regulatory programs and agencies, the underlying principles are common. Assessment could involve use of specific scoring tools and/or best professional judgement. Acceptable studies that meet standards for inclusion are assessed further in subsequent steps of the WoE approach, while unacceptable studies may be excluded from further consideration. For example, unpublished data, or data irrelevant to the risk assessment endpoint in question, may be excluded from further consideration or may be given a lower weight when assembling the lines of evidence. When necessary, the rationale for including (or excluding) studies could be documented in the relevant report.

4.3 Assembling Lines of Evidence

The types and sources of evidence considered are diverse and vary considerably in level of detail. Depending on the context of the risk assessment in question, individual studies or data sources are often assessed as distinct lines of evidence on their own, or considered in concert with other similar studies that together constitute a particular “line of evidence”. Such lines can be organized according to unifying characteristics, such as source or type of data (e.g., animal data, human data, clinical trials, and literature data). Separate lines of evidence can also be drawn along sub- components of risk, such as hazard, exposure, human health, environmental safety, or other characteristics such as studies which support or counter a particular conclusion. These lines can be further subdivided into more specific lines. For example, “hazard” can be divided into specific organ systems (hazard to the liver, kidneys, brain, etc.).

4.4 Assessing Lines of Evidence

Lines of evidence are assessed against various criteria that are dependent on the context of the particular endpoint in question. Risk assessments can be hypothesis driven, and designed to answer yes/no questions (e.g., is substance x a carcinogen?). In such instances, several lines of evidence (e.g., carcinogenicity studies, genotoxicity studies, or mechanistic data) can each be assessed based on criteria such as the strength/robustness of evidence in support of, or against, a given conclusion for each particular line.

Other risk assessments can address more general questions (e.g., what product/source is the likely cause of illness outbreak y?). In such instances, some lines of evidence, such as epidemiological data, can be assessed based on specific criteria such as strength of association, consistency, specificity, temporality, biological gradient/ dose-response, plausibility, coherence, experimental evidence, and analogy (e.g., the Bradford Hill (1965) criteria for causal inference). Depending on the context of the particular line of evidence involved, other criteria not described here could also be applicable. The assessment can be quantitative, by assigning a weight or value to each line of evidence assessed, in the form of probabilities, alphanumeric values, or qualitative by descriptions such as “weak” or “strong”, or implicit, in the form of logic models and decision trees that by default emphasize the importance of certain lines of evidence over others.

Assigned values or descriptions reflect the relative “strength” of a particular line of evidence, which is negatively impacted by the uncertainty and variability in datasets contributing to each line of evidence. Departmental documents elaborating on uncertainty and/or variability include, but are not limited to: A Framework for the Application of Precaution in Science-based Decision Making about Risk (Privy Council Office, 2003) and the Health Products and Food Branch’s Guide for Conducting Health Risk Assessments in Humans (Health Canada, 2011).

4.5 Integrating Multiple Lines of Evidence

The determination of the relative contributions of various lines of evidence to the overall conclusion can be performed in a single step, qualitative process, using best professional judgment. More systematic methods of quantitative integration can also be employed, where scores for individual lines of evidence may be adjusted by weighting factors that reflect the relative importance of a line within the overall body of evidence, and then mathematically integrated into a final value. However, scoring is not easily applicable in a context such as risk assessment, due to the large complexity of the different sources of information available.

The integration of values/weights is an iterative process that is repeated at many levels: within individual studies, across similar studies into a collective value for a particular line of evidence, and across multiple lines of evidence into an overall risk assessment conclusion or recommendation. For example, to determine whether a compound affects the liver, one collectively examines and integrates clinical chemistry findings along with organ weight and histopathology data within a single study, or across multiple similar studies (e.g., to assess dose-response). For integration across collection of studies for a given assessment endpoint (e.g., whether a compound is carcinogenic) one can collectively examine and integrate carcinogenicity studies, genotoxicity studies and mechanistic data. For conclusions regarding overall risk, it is necessary to integrate lines of evidence related to hazard and exposure. Further integration of human health risk and environmental risk may contribute to an overall risk profile.

5. Application at Health Canada

Assessment of scientific evidence is a crucial component of risk assessment and decision making at Health Canada. Moreover, for many of the regulatory programs in the Department, risk assessment conclusions (referred to as risk characterization) are often made based on the likelihood of association between a particular substance/activity and associated health effects. In this context, a WoE approach  is frequently cited as the basis on which conclusions are made using the best available information to date that can be gathered, assessed, and integrated using various qualitative and quantitative methods.

The mandate and scope of risk assessment and/or risk management activities of the various programs vary significantly across the Department (see A Primer on Scientific Risk Assessment at Health Canada, Saner, 2010). Each program operates within the constraints of program-specific legislation. Differences in legislation and program goals impact time available for assessment of each particular product or activity, the amount and quality of information that is available to date for assessment, and the degree of flexibility in interpretation and application of WoE as a risk assessment approach. Each program is also impacted by international guidelines for specific subject areas and the sector-specific context in which regulations may be often harmonized globally. The varying issues and the context under which regulatory decisions are made, and the scope of potential risk management options and recommendations that can be explored also differ across and within programs.

A survey was conducted to determine how WoE was applied across the department. All branches surveyed responded, including the Health Products and Food Branch (HPFB), the Healthy Environments and Consumer Safety Branch (HECSB), and the Pesticide Management Regulatory Agency (PMRA). The general principles of the WoE approach are applied by most programs surveyed. Specifically, most risk assessments follow the steps of gathering and assessing individual studies and sources of evidence, assembling studies into context specific lines of evidence, and assessing and integrating multiple lines of evidence into an overall conclusion or recommendation. Most programs interpret WoE to include concepts such as the totality of evidence (i.e., the evidence to be gathered and considered), as well as the weighing of evidence (i.e., how such evidence is assessed and integrated into a final conclusion) (see Annex 2).

The application of specific criteria and tools are context specific, and are outlined in various program specific guidelines, standard operating procedures, working documents, etc. Program documents outlining application of the WoE approach have been specifically developed for such purposes when considered necessary. For example:

• Weight of Evidence: Factors to Consider for Appropriate and Timely Action in a Foodborne Illness Outbreak Investigation (Health Canada, Public Health Agency of Canada, and Canadian Food Inspection Agency, 2011);
• Framework for Initiating and Conducting Risk Analysis Activities on Microbial Hazards in Food (Health Canada, 2017);
• Food Investigation Response Manual (Canadian Food Inspection Agency, 2017);
• Science Policy Note: General Exposure Factor Inputs for Dietary, Occupational, and Residential Exposure Assessments (Health Canada, 2014)
• Federal Contaminated Site Risk Assessment in Canada: Supplemental Guidance on Human Health Risk Assessment for Country Foods (HHRAFoods) (Health Canada, 2010);
• Notice to Product License Applicants—Traditional Claim Submissions: Evidence Criteria and Evidence Assessment Template (Natural and Non-prescription Health Products Directorate, 2010);
• Pathway for Licensing Natural Health Products used as Traditional Medicines (Natural and Non-prescription Health Products Directorate, 2012a)
• Pathway for Licensing Natural Health Products Making Modern Health Claims (Natural and Non-prescription Health Products Directorate, 2012b)

Program documents of a more general nature include:

• Health Products and Food Branch's Guide for Conducting Health Risk Assessments in Humans (Health Canada, 2011);
• Framework for Science-Based Risk Assessment of Micro-Organisms Regulated under the Canadian Environmental Protection Act, 1999 (Environment Canada, Health Canada, 2013);
• All Hazards Risk Assessment Methodology Guidelines 2012–2013 (Public Safety Canada 2018)

As mentioned above, documentation on how the risk assessment is conducted and the rationale  for either including or excluding certain sources of evidence is a critical component of the decision making process. Similarly, while the WoE approach is consistently applied in most risk assessments across the Department, explicit use of WoE terminology is not always documented. In some instances, WoE terminology is used, but the specific application of the WoE approach is not elaborated.

On occasion, WoE terminology is used when actually referring to levels of evidence or standards of quality of individual studies. In some instances, WoE terminology is also used in place of actual descriptions of the strength/robustness of overall conclusions/recommendations, or in place of legal terms such as preponderance of evidence, which simply means more likely than not.

The majority of risk assessment reports, however, provide a logical narrative description of the  relative strengths or weaknesses of various lines of evidence considered. For most risk assessments, individual lines of evidence are pooled and integrated into a final conclusion based on best professional judgment, and not mathematical formula. Narrative descriptions of the rationale for such judgments are usually provided, including explanations of how certain lines of evidence are more important than others in determining the overall risk assessment conclusion/recommendation. Some reports, however, simply list lines of evidence assessed and proceed directly to the overall risk assessment conclusion, without explicit documentation of how the multiple lines of evidence relate to one another, or the rationale  behind the integration process.

6. International Context

The WoE approach is routinely applied by most scientific risk assessment agencies internationally and while several definitions for WoE exist, there is no single, universal standardized/commonly agreed upon definition or specific guidance on how to implement a WoE approach. For example, recent guidance on the use of the WoE approach and “totality of evidence” has been published by the European Food Safety Authority’s Scientific Committee (EFSA, 2017a), which stated that “weight of evidence assessment is a process in which evidence is integrated to determine the relative support for possible answers to a scientific question. The term ‘weight of evidence’ on its own is the extent to which evidence supports possible answers to a scientific question.”

The United States Environmental Protection Agency (EPA, 2003) outlines WoE in various guidelines, in both the totality of evidence context, and the methodological context of the weighing of multiple lines of evidence, e.g.:

“The weight-of-evidence approach considers all relevant information in an integrative assessment that takes into account the kinds of evidence available, the quality and quantity of the evidence, the strengths and limitations associated with each type of evidence and explains how the various types of evidence fit together.”

However, in a review of the EPA’s Integrated Risk Information System (IRIS) process, the National Research Council (2014) found that:

“systematic review and weight-of-evidence analysis have historically been described in various ways, and the terms are sometimes used interchangeably; this vagueness in use of terminology results in some confusion as to what the terms mean in practice… The committee views weight-of-evidence analysis as a judgment-based process for evaluating the strength of evidence to infer causation. However, it found that the phrase as used in practice has become too vague and is of little scientific use. An IRIS assessment must come to a judgment about whether a chemical is hazardous to human health and must do so by integrating a variety of lines of evidence. Therefore, the committee found the term evidence integration to be more useful and more descriptive of the process that occurs  after completion of systematic reviews.”

Similarly, the U.S. Environmental Protection Agency’s National Center for Environmental Assessment (2015) takes an integrated approach to science assessments for reviews of national ambient air quality standards:

“The U.S. EPA integrates the evidence from across scientific disciplines or study types and characterizes the weight of evidence for relationships… drawing upon the results of all studies judged of adequate quality and relevance per the criteria… consider aspects, such as strength, consistency, coherence, and biological plausibility of the evidence, and develop causality determinations on the nature of the relationships… includes evaluating strengths and weaknesses in the overall collection of studies across disciplines.”

The European Chemicals Agency (ECHA, 2011 and 2016) outlines interpretations regarding the methodological context of weighing evidence as follows:

“The weight of evidence approach commonly refers to combining evidence from multiple sources to assess a property under consideration. It can therefore be a useful technique where, for example, each piece of information or test alone is not sufficient to address a standard information requirement but where it may be possible to combine the strengths and weaknesses of the individual studies to reach a conclusion for a particular property.

The term weight of evidence (WoE) is neither a scientifically well-defined term nor an agreed formalised concept characterised by defined tools and procedures. It can, however, be regarded  as an evidence-based approach involving an assessment of the relative weights (values) of different pieces of the available information that have been gathered. Application of this concept can be achieved either in an objective way by using a formalised procedure or by using expert judgement. Factors such as the quality of the data, consistency of results, nature and severity of effects, relevance of the information will have an influence on the weight given to the available evidence.”

This concept of weighing evidence is supplemented by the totality of evidence concept within the Regulation on Registration, Evaluation, Authorisation and Restriction of Chemicals (ECHA, 2017):

“There may be sufficient weight of evidence from several independent sources of information leading to the assumption/conclusion that a substance has or has not a particular dangerous property, while the information from each single source alone is regarded insufficient to support this notion.

There may be sufficient weight of evidence from the use of newly developed test methods, not yet included in the test methods referred to in Article 13(3) or from an international test method recognised by the Commission or the Agency as being equivalent, leading to the conclusion that a substance has or has not a particular dangerous property.

Where sufficient weight of evidence for the presence or absence of a particular dangerous property is available:

• further testing on vertebrate animals for that property shall be omitted,
• further testing not involving vertebrate animals may be omitted. 

In all cases adequate and reliable documentation shall be provided.”

The World Health Organization’s International Programme on Chemical Safety has published two guidance documents regarding uncertainty in risk assessment: Uncertainty and Data Quality in Exposure Assessment (WHO, 2008), which explicitly addresses WoE: “to the extent possible, the combined effect of different sources of uncertainty on the exposure or risk predictions, perhaps based on a weight-of-evidence methodology in the absence of quantitative data, should also be considered”, and a Guidance Document on Evaluating and Expressing Uncertainty in Hazard Characterization (WHO, 2017).

The Food and Agriculture Organization of the United Nations and the World Health Organization (FAO/ WHO, 2009) discussed using a WoE approach to the risk characterization of microbiological hazards in food: “the weight of evidence should be evaluated according to clearly specified, scientific criteria. As more criteria are satisfied, the weight of evidence indicates a more credible risk.” FAO/WHO anticipated that “weight-of-evidence determinations will become increasingly prominent in risk assessments of microbiological pathogens in food.”

The Organisation for Economic Co-operation and Development (OECD, 2015, 2017, 2018) defines WoE as “a comprehensive, integrated, often qualitative judgment of the extent and quality of information supporting an hypothesis for which the approaches and tools vary, depending on the context.” WoE methodology is used in their “Adverse Outcome Pathway (AOP)/Mode Of Action (MOA)” framework for the development and use of “Integrated Approaches to Testing and Assessment” (IATA):

“Evaluation of existing information or generation of additional data within an IATA can be performed on the basis of a non-formalised Weight of Evidence (WoE) approach or by using predefined, structured approaches such as Sequential Testing Strategies (STS), Integrated Testing Strategies (ITS) or their combination.”

In considering the use of a WoE approach, Codex Alimentarius (2014) cautions that “The weight of evidence integrating quantitative and qualitative data may permit only a qualitative estimate of risk.”

Taken together, the above definitions from key international partners are consistent with current Health Canada interpretations of the WoE approach.

7. Conclusions

While specific tools and methodologies are often context-specific to particular program areas, the underlying principles of the WoE approach, in which multiple sources of information are gathered, assessed, and integrated into an overall conclusion, are commonly applied across the Department, and are judged to be consistent with international practice.

Presently, inconsistencies occur not in the high level applications of the overall WoE approach. Rather, they result when WoE terminology is applied when actually dealing with standards of quality of individual studies or strength of overall conclusions/recommendations.

Given the context specific nature of each risk assessment and the diversity of tools and criteria applicable, transparent documentation of the specific application of the WoE approach is especially important. There are opportunities for harmonization, and adherence to a simple checklist is a step towards this goal (see Annex 3). Program areas are encouraged to take the relevant steps (e.g., updating internal guidelines) to further improve the documentation aspect in reports that provide the risk assessment in support of subsequent risk management options/regulatory decision, which includes elaborating on what is meant by WoE, when necessary. Additionally, graphically based evidence maps, profiles, or tables may be helpful as supplementary tools for communication from risk assessors to risk managers.

8. References

• Bradford Hill A. 1965. The environment and disease: association or causation? Proc R Soc Med. 58(5): 295–300.
• Canadian Food Inspection Agency. 2017. Food Investigation Response Manual. Available at: www.inspection.gc.ca/ food/safe-food-production-systems/food-recall-and-emergency-response/food-manual/eng/1378402475724/ 1378403080658?chap=0 (accessed 2018-03-16)
• Canada Privy Council Office. 2003. A Framework for the Application of Precaution in Science-based Decision Making about Risk. Available at: http://publications.gc.ca/collections/Collection/CP22-70-2003E.pdf (accessed 2018-03-15)
• Codex Alimentarius. 2014. Principles and Guidelines for the Conduct of Microbiological Risk Assessment (CAC/GL 30-1999). Available at: www.fao.org/fao-who-codexalimentarius/codex-texts/guidelines/en/ (accessed 2018-03-20)
• ECHA: European Chemicals Agency. 2011. Guidance on information requirements and chemical safety assessment— Chapter R.4: Evaluation of available information. Available at: https://echa.europa.eu/documents/10162/13643/ information_requirements_r4_en.pdf/d6395ad2-1596-4708-ba86-0136686d205e  (accessed  2018-03-19).
• ECHA. 2016. Practical guide: How to use alternatives to animal testing to fulfil your information requirements for REACH  registration.  Available  at: https://echa.europa.eu/documents/10162/13655/practical_guide_how_to_use_ alternatives_en.pdf/148b30c7-c186-463c-a898-522a888a4404 (accessed  2018-03-19)
• ECHA. 2017. Consolidated version of the REACH Regulation: Regulation (EC) No 1907/2006 of the European Parliament  and of the Council of 18 December 2006 concerning the Registration, Evaluation, Authorisation and Restriction of Chemicals (REACH). Available at: https://echa.europa.eu/regulations/reach/legislation (accessed 2018-03-19)
• Environment Canada, Health Canada. 2013. Framework for Science-Based Risk Assessment of Micro-Organisms Regulated under the Canadian Environmental Protection Act, 1999. Available at: www.ec.gc.ca/subsnouvelles-newsubs/ default.asp?lang=En&n=120842D5-1 (accessed 2018-03-16))
• EFSA: European Food Safety Authority. 2017a. Guidance on the use of the weight of evidence approach in scientific assessments. EFSA Journal 2017;15(8):4971. Available at: http://onlinelibrary.wiley.com/doi/10.2903/j.efsa.2017.4971/epdf (accessed 2018-03-15)
• EFSA. 2017b. Revised Draft for Internal Testing: Guidance on Uncertainty in EFSA Scientific Assessment. Available at: www.efsa.europa.eu/sites/default/files/160321DraftGDUncertaintyInScientificAssessment.pdf    (accessed    2018-03-15)
• EPA: United States Environmental Protection Agency. 2003. A Summary of General Assessment Factors for Evaluating the Quality of Scientific and Technical Information. Available at: www.epa.gov/sites/production/files/2015-01/documents/ assess2.pdf (accessed 2018-03-19)
• EPA: United States Environmental Protection Agency. 2015. Preamble to the Integrated Science Assessments (ISA). Available at: https://cfpub.epa.gov/ncea/isa/recordisplay.cfm?deid=310244 (accessed 2018-03-29)
• FAO/WHO: Food and Agriculture Organization of the United Nations and the World Health Organization. 2009. Risk characterization of microbiological hazards in food: Guidelines. Microbiological Risk Assessment Series 17. Available at: www.fao.org/docrep/012/i1134e/i1134e.pdf (accessed 2018-03-20)
• Health Canada. 2000. Health Canada Decision-Making Framework for Identifying, Assessing, and Managing Health Risks. Available at: www.canada.ca/content/dam/hc-sc/migration/hc-sc/ahc-asc/alt_formats/hpfb-dgpsa/pdf/pubs/ risk-risques-eng.pdf (accessed 2018-03-15)
• Health Canada. 2010. Federal Contaminated Site Risk Assessment in Canada: Supplemental Guidance on Human Health Risk Assessment for Country Foods (HHRAFoods). Available at: http://publications.gc.ca/collections/collection_2012/ sc-hc/H128-1-11-641-eng.pdf  (accessed  2018-03-16)
• Health Canada. 2011. Health Products and Food Branch's Guide for Conducting Health Risk Assessments in Humans. Available at: http://mysource.hc-sc.gc.ca/sites/default/files/hpfbs_guide_for_conducting_hras_in_humans_e_final.pdf (accessed 2018-03-15)
• Health Canada. 2014. Science Policy Note: General Exposure Factor Inputs for Dietary, Occupational, and Residential Exposure Assessments. Available at: www.canada.ca/content/dam/hc-sc/migration/hc-sc/cps-spc/alt_formats/ pdf/pubs/pest/pol-guide/spn2014-01/spn2014-01-eng.pdf (accessed 2018-03-16)
• Health Canada. 2017. Framework for Initiating and Conducting Risk Analysis Activities on Microbial Hazards in Food. Cat.: H164-198/2017E-PDF, ISBN: 978-0-660-06811-4, Pub.: 160231. Available from publications@hc-sc.gc.ca
• Health Canada, Public Health Agency of Canada, Canadian Food Inspection Agency. 2011. Weight of Evidence: Factors to Consider for Appropriate and Timely Action in a Foodborne Illness Outbreak Investigation. Available at: www.canada.ca/content/dam/hc-sc/migration/hc-sc/fn-an/alt_formats/pdf/pubs/securit/2011-food-illness- outbreak-eclosion-malad-ailments-eng.pdf  (accessed  2018-03-16)
• National Research Council. 2014. Review of EPA's Integrated Risk Information System (IRIS) Process. Committee to Review the IRIS Process; Board on Environmental Studies and Toxicology; Division on Earth and Life Studies. National Academies of Sciences, Engineering, Medicine – National Academies Press. Available at: http://nap.edu/18764 (accessed 2018-03-19)
• Natural and Non-prescription Health Products Directorate, Health Canada. 2010. Notice to Product License Applicants – Traditional Claim Submissions: Evidence Criteria and Evidence Assessment Template. Available at: www.canada.ca/en/ health-canada/services/drugs-health-products/natural-non-prescription/legislation-guidelines/guidance-documents/ traditional-claim-submissions-evidence-criteria-evidence-assessment-template.html    (accessed    2018-03-16)
• Natural and Non-prescription Health Products Directorate, Health Canada. 2012a. Pathway for Licensing Natural Health Products used as Traditional Medicines. Available at: www.canada.ca/en/health-canada/services/drugs-health-products/ natural-non-prescription/legislation-guidelines/guidance-documents/pathway-licensing-traditional-medicines.html (accessed 2018-03-16)
• Natural and Non-prescription Health Products Directorate, Health Canada. 2012b. Pathway for Licensing Natural Health Products Making Modern Health Claims. Available at: www.canada.ca/en/health-canada/services/drugs-health-products/ natural-non-prescription/legislation-guidelines/guidance-documents/pathway-licensing-making-modern-health-claims.html (accessed 2018-03-16)
• OECD: Organisation for Economic Co-operation and Development. 2015. Report of the Workshop on a Framework for the Development and Use of Integrated Approaches to Testing and Assessment. Series on Testing and Assessment No. 215, ENV/JM/MONO(2015)22. Available at: www.oecd.org/officialdocuments/publicdisplaydocumentpdf/?cote=ENV/JM/ MONO(2015)22&doclanguage=en (accessed 2018-03-20)
• OECD. 2017. Revised Guidance Document on Developing and Assessing Adverse Outcome Pathways. Series on Testing & Assessment No. 184, ENV/JM/MONO(2013)6. Available at: www.oecd.org/officialdocuments/publicdisplaydocumentpdf/ ?cote=env/jm/mono(2013)6&doclanguage=en (accessed  2018-03-20)
• OECD. 2018. Users' Handbook Supplement to the Guidance Document for Developing and Assessing AOPs. Series  on Testing & Assessment No. 233, Series on Adverse Outcome Pathways No. 1, ENV/JM/MONO(2016)12. Available at: https://one.oecd.org/document/ENV/JM/MONO(2016)12/en/pdf   (accessed   2018-03-20)
• Public Safety Canada. 2018. All Hazards Risk Assessment Methodology Guidelines 2012-2013. Available at: www.publicsafety.gc.ca/cnt/rsrcs/pblctns/ll-hzrds-ssssmnt/index-en.aspx (accessed 2018-03-16)
• Saner M. 2010. A Primer on Scientific Risk Assessment at Health Canada. HC Pub.: 100140; Cat.: H22-4/3-2010; ISBN: 978-1-100-15377-3. Available at: www.canada.ca/content/dam/hc-sc/migration/hc-sc/sr-sr/alt_formats/pdf/pubs/ about-apropos/2010-scientif-ris-eng.pdf  (accessed  2018-03-16)
• WHO (World Health Organization). 2008. International Programme on Chemical Safety: Uncertainty and Data Quality in Exposure  Assessment.  Available  at: www.who.int/ipcs/publications/methods/harmonization/exposure_assessment.pdf?ua=1 (accessed 2018-03-15)
• WHO. 2017. International Programme on Chemical Safety: Guidance Document on Evaluating and Expressing Uncertainty in Hazard Characterization. Available at: http://apps.who.int/iris/bitstream/10665/259858/1/9789241513548-eng.pdf?ua=1 (accessed 2018-03-15)

Annex 1

Graphical Representation of the Weight of Evidence Approach in the Context of Scientific Risk Assessments
Totality of Evidence: conceptual interpretations of scope and nature of evidence for consideration

1. Gathering "All" Evidence

Text description

Step 1 depicts the gathering of all evidence, which is represented by 7 multi-coloured shapes, or ‘evidence’.

2. Assessing Individual Studies (for ‘quality’, ‘relevance’, etc…)

(*Note: Individual studies may not meet standards for all criteria, but still could be considered if contributing to a weight of evidence case when pooled with other studies.)

Text description

Step 2 depicts the process of assessing individual studies by retaining certain evidence (using an arrow labelled as ‘inclusion’) and removing two of the multi-coloured shapes from evidence (using an arrow labelled as ‘exclusion’).

3. Assembling lines of evidence (context specific)

Text description

Step 3 depicts three separate ‘lines of evidence’ by categorizing the multi-coloured shapes into either Line 1, 2 or 3 of evidence.

Weighing Evidence: methodological approaches for assessing and integrating evidence into an overall conclusion

4. Assessing Lines of Evidence (for causality, hazard, exposure, etc…)

5. Integrating Multiple Lines of Evidence

Example 1: Weight of Evidence Summary for Foodborne Illness Outbreak^{Footnote a}

Lines of Evidence	Criteria Assessed	Qualitative Description
Line 1 (Laboratory Investigation: Strength of Microbiological Evidence: Determining the Relevance of Food and Clinical Isolates that Match by Pulsed-Field Gel Electrophoresis (PFGE)*)	Does the organism show suitable diversity by PFGE? Are clinical and food isolate PFGE patterns indistinguishable by enzymes? What is the historic frequency of the PFGE pattern combination? Are other subtyping results available; are they consistent with PFGE?	(Weak…Strong)
Line 2 (Food Safety Investigation: Traceback and Trace Forward)	Does the product identity allow direct tracing to the manufacturer? Can the point of purchase be identified? Can the product identity lead to manufacturer? Distribution channel Can the product be traced back to the wholesaler?	(Weak…Strong)
Line 3 (Epidemiological Investigation)	Plausibility Consistency Specificity Strength of the association Temporal(ity) Dose-Response Etc..	(Weak…Strong)
Overall conclusion	The WoE case for product Y as source of contamination for Foodborne Illness Outbreak X is (Weak…Strong)
Annex 1 table 1 Footnote a Weight of Evidence: Factors to Consider for Appropriate and Timely Action in a Foodborne Illness Outbreak Investigation. Health Canada, Public Health Agency of Canada, and Canadian Food Inspection Agency. January 2011 Annex 1 table 1 Return to footnote a referrer

Example 2: Weight of Evidence Summary for neurobehavioural effects of bisphenol-A at low doses in rodents (below reproductive/developmental NOAEL)^{Annex 1 table 2 Footnote a}

Criteria Assessed	Qualitative Description
Rigour of overall dataset	limited
Power of overall dataset	limited
Corroboration/consistency	limited
Biological Plausibility/coherence	limited
Overall weight of evidence for neurodevelopmental effects in rodents at low level exposures (exposures below the NOAEL for reproductive/developmental toxicity)	limited
Annex 1 table 2 Footnote a Final Screening Assessment for the Phenol, 4,4'-(1-methylethylidene)bis-(Bisphenol A). Chemical Abstracts Service Registry Number 80-05-7. Environment Canada and Health Canada. October 2008. Annex 1 table 2 Return to footnote a referrer

Annex 2

Working level interpretation(s)/application(s) of Weight of Evidence

Program	Totality of EvidenceAnnex 2 table 1 Footnote *: Conceptual interpretations of the nature and scope of evidence sources for consideration		Weighing EvidenceAnnex 2 table 1 Footnote **: Methodological approaches for the assessment and integration of multiple lines of evidence to derive at a final conclusion
	Consideration of all available lines of evidence to date, as opposed to a subset of data	Consideration of studies that individually may not meet standards for all criteria, but contributing to a weight of evidence case when pooled with other studies	Qualitative (e.g., listing, best professional judgment)	Semi-quantitative (e.g., causal criteria, logic models, alphanumeric scoring or indexing)	Quantitative (e.g., probabilistic tools or Multi- Criteria Decision Analysis [MCDA])
Healthy Environments and Consumer Safety Branch (HECSB)
WAQBAnnex 2 table 1 Footnote 1 / Water
WAQB / Air
New Substances – NSACBAnnex 2 table 1 Footnote 2
Existing Substances – ESRABAnnex 2 table 1 Footnote 3
ERHSDAnnex 2 table 1 Footnote 4
CPSDAnnex 2 table 1 Footnote 5
Health Products and Food Branch (HPFB)
TPDAnnex 2 table 1 Footnote 6
BGTDAnnex 2 table 1 Footnote 7 / Biologics
MHPDAnnex 2 table 1 Footnote 8
NNHPDAnnex 2 table 1 Footnote 9
VDDAnnex 2 table 1 Footnote 10
FDAnnex 2 table 1 Footnote 11 / Novel Foods
FD / Nutrition Labelling and Claims
FD / Microbial Hazards
Pesticide Management Regulatory Agency (PMRA)
HEDAnnex 2 table 1 Footnote 12
= Yes   = No Annex 2 Table 1 Footnote 1 Water and Air Quality Bureau, Safe Environments Directorate (SED) Annex 2 Table 1 Return to footnote 1 referrer Annex 2 Table 1 Footnote 2 New Substances Assessment and Control Bureau, SED Annex 2 Table 1 Return to footnote 2 referrer Annex 2 Table 1 Footnote 3 Existing Substances Risk Assessment Bureau, SED Annex 2 Table 1 Return to footnote 3 referrer Annex 2 Table 1 Footnote 4 Environmental and Radiation Health Sciences Directorate Annex 2 Table 1 Return to footnote 4 referrer Annex 2 Table 1 Footnote 5 Consumer Product Safety Directorate Annex 2 Table 1 Return to footnote 5 referrer Annex 2 Table 1 Footnote 6 Therapeutic Products Directorate Annex 2 Table 1 Return to footnote 6 referrer Annex 2 Table 1 Footnote 7 Biologics and Genetic Therapies Directorate Annex 2 Table 1 Return to footnote 7 referrer Annex 2 Table 1 Footnote 8 Marketed Health Products Directorate Annex 2 Table 1 Return to footnote 8 referrer Annex 2 Table 1 Footnote 9 Natural and Non-prescription Health Products Directorate Annex 2 Table 1 Return to footnote 9 referrer Annex 2 Table 1 Footnote 10 Veterinary Drugs Directorate Annex 2 Table 1 Return to footnote 10 referrer Annex 2 Table 1 Footnote 11 Food Directorate Annex 2 Table 1 Return to footnote 11 referrer Annex 2 Table 1 Footnote 12 Health Evaluation Directorate Annex 2 Table 1 Return to footnote 12 referrer Annex 2 Table 1 Footnote * In general, the totality of evidence concept does not involve any actual “weighing” of multiple lines of evidence relative to each other, and is thus not interpreted as part of the WoE concept by certain programs. Nevertheless, this concept is commonly recognized as part of the scope of the WoE approach in the risk assessment context by most programs across the Department. Some differences are also observed regarding the sub-concept of considering “all” available evidence and such apparent differences may be the result of more literal interpretations of “all” available evidence by these programs compared to others, rather than a true reflection of actual differences of risk assessment practices. Moreover, a precise interpretation of “all” is also dependent on the program area/regulatory requirements in terms  of the type of evidence that are required in order to support a submission. For example, for programs conducting risk assessments on therapeutic products such as drugs and biologics, the requirements come from guidelines of the International Conference on Harmonisation of Technical Requirements for Registration of Pharmaceuticals for Human Use (ICH). On the other hand, pesticide evaluations utilize guidance lists of required and conditionally required data, which differ depending on how and where the product is used. Annex 2 Table 1 Return to footnote * referrer Annex 2 Table 1 Footnote **  Qualitative methods of assessing and integrating multiple lines of scientific evidence in Departmental risk assessment programs seem to  be the dominant application of the WoE approach for risk assessments across the Department. Specific qualitative methods can range  from simple listing of the evidence assessed, to more detailed narrative descriptions that explain the rationale behind the application of  best professional judgment, in which some lines of evidence are considered more important, and are given more weight, compared to others. Semi-quantitative methods include systematic assignment of alphanumeric scores for each line of evidence, as well as logic  models, decision trees and causality analysis that implicitly give more weight to certain lines of evidence over others, even if actual numeric scores are not assigned. Frequently referred to as “levels of evidence”, scoring tools are more often applied by programs involved in the regulation of therapeutic products, for which such standards and scoring systems exist, and are practiced, by international counterparts. Semi-quantitative methods employing hierarchal descriptors instead of alphanumeric scores are frequently employed in the context of assigning value to causality criteria used in foodborne illness outbreaks. Similarly, semi-quantitative descriptors for key events could be used for Mode of Action (MoA) assessment of chemicals for carcinogenicity/mutagenicity. Quantitative methods are not widely used   across the Department, often due to limitations in availability of appropriate data. Annex 2 Table 1 Return to footnote ** referrer

Annex 3

Checklist for Transparent Documentation of Weight of Evidence Approach

When weight of evidence terminology is used, specify intended meaning in relation to the following concepts:

• Totality of Evidence: conceptual interpretations of the nature and scope of evidence sources for consideration
• Weighing Evidence: methodology for assessment and integration of multiple lines of evidence

For the risk assessment process, are the following documented?

• evidence gathered: all available to date, individual sources and types
• evidence included for further consideration, and why (i.e., inclusion criteria)
• evidence excluded from further consideration, and why (i.e., exclusion criteria)
• lines of evidence assembled (list individual studies under each line)
• assessment criteria applied to lines of evidence, and scoring tools used (if any)
• values/weighting assigned to each line of evidence (e.g., descriptions, alphanumeric)
• integration scheme (e.g., best professional judgment, mathematical formula, criteria framework)
• overall conclusion/recommendation(s)