Elements of Real World Data/Evidence Quality throughout the Prescription Drug Product Life Cycle

Updated: March 5 2019

Contents

• Executive Summary
• Introduction
• Elements of Protocol Development
• Elements of Data Quality
• Conclusions

Executive Summary

This document accompanies Health Canada’s Notice on ‘Optimizing the Use of Real World Evidence to Inform Regulatory Decision-Making.’

The aim of this document is to provide overarching principles to guide the generation of real world evidence (RWE)^{Footnote 1} that would be consistent with the regulatory standard of evidence in place in Canada and internationally. It includes an overview of some of the elements that should be addressed in protocol development and characterizes some of the data quality concerns within submissions containing RWE. This document is not intended to be prescriptive. There are several existing guidelines that can be used to characterize the quality of evidence in real world studies. To date, no specific guideline has been identified for implementation by Health Canada.

The availability of real world data (RWD)^{Footnote 2} has been increasing steadily worldwide. These data provide an alternate source of information that can be leveraged to produce evidence of drug safety, efficacy and effectiveness across the product life cycle.

Health Canada already considers RWE during the pre- and post-market drug regulatory process to inform decision-making, and will continue to develop internal principles for the evaluation of the quality of RWE across the drug life cycle. Health Canada acknowledges that RWE has the potential to provide valuable support in areas where controlled clinical trials are not feasible or are challenging to conduct, and looks forward to continuing to explore the use of RWE to improve Canadians’ access to drugs that are safe and efficacious.

Introduction

Data derived from real world sources and the evidence derived from its analysis and/or synthesis can be used for a variety of purposes, by a variety of stakeholders. Some examples include understanding the real world setting of care and disease, assessing the efficacy or effectiveness of therapy and providing new evidence of relative effectiveness of new medicines, and for funding decisions. Health Canada’s goal is to improve Canada’s ability to better leverage real world evidence (RWE) throughout the drug product life cycle to optimize safety and efficacy, and overall, improve the accessibility, affordability and appropriate use of drugs. The ‘Strengthening the Use of Real World Evidence for Drugs’ project has been developed to achieve this goal. This project is part of Health Canada’s Regulatory Review of Drugs and Devices (R2D2) initiative, which aims to deliver on elements of the Minister of Health’s commitment to improve the affordability, accessibility and appropriate use of therapeutic products as part of the Health Accord, as highlighted in the Minister’s November 2015 Mandate Letter, and reiterated at the January 2016 Health Ministers’ meeting. The project is discussed in further detail in the Health Canada document ‘A Mapping Exercise: Identification of Opportunities for Using Real World Data (RWD) to Inform Evidence-Based Regulatory Decision Making Throughout the Drug Life Cycle’.

RWD sources are increasing in quantity and quality and these sources of data can provide useful information for the enhancement of more robust decision-making with respect to safety and efficacy, as well as utilization and treatment patterns. Also, RWE has the potential to be a powerful tool when making decisions about access. However, it is not without its challenges including the potential for bias (e.g. confounding), and issues that may impact the robustness of the findings. Hence, careful consideration will be required when pursuing RWE development to support regulatory decisions that depend upon context and regulatory need.

To inform market authorization decisions, prospectively planned clinical trials have been and continue to be considered the most robust tool for providing evidence of drug safety and efficacy. While following specific patient populations in a highly specialized environment facilitates the collection of high quality controlled data, it can limit the generalizability in real world settings. Moreover, conducting clinical trials is not always feasible and thus may not always be deemed ethical for certain diseases/disorders (such as rare diseases) or patient populations, where excessive trial costs or small available patient populations may introduce constraints. Expanding data and evidence sources to include RWD/RWE may address some of these concerns, and offer new opportunities to gain insight on public health, advance health care, and increase both the extent and rate of drug access for patient populations.

Factors to consider when evaluating the quality of the evidence include the data quality, the study design, statistical analysis and the manner in which the results are interpreted relative to the scientific question being posed, which should be precise and clear. These factors should be comprehensively evaluated to ascertain if the level of evidence is sufficient. The scientific question of interest is an integral component in the assessment of the level of evidence. It should always be precisely defined with careful consideration of estimands. It serves as a foundation to ascertain if the study is appropriately designed and analyses are completed to support the findings of a study. This document focuses on elements of protocol development and data quality. Good research principles should continue to be implemented to ensure augmented quality of evidence, as outlined in the International Conference of Harmonization (ICH) Guidelines, including the Statistical Principles for Clinical Trials E9.

RWE obtained from high-quality methodology can enhance the ability to make informed regulatory decisions. Health Canada will consider the use of RWE to support regulatory decision making regarding the benefits and risks of prescription drug products.

Elements of Protocol Development

All research protocols should be well-designed and include a comprehensive assessment of good research principles. Below are fifteen key elements that should be considered for each protocol, and are reflective of the “European Network of Centres for Pharmacoepidemiology and Pharmacovigilance (ENCePP) Protocol Checklist” and “The Guidelines for Good Pharmacoepidemiology Practices (GPP)”. Both prospective and retrospective designs should attempt to address each element, or provide justification why it may not be applicable to the specific study. This list will be updated as the process evolves.

1. Research Question: A clearly defined research question is integral to all studies. The protocol should include well-developed research question(s) and hypothesis (es) with special attention to construction of appropriate estimands. In addition, the protocol can state the research objectives, aims and rationale. The protocol may also include a critical review of the literature, including a summary of applicable information from other studies and if these studies address the knowledge gaps.
2. Milestones: The protocol should provide the timelines for key milestones in the data collection process, including start/end dates of data collection or extraction of data from existing data sources, progress and interim reports, registration of the study and provision of final study reports. As indicated by GPP, the International Society of Pharmacoepidemiology (ISPE) supports the registration of research protocols (e.g. EnCePP registry, ClinicalTrials.gov) if applicable. Further information can be found within the GPP guidelines, but all registrations or archiving of protocols should be noted in this section.
   Additional information to be included in the preliminary information of the protocol should include: i. A descriptive study title including the study design, population and a version identifier if appropriate; ii. An investigator list including the names, titles, degrees, addresses, and affiliations of all responsible investigators and collaborating institutions; and iii. Name and address of each study sponsor.
3. Research Design: The study design and justification should be clearly articulated, including the type of data collected and specifications of measures of occurrence, association and reporting of adverse events.
4. Source and Study Populations: The study population should be well-defined and the protocol should provide an appropriate level of information in terms of the sampling framework, demographic characteristics, clinical exposures and duration of follow up. Information on inclusion and exclusion criteria and a description of the generalizability of the study may also be included.
5. Exposure Definition and Measurement: The exposure of the subject in the study to the therapy of interest (i.e. dose, dosage form and dosage regimen) should be well-defined and measured, and the protocol should address the validity of any measures. Exposure intensity, time frame and, where appropriate, the mechanism of action should be articulated. The length of the observation of the subjects in the study should be well-defined. Any comparators and/or co-medications should be identified and described.
6. Outcome Definition and Measurement: All primary and secondary outcomes, including dose delivery and intensity, should be clearly defined and measured, and the protocol should address the validity of any measures. In addition the protocol should describe how the outcomes are relevant to the claim.
7. Bias: The protocol should describe any potential sources of confounding, or other sources of bias (e.g. selection bias, information bias). Approaches taken to demonstrate the robustness of the findings to any sources of bias should be described.
8. Effect Measure Modification: The protocol should address the collection of items that could modify the effect and how they were included in the analysis.
9. Data Sources and Collection: The protocol should clearly describe the data sources utilized and the appropriateness of these data to capture all relevant exposures, outcomes and covariates of interest. In addition, the protocol should clearly state any coding systems used for classification of the exposure and outcomes (e.g. Anatomical Therapeutic Chemical (ATC) classification, International Classification of Diseases (ICD) and any methods used for data linkage.
   Data collection methods should be clearly described including pretesting procedures and formal training of personnel in primary data collection. For administrative data, information should also include procedures for record linkage and validation methods to assess representativeness of the data source to the population of interest.
10. Statistical Analysis Plan: A description and justification for the chosen approach for statistical analyses should be well-described, including methods of estimation, a rationale for the study size and/or statistical precision, descriptive analyses, stratified analyses, defined point estimates and confidence intervals, types of comparators, plans to control for confounding, outcome misclassification, sensitivity analyses, type I error control, and plans for handling missing data.
11. Data Management and Quality Control: A description of data storage, management and statistical software should be included in the protocol if available. The protocol should also note any systems implemented for independent review of study results. A description of quality assurance and quality control procedures for all phases of the study should be provided. This may include a description of mechanism(s) to ensure data quality and integrity or reference(s) to mechanisms already published. This section may also include certifications and qualification of supporting research groups/laboratories.
12. Feasibility and Limitations: Study feasibility should be discussed. The protocol should discuss any limitations on the ability to draw conclusions from the data, including the impact of bias (e.g. selection, information), generalizability, and any residual or unmeasured confounding, as well as methods to reduce the impact. Methods to reduce these potential limitations should be prospectively discussed and quantitatively demonstrated where possible.
13. Ethical and Data Protection Issues: Efforts to protect study participants should be included in this section, including confidentiality measures, safeguards of personal information, involvement and outcome of Institution Research Ethics Boards including a Data Safety Monitoring Board, as well as exemption status and other elements of data protection.
14. Amendments and Deviations: Any amendments or deviations to the protocol should be dated, well-described, and justified. This information includes significant deviations and rationale ofthe protocol.
15. Plan for Communication of Study Results: The plans for both communicating study results and disseminating results externally should be discussed. Regulatory requirements promote transparency of research and should be followed. As recommended by GPP, authorship should follow established guidelines (e.g. International Committee of Medical Journal Editors).

As previously noted, the elements for protocol development are not intended to be prescriptive, but should be considered in the evaluation of RWE.

Elements of Data Quality

There are two principal data collection techniques for building RWE: prospective and retrospective. Each approach carries unique challenges with respect to generating high-quality evidence. In evaluation of RWE, the data collection process should be taken into account to ensure high level of data quality. Elements of protocol development as previously outlined should be addressed regardless of method of data collection. The principles of data collection that are used in the generation of high-quality RWE are similar to those used in controlled clinical trials. Verification of the robustness of the findings used to support the RWE can be a challenge; hence, special care should be taken to address potential sources of uncertainty in studies submitted for review.

Prospective Data Collection

Prospectively collected data requires similar rigour as would be anticipated for a controlled clinical trial. Transparency and clearly defined procedures are important to augment the quality of the data. Prospectively collected data should be made available when possible.

Retrospective Data Collection

Retrospectively collected data has additional elements that should be addressed. The same elements as outlined above apply to pre-existing data. While Health Canada acknowledges that additional challenges will exist with retrospective data, the same standard will apply to these data whenever possible. A well-defined research question should be pre-specified to clearly state the intent of the study. Sufficient data elements should be well-defined to identify and assess exposures, outcomes, endpoints and covariates of interest, and validation of these study aspects should be included. Investigators should address the limitations of data resulting from a retrospective design and attempt to mitigate risks as appropriate. All data should be traceable to the source, and source data should be available whenever possible. Curated data will be preferred in all instances. Due to the retrospective nature of these designs, the statistical assumptions may require testing to ensure their validity (e.g. comparability of study arms). All procedures should be clearly articulated and transparent. Validation of data sets should be clearly described and include information addressing data completeness (e.g. methodology to address missing data). Approaches should be taken to validate all relevant elements of existing administrative datasets and should be described and provided whenever possible.

Conclusions

The use of RWE in regulatory decision making continues to be of importance to Health Canada. There are several aspects of data quality that should be considered to ensure that research studies remain high quality and regulatory grade. High quality data collection should be present, agnostic of the study design. Elements of protocol and data development have been outlined to reflect good research practices. The expectation is that submissions will clearly report the various elements outlined above, and if not present, justification and appropriate rationale for their omission will be provided. This document has focused on good practices for protocol development and data collection. Health Canada will continue to develop principles to evaluate the quality of RWE across the drug product life cycle to improve access to safe and efficacious drugs for Canadians.