An innovative tool to prioritize the assessment of investigational COVID-19 therapeutics

Abstract

Background: As the COVID-19 pandemic unfolded, hundreds of investigational COVID-19 therapeutics emerged. Maintaining situational awareness of this extensive and rapidly evolving therapeutic landscape represented an unprecedented challenge for the Public Health Agency of Canada, as it worked to promote and protect the health of Canadians. A tool to triage and prioritize the assessment of these therapeutics was needed.

Methods: The objective was to develop and conduct an initial validation of a tool to identify investigational COVID-19 therapeutics for further review based on an efficient preliminary assessment, using a systematic and reliable process that would be practical to validate, implement and update. Phase 1 of this pilot project consisted of a literature search to identify existing COVID-19 therapeutic assessment prioritization tools, development of the Rapid Scoring Tool (RST) and initial validation of the tool.

Results: No tools designed to rank investigational COVID-19 therapeutics for the purpose of prioritizing their assessment were identified. However, a few publications provided criteria to consider and therapeutic ranking methods, which helped shape the development of the RST. The RST included eight criteria and several descriptors (“characteristics”). A universal characteristic scoring scale from −10 to 10 was developed. The sum of all the characteristic scores yielded an overall benefit score for each therapeutic. The RST appropriately ranked therapeutics using a systematic, reliable and practical approach.

Conclusion: Phase 1 was successfully completed. The RST presents several distinct aspects compared with other tools, including its scoring scale and method, and capacity to factor in incomplete or pending information. It is anticipated that the framework used for the RST will lend itself to use in other dynamic situations involving many interventions.

Introduction

Background

At the beginning of the pandemic, the rapid global transmission of SARS-CoV-2, the virus that causes COVID-19, prompted extensive research into a range of treatment options. As the pandemic unfolded, hundreds of investigational (i.e., prior to market authorization) pharmaceutical COVID-19 therapeutics emerged ^{Footnote 1}. Maintaining situational awareness of this extensive and rapidly evolving therapeutic landscape represented an unprecedented challenge for the Public Health Agency of Canada (PHAC), as it worked to promote and protect the health of Canadians ^{Footnote 2}. A timely and thorough assessment of all investigational therapeutics was not feasible. Therefore, a practical tool to systematically, reliably and efficiently triage and prioritize the assessment of these therapeutics was needed to help inform their potential applicability for Canada.

To identify existing COVID-19 investigational therapeutic assessment prioritization tools, a literature search was conducted in Ovid MEDLINE^® with the assistance of a PHAC librarian, using the focused search concepts “decision support techniques,” “COVID-19 therapeutic treatment or assessment” and variations of their terms. A total of 302 articles were identified; 46 were deemed relevant and these were reviewed. The search identified no tools designed to rank investigational COVID-19 therapeutics to prioritize their assessments. However, several publications provided criteria to consider when conducting health technology assessments or making therapeutic formulary decisions ^{Footnote 3Footnote 4Footnote 5Footnote 6Footnote 7Footnote 8Footnote 9}. Furthermore, some of these publications and their references featured different therapeutic ranking methods and evaluation frameworks ^{Footnote 4Footnote 7Footnote 9Footnote 10Footnote 11Footnote 12Footnote 13Footnote 14Footnote 15Footnote 16}. Although they had important limitations (e.g., required a pre-defined list of therapeutics with known properties, complex to implement or adapt quickly), certain elements, such as their assessment criteria and use of positive and negative scoring, were found relevant to incorporate into a tool that PHAC developed in the fall of 2022. This article reports on the first phase of this pilot project to develop what has become known as the Rapid Scoring Tool (RST).

Objective

To develop and conduct an initial validation of a tool to identify investigational COVID-19 therapeutics for further assessment, based on an efficient preliminary review, using a systematic and reliable process that would be practical to validate, implement and update.

Intervention

Setting

During the pandemic, a team of four individuals from the PHAC COVID-19 Therapeutics team was formed to develop the RST. The members had backgrounds in critical appraisal, clinical and research pharmacy, therapeutic evaluation, program evaluation, epidemiology, immunology and public health. Investigational COVID-19 therapeutics were identified primarily from a daily scan of key COVID-19 sources of information (e.g., updates and pre-prints of key COVID-19 trials) and ClinicalTrials.gov. The RST was developed using Microsoft Excel^®.

Intervention

The pilot project had two phases:

• Phase 1: Development (stages one and two) and initial validation (stage three) of the RST
• Phase 2: Further validation and enhancement of the RST

Stage one: Design the RST. The RST team developed the RST, which included defining the decision problem it was intended to address ^{Footnote 13Footnote 14Footnote 15}, the broad categories or “criteria” that would be used to assess therapeutics (e.g., safety), and more precise descriptors or “characteristics” within each criterion. The criteria and characteristics were developed based on literature findings, feasibility of implementation and over a dozen internal discussions with stakeholders, both within and outside of the COVID-19 Therapeutics team, involved in the assessment and monitoring of therapeutics (i.e., medical advisors, managers, epidemiologists, policy analysts and research analysts). Next, a “characteristic” scoring scale was constructed based on the decision problem. This universal scale was used to assign a score to each characteristic. For each therapeutic, an overall perceived benefit (“overall benefit”) score was calculated by summing the scores of all the characteristics that applied to that therapeutic.

Stage two: Pilot test the RST. During stage two, therapeutics were entered into the RST and ranked by their “overall benefit” score to identify those to assess more thoroughly. Two members of the RST team independently selected the appropriate characteristics (one for each criterion) from the list of possible characteristics, using key sources of information. All discrepancies were resolved through discussion with a third member until full agreement among the three members was reached. When adjustments to the criteria, characteristics and/or their associated scores were required, an iterative consensus approach within the RST team was used, with input from stakeholders, to validate and maintain internal consistency (i.e., alignment and coherence among the RST components). Face validity of the ranking, internal consistency and reliability of the RST were deemed to have been achieved once 10 consecutive therapeutics had been entered without discrepancies (i.e., the need to involve a third member of the team) or the need to adjust the RST and the ranking was deemed appropriate by the members of the RST team.

Stage three: Conduct an initial validation of the RST. This stage consisted of further validation of the RST using the input from three members of the COVID-19 Therapeutics Team who had not used the RST to assess individual therapeutics. Together, they had critical appraisal skills, medical, nursing and public health backgrounds. They were provided with detailed information on 15 randomly selected therapeutics in the RST (using the RAND function of Microsoft Excel) They were given time to ask questions and deliberate, and asked to indicate their level of agreement or disagreement (using a Likert scale) with the RST’s ordinal ranking of these therapeutics (i.e., which therapeutic ranked first, second, etc.). They were also asked to provide statements describing the intervals between rankings (e.g., therapeutic A is clearly of greater overall benefit compared with therapeutic B; therapeutics C and D offer very similar overall benefit). The rankings were considered validated (‘’appropriate’’) if at least two of the three individuals agreed or strongly agreed (consensus agreement) with the ordinal ranking of therapeutics and on 75% or more of the 12 ranking statements. This consensus agreement approach was adopted to leverage the benefits of collaborative decision-making, while mitigating risks associated with individual biases; the 75% threshold was considered practical and meaningful to describe substantial consensus.

Outcome measures

Table 1 provides the list of outcome measures and stages during which they were assessed.

Outcomes

Design of the Rapid Scoring Tool

The decision problem pertained to the need to efficiently triage and prioritize the large number of investigational COVID-19 therapeutics for further assessment, based on a preliminary assessment of their perceived benefit, within the Canadian context. The criteria included in the RST at the time of writing, and the elements that were used to develop the characteristics for each criterion, are listed in Table 2.

Figure 1 shows the scale developed and used to assign a score to each characteristic, with scores ranging from −10 to +10. In most cases, characteristics had only a moderate effect on the perceived benefit of a therapeutic and, as a result, most scores were in the −5 to +5 range.

Figure 1 : Descriptive text

The figure depicts a row of five cells. From left to right, the cells contain the following text:

• This characteristic would substantially decrease the perceived benefit of a therapeutic within the Canadian context, based on a preliminary assessment
• This characteristic would moderately decrease the perceived benefit of a therapeutic within the Canadian context, based on a preliminary assessment
• This characteristic would have no impact (i.e., “neutral”) on the perceived benefit of a therapeutic within the Canadian context, based on a preliminary assessment
• This characteristic would moderately increase the perceived benefit of a therapeutic within the Canadian context, based on a preliminary assessment
• This characteristic would substantially increase the perceived benefit of a therapeutic within the Canadian context, based on a preliminary assessment

Above each cell is a numerical value indicating the score associated with the description within that cell. The scores, from left to right are −10, −5, 0, +5 and +10. There is a bidirectional arrow above the scores, with increasing thickness as it approaches the limits of −10 and +10.

Table 3 provides an example of a criterion, its associated characteristics and their scores from the RST. For example, if a therapeutic was shown to be associated with serious liver toxicity during a Phase 3 trial leading to a serious warning and precaution, characteristic 4, “Serious warnings and precautions (…)” would be the characteristic selected for the safety criterion for that therapeutic. During the development of the tool, it was decided that this characteristic would decrease the perceived benefit of a therapeutic having this characteristic, within the Canadian context and based on a preliminary assessment, and be assigned a score of −2 (as per Figure 1).

Methods

Implementation of the Rapid Scoring Tool

Figure 2 depicts a simplified version of the workflow used for developing the RST during Phase 1. Some therapeutics could be excluded from further assessment based on a single characteristic. These characteristics of exclusion were assigned a score of −100 to ensure that therapeutics with these characteristics have low “overall benefit” scores and would not be among the top-ranked therapeutics for further assessment. Characteristics of exclusion are shown in Box 1. Given the rapidly changing pandemic environment, all therapeutics were reassessed periodically (whenever new information arose from daily scans of key COVID-19 resources or every six months, whichever occurred first).

Figure 2: Descriptive text

The figure depicts a simplified version of the main steps taken to develop the Rapid Screening Tool (RST). The steps are as follows: Define the question to be answered or decision problem; Determine the criteria; Determine all the characteristics of each criterion; Determine and define the characteristics’ score range, based on the question or decision problem at the beginning of the workflow; Assign a score to each characteristic; A trained data entry person selects the appropriate characteristics for each therapeutic; Review the ranking of therapeutics periodically. The question “Does the ranking have face validity?” is shown in a diamond shape. If the answer is yes, an arrow loops back to the step “Review the ranking of therapeutics periodically.” If the answer is no, the following step is “Review and adjust the criteria, characteristics and characteristics’ scores, as needed, while ensuring internal consistency of the RST” which is shown above an arrow that loops back to a curly bracket grouping the steps from “Determine the criteria” to “Assign a score to each characteristic.” An icon of three people reaching consensus appears above a curly bracket grouping the steps from “Define the question to be answered or decision problem” to “Assign a score to each characteristic.” An icon of three people reaching consensus also appears above the steps “Review the ranking of therapeutics periodically” and ““Review and adjust the criteria, characteristics and characteristics’ scores, as needed, while ensuring internal consistency of the RST.” An icon of a woman using a computer is shown above the step “A trained data entry person selects the appropriate characteristics for each therapeutic.”

Abbreviation: RST, Rapid Scoring Tool

Results

Outcome measures of the Rapid Scoring Tool

After approximately 30 therapeutics were entered in the RST during stage two, the outcome measures, including appropriateness of ranking and the systematic nature, reliability and practicality of the RST, as well as the time required for completing a preliminary assessment, had been met. A standard operating procedure was developed to ensure ongoing consistency using the RST. Appropriateness of ranking was also met during stage three. Consensus agreement was reached for the ordinal ranking of all therapeutics, and for 10 of the 12 (83%) statements describing the intervals between rankings; disagreements pertained to two therapeutics. Adjustments were made to the RST, and the overall ranking of these therapeutics relative to the others was reviewed until consensus was reached. Ten months into Phase 1, 69 investigational COVID-19 therapeutics had undergone a preliminary assessment using the RST.

Discussion

In a dynamic pandemic environment, it was challenging to identify therapeutics (with incomplete information) in a timely manner for further assessment to enhance situational awareness. The RST enabled this through a continuous iterative process to update and validate the criteria, characteristics and characteristic scores, as well as its unique scoring scale. The RST scoring scale standardized all characteristic scores and directly incorporated the concept of “importance” that other tools typically address by assigning weights to criteria ^{Footnote 7Footnote 13Footnote 14Footnote 15}. Appendix was developed to provide further details on these key aspects of the RST, as well as some of their benefits compared with other commonly-used tools, such as the System of Objectified Judgment Analysis based tools ^{Footnote 7Footnote 17Footnote 18Footnote 19} and other Multi-Criteria Decision Analysis-based tools ^{Footnote 11Footnote 16Footnote 20}.

In addition to its primary role, identifying therapeutics for further assessment, the RST served as a structured repository for key information pertaining to the therapeutics, which facilitated timely updating with new information and monitoring. This further enhanced situational awareness of the investigational therapeutic landscape.

Limitations

The RST has limitations that are inherent to the context in which it was developed and operationalized. How they were considered and mitigated is described below. To optimize the efficiency of the preliminary assessment, the RST relied on a subset of assessment criteria used in more thorough reviews. For example, implementation factors were not part of the RST, as this information was often not available or could not be determined rapidly. A different subset of assessment criteria might have affected the ranking of therapeutics. The initial validation during stage three, however, suggested that the subset of criteria and characteristics selected was adequate for identifying therapeutics for further assessment.

The score assigned to each characteristic was agreed upon by a specific group of individuals. A different group might have assigned different scores, which could have affected the ranking of therapeutics. This limitation is inherent to any decision-making process ^{Footnote 18Footnote 21} and was mitigated by involving individuals with different backgrounds and roles in the design and validation of the RST.

The initial validation of the RST was led by the RST team, which might have affected the results. Several steps were taken to mitigate this potential limitation, such as using a structured presentation with questions that were carefully worded for clarity and neutrality, and efforts to avoid motivational biases.

Implications and next steps

The RST enabled timely identification of therapeutics to be assessed more thoroughly, as well as efficient tracking of the therapeutic landscape in an evolving environment. Its iterative approach ensured that it integrated the most up-to-date information on the criteria, characteristics, scores, and therapeutics. By nature of this design, stages two and three of Phase 1 will be repeated periodically.

Phase 2 of this pilot project will consist of assessing the validation and reliability of the RST with additional therapeutics and stakeholders, and formal statistical and sensitivity analyses. It is anticipated that an adapted framework would lend itself to other dynamic situations involving many interventions.

Conclusion

Phase 1 of the pilot project was successful. The RST enabled a systematic, reliable and efficient prioritization of investigational COVID-19 therapeutics for further assessment and enhanced situational awareness of the emerging therapeutic landscape during a dynamic pandemic. The RST presents several distinct aspects compared with other tools, including its scoring scale and method, and capacity to factor in incomplete or pending information.

Authors' statement

• LB — Conceptualization, methodology, validation, writing–original draft, writing–review & editing
• SC — Methodology, validation, writing–review & editing
• MA — Methodology, validation, writing–review & editing
• ES — Methodology, validation, writing–review & editing
• MGR — Validation, writing–review & editing
• CM — Validation, writing–review & editing
• SGS — Validation, writing–review & editing
• JA — Writing–review & editing

Competing interests

None.

Acknowledgements

We thank Taline Ekmekjian from the Public Health Agency of Canada’s Library Services for her assistance in developing the search strategy. We also thank Annecy Houston, Jane Kolbe, Lauren Kreps, Niyi Lawuyi, Dr. Nadine Sicard, Claudia Vela, Dr. Ping Yan and Dr. Shenghai Zhang for providing feedback to earlier versions of the RST and for their insightful suggestions.

Funding

This work was supported by the Public Health Agency of Canada.

Appendix