Evaluating transmission risk of COVID-19 at gatherings

Abstract

Background: The coronavirus diseases 2019 (COVID-19) pandemic has presented an unprecedented public health challenge. Prior to vaccination, non-pharmaceutical interventions, including closures, were necessary to help control the epidemic. With the arrival of variants of concern and insufficient population vaccination coverage, ongoing evaluation of transmission risk in settings and the use of non-pharmaceutical interventions are necessary to help control the epidemic. This study aimed to produce a framework for evaluating transmission risk in settings where individuals gather and inform decision-making.

Methods: A multi-criteria decision analysis process was used to structure the framework. Fifteen criteria were identified as important to consider for COVID-19 transmission risk based on the literature. This list was ranked by experts and then categorized. The analysis was structured by the consensus list of criteria and relative positioning of each criteria within the list to produce sets of factors to consider when assessing transmission risk at gatherings.

Results: Fifteen experts from across Canada participated in ranking the criteria. Strong consensus was found on the relative importance of criteria and this relative consensus was used to create four categories: critical (3 criteria); important (6 criteria); good to consider (5 criteria); and if time permits (1 criterion).

Conclusion: The resulting consensus list and categories constitutes a set of important elements that can be applied to any setting as an objective and transparent framework to assess transmission risk in the venue. In conjunction with further consideration of the local epidemiology of COVID-19, an overall risk of transmission assessment can be established and uniformly implemented.

Introduction

The emergence of the novel coronavirus severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) and the associated disease (coronavirus disease 2019; COVID-19) was initially observed as an outbreak in Wuhan, China in late 2019, and resulted in the ongoing pandemic^{Footnote 1}. The virus was first detected in Canada in early 2020 and has caused 1,670,241 cases and 28,367 deaths (as of October 13, 2021)^{Footnote 2}. The SARS-CoV-2 is a highly transmissible respiratory virus that can cause a range of symptoms from none to mild or severe disease and death^{Footnote 3}. This has created an unprecedented disease management challenge for public health and numerous public health measures have been implemented with variable stringency in an attempt to slow the epidemic and reduce its impact. These include increased personal physical distancing and non-pharmaceutical interventions, such as case detection and isolation, tracing of contacts and quarantine and community masking^{Footnote 4}, to reduce transmission opportunities in the community. However, when and where transmission is high, a range of restrictive closures have been imposed by provincial and local governments, including closures of schools, universities and non-essential businesses, bans or limitations on gatherings, limitations on travel within and between jurisdictions and encouragement of teleworking in an attempt to limit transmission opportunities. Together these actions helped to minimize person-to-person contacts in Canada and resulted in the epidemic coming under control with low reported case incidence during summer 2020. However, the fall 2020 return to school and re-opening of businesses in many regions of Canada resulted in a resurgence of the epidemic and a second wave peaking higher in total cases, hospitalizations and deaths than the spring wave^{Footnote 2}. As capacity for control of the epidemic by testing and tracing alone was surpassed, re-implementation of some levels of restrictive closures was deemed necessary to help reduce rates of contact between Canadians, regain previous levels of epidemic control and limit the risk of exceeding healthcare capacity. Closures of schools and businesses have important social and economic impacts on society. The challenge facing decision-makers is how to navigate the trade-off between preventing COVID-19 transmission and the negative potential health, social and economic impacts of restrictive measures^{Footnote 5Footnote 6Footnote 7Footnote 8Footnote 9Footnote 10Footnote 11Footnote 12}. A full assessment that includes negative impacts of closures is outside the scope of this work, as at this time the focus is restricted to informing assessment of transmission risk.

The arrival of vaccine doses in Canada in December 2020 raised hopes that restrictive measures could be eased. However, the recent emergence of new, more transmissible and in some cases more virulent variants of concern (VOC) meant that caution was and will be needed when lifting restrictive measures and re-opening businesses and places where people gather—particularly until sufficient vaccination coverage and natural immunity of the Canadian population has been achieved. Even then, the capacity to inform decisions on restrictive closures will remain relevant with the continuing threat of immune escape VOC and the potential of waning immunity.

This project began in late 2020, prior to vaccine arrival in Canada, with the aim to explore available evidence on COVID-19 transmission in different settings and contribute to informing decision-making around closures. Settings are meant to broadly encompass all locations presenting a transmission risk for COVID-19 that a decision-maker may wish to assess. These include transmission at private gatherings in people’s homes, as well as transmission in public places such as schools, grocery stores, retail stores, concerts and bars among others. The explicit consideration of high-risk populations can also be included in the assessment. While the potential cascading effects of closures are numerous and still being studied, characteristics of a setting contributing to transmission risk remains independently important to evaluate, even as vaccination is being rolled out since the presence of variants of concern continues to pose an important transmission risk. The objective of this project was to produce a framework to assist in ranking settings by the risk they pose for COVID-19 transmission, and potentially identify areas where mitigating measures can be targeted to help reduce transmission risk in these same settings in order to help inform decision-making.

Methods

Multi-criteria decision analysis (MCDA) is part of a family of decision aid tools from the field of operations research and used in numerous sectors to systematically evaluate alternatives over multiple criteria^{Footnote 13}. Multi-criteria decision analysis approaches help to structure reflection around a decision problem by allowing the integration of multiple types of evaluations and the highlighting of strengths and weaknesses of the alternatives under evaluation. Participatory MCDA processes generally consist of a multi-step problem structuring phase where the problem is defined, stakeholders identified, criteria defined and weighted, items to be ranked identified and evaluated over the criteria, followed by a decision analysis phase where the multi-criteria analysis is carried out along with a sensitivity analysis and interpretation of results (Table 1). In this article, a “rapid and light” version of a participatory MCDA process was used to structure a framework for the evaluation of settings at risk for COVID-19 transmission, while taking into account time constraints of stakeholders and considerable data gaps in the literature. The objective was to identify criteria and indicators that would be most informative for assessing transmission risk in settings and produce a consensus ranking of these criteria by experts. The results of this exercise are presented along with a discussion of how the results could be used to help assess settings for transmission risk.

Four steps from a participatory MCDA methodology were adapted to allow for the construction of an expert-ranked consensus list of criteria that could be used as a decision-aid. A quick scan of the literature was conducted to search for broad factors contributing to COVID-19 transmission risk and produce a preliminary list of key criteria. The scan was conducted in an evergreen database of COVID-19 literature maintained within the Public Health Agency of Canada that compiles citations daily from seven databases. The search was used to draft an initial list of twenty-three criteria, and indicators for their rapid evaluation. A limited number of studies were available and consisted primarily of rapid reviews of reports where clusters had occurred with many of the early outbreaks reported having occurred before widespread use of public health measures. Preliminary criteria related to common factors present in settings where outbreaks had occurred. Droplet and aerosol transmission was thought to account for most transmissions and pointed towards elements favoring close contact in closed and crowded spaces as primary drivers of transmission.

The preliminary list of criteria was presented to a group of 62 provincial public health experts for review and comment. The list was condensed to 15 criteria, including 10 site and event characteristics, one participant-level characteristic and four potential mitigation measures (Table 2). In order to keep the final number of criteria manageable, a number of criteria from the original list were combined (e.g. indoor/outdoor location and ventilation) and criteria to evaluate secondary activities at settings were not included (e.g. shared dining or break rooms). This list was then presented to a group of experts within the Pan-Canadian Public Health Network involved in the COVID-19 response, for ranking during December 2020 via an online tool created explicitly for this purpose. The individual expert-ranked lists were combined using general Mallows models^{Footnote 14} to produce a consensus ranking (see Appendix for more details on the general Mallows models). The R package PerMallows^{Footnote 15} was used to analyze the rankings.

Results

Fifteen characteristics (i.e. criteria) were presented to a set of experts for ranking from highest to lowest level of importance when evaluating transmission risk of COVID-19 in various settings. Fifteen experts returned rankings by the deadline, and while the number of respondents was low, the respondents represented the geographic regions in Canada most affected by COVID-19 at the time.

Consensus ranking by means of generalized Mallows models

A generally good consensus emerged among experts on the relative importance of the criteria with some individual variations in specific ranking positions (Figure 1). Table 3 shows the rankings created by each participant. The consensus ranking resulting from the generalized Mallows models with the Kendall and Hamming distance is shown in Table 4. While there was broad agreement between the two consensus rankings, differences emerged as a result of wider variation in respondent rankings for some criteria.

Location and ventilation and ease of contact tracing were the criteria on which experts most strongly agreed in terms of absolute rank ordering (i.e. most important and least important criteria). Location and ventilation appears exclusively in the first seven ranks and almost always (n=14/15 times) in the first four ranks. Criteria with stronger disagreement among experts regarding absolute rank position were number of households or individuals and level of expelled air.

Criteria categorization based on combined consensus rankings

Criteria were placed into categories based on their level of agreement between experts and combined Kendall and Hamming rank orderings (Table 4). Four categories were created as a result of these ranking agreements among experts: “critical”; “important”; “good to consider”; and “if time permits”.

Categories 1 and 2: Critical and important criteria

The first set of criteria (“critical”) consisted of three criteria that were consistently ranked within the first few positions by experts: 1) density of the crowd; 2) contact between participants; and 3) location and ventilation.

The second set of criteria (“important”) were almost consistently ranked within the top half of the ranking by experts (with the exception of level of expelled air), though some variability in specific rank positioning was observed among experts: 1) number of households; 2) level of expelled air (of activity); 3) duration of the event or activity; 4) use of personal protective equipment; 5) mixing of networks; and 6) mixing of participants. While level of expelled air showed wide variation in expert ranking (present in the top half for some experts and the bottom half for others); given available literature data on this criterion, it was placed in the “important” category.

Categories 3 and 4: Good to consider and if time permits criteria

The third set (“good to consider”) consisted of five criteria. Experts generally ranked these criteria in the lower half of their rankings, though relative importance attributed to each criteria varied between experts. This set included the following: 1) the use of engineering controls and environmental cleaning; 2) related activity; 3) administrative scheduling; 4) age structure of participants; and 5) shared equipment or surfaces. Finally, given very strong agreement for the rank positioning of the last criterion, ease of contact tracing, which was almost consistently ranked last among experts, this criterion was placed in the “if time permits” category.

Discussion

This project aimed to identify key factors (criteria) to consider when making decisions around COVID-19 transmission risk in various settings where people gather. The use of generalized Mallows models allowed for the analysis and quantification of the consensus among experts on the rank importance of different transmission risk factors (criteria). A lack of approximate consensus on a given criterion can lead to large differences between models with different metrics; however, using the Kendall and Hamming metrics, broad consensus was found among the most important and least important criteria.

The consensus-ranked list of transmission criteria and corresponding categories resulting from this exercise contribute to a framework for ranking settings for COVID-19 transmission risk based on criteria identified from both the literature and expert opinion. How a setting is evaluated or scored with respect to a specific criterion will depend on a range of factors specific to the local community where the evaluation is undertaken, including local transmission, public health measures in place, current adoption of those measures by the local population and setting-specific characteristics.

Although this framework is intended to assist in evaluating transmission risk, all risk assessments should be performed in the context of the local epidemiology and with consideration of the specific characteristics of the gathering/event/venue being evaluated. A ranking of transmission risk of settings produced in one geographical location will not necessarily be the same as that performed in another geographical location due to local epidemiological variation even if the same criteria are used.
Based on the formative research conducted, the consensus list captures elements that are most directly related to transmission risk. When evaluating settings and their risk for transmission, it is important to keep in mind the activities performed on site and their related contexts since related activities may affect transmission risk; e.g. shared transport to the setting or shared accommodations. These related activities may present additional opportunities for transmission that may be important to consider for inclusion in the risk assessment.

Many of the criteria presented are inter-related or synergistic and, as a result, may be difficult to evaluate individually (e.g. the number of participants at an event and the density of the crowd). The use of scenarios may help tease apart some of these factors. For example, a scenario could be defined to evaluate transmission risk when a certain percentage of the population is vaccinated, and a separate scenario defined to consider a different target vaccination percentage. Alternate scenarios could be defined to consider different levels of community transmission as local prevalence of COVID-19 will change the likelihood of encountering an infected individual. Expert review and discussion of the evaluations is also important as it will promote cross-examination and consideration of a broad set of local factors.
Operationalization of this list and resulting categories is left to the discretion of regional public health experts, though some suggestions are discussed. Assessment as part of a multi-step MCDA process would enable a systematic evaluation between settings; however, a full illustration of this approach is beyond the scope of this article.

Variant of concern considerations

Transmission risk evaluation will continue to be necessary until sufficient vaccination coverage can be reached to achieve herd immunity. With the emergence of VOCs across the country, additional waves of cases may continue to threaten healthcare capacity in Canada despite vaccine rollout. As such, our experts were consulted once again in March 2021 to see if their rankings of the criteria would change given the emergence of VOCs. Given a lack of evidence that VOCs affect transmission risk differently, the experts that responded (n=10/15) left their rankings unchanged. This should be monitored as further knowledge is gathered on the subject.

Using the criteria for evaluation of settings and gatherings and limitations

Although a full multi-criteria evaluation of settings is outside of the scope of the current paper, some guidelines on the use of the criteria for evaluating settings are suggested below. Software packages exist for MCDA analysis, including a recently developed free package for R (https://cran.r-project.org/web/packages/MCDA/MCDA.pdf) and other academic or paid software options. These software packages allow MCDA analysis without the need for statistical experts to carry out the evaluations.

Before undertaking an evaluation of settings, the scope and scale of the assessment should be clearly defined. For example, it is important to define whether the evaluation is being performed to assess the daily exposure of individuals to transmission at any given setting versus assessment of the daily exposure of individuals working all day at a setting since a setting may pose different risks to a casual visitor versus an employee who is exposed over several hours. It is important to consider whether specific subgroups are to be considered in the assessment; for example, are clinically-vulnerable individuals included in the scope or is a separate assessment required to properly assess this subgroup? As settings are considered for inclusion, creating a description of the setting in the context being assessed is useful (e.g. in a grocery store—where a typical visit generally lasts approximately 15 minutes—masks are currently mandatory). Additional variations of settings can be added to assess variations that may be relevant to consider (e.g. variations where mask use is not mandatory, etc.). A quick review of the criteria should be undertaken by the experts participating in the process to assess whether all proposed criteria remain relevant to the local context (e.g. a criterion for which all settings have the same score is not discriminating and may be omitted from the evaluation).

Depending on the data available to a decision-maker for assessing settings, and taking into account levels of uncertainty, variability and missing data, the essential and important criteria should be evaluated where possible as a first level of assessment of COVID-19 transmission risk between settings. Expert judgement and opinion can be used to fill in missing data. If sufficient information is available or can be appropriately assessed by experts, a more complete MCDA-style assessment of settings can be undertaken. A systematic evaluation process, such as offered by an MCDA evaluation, can be used to better understand the relative transmission risk between settings and, in particular, to highlight the strongest contributing factors as well as strongest protective factors for transmission risk between settings. This type of evaluation could help inform where mitigation measures should be considered to help reduce transmission risk. A setting that has criteria that score as poor or insufficient should be considered for mitigation and potential monitoring of transmission risk. As previously suggested, the use of scenarios can also be used to consider the changing epidemiology and its impact on transmission. For example, scenarios with different levels of vaccination, new levels of dominance of a VOC and levels of community transmission can be defined and used to evaluate how they may affect relative transmission risk of settings.

Depending on the data available and levels of uncertainty around these data, any resulting ranking will not represent a strict absolute assessment or ranking of settings, but rather a working local evaluation that reflects the information available and the relevant experts participating in the process.

As a reminder, the use of this framework is meant to help inform decision-making around transmission risk rather than make decisions, since other factors should be considered in a decision-making process around closures. To conduct a more complete assessment of closures/re-openings, additional dimensions beyond transmission risk factors alone such as social considerations, economic and other health factors could be considered for inclusion in a multi-step participatory MCDA process.

Conclusion

This project drew upon the latest evidence concerning transmission risk factors for COVID-19 in venues from which criteria for the evaluation of transmission risk was developed and then evaluated by experts. The resulting consensus list constitutes a set of important generic elements that can be applied to any setting as an objective and transparent framework to assess transmission risk in the venue. With further consideration of the local epidemiology of COVID-19, an overall risk of transmission assessment can be established. This work focused on the factors most directly related to COVID-19 transmission as a first level of concern in evaluating settings. Depending on the decision-making context (e.g. decisions around closures or re-openings) additional factors should be considered for inclusion in the decision-making process, including economic and social impacts. Additional layers of information could be added to the participatory MCDA process to include economic, social and health criteria so that trade-offs could be more fully examined, allowing for more informed decisions by decision-makers around closures and re-openings to reduce the transmission risk of COVID-19.

Authors’ statement

• VH — Conception and design, data acquisition and interpretation, writing–original draft, review and editing
• DM & JL — Data acquisition, data analysis, writing–review and editing
• AF, AO, PT and NHO — Conception and design, revising and editing of writing and critical review
• LW — Interpretation, revising and editing of writing and critical review

Competing interests

None.

Acknowledgements

The authors would like to thank members of the technical advisory group and the Canadian Pandemic Influenza Preparedness (CPIP) task force for their support and feedback in this project; as well as the Knowledge Synthesis team members within Public Health Risk Sciences Division of Public Health Agency of Canada for their invaluable knowledge synthesis products. The authors would also like to thank members of the Communicable and Infectious Disease Steering Committee and modelling secretariat for logistical support with this project.

Funding

None.

Appendix

A1: Rank orderings and generalized Mallows models

Imagine that we have a set of N rankings over n choices. In our case, N represents the number of experts and n denotes the criteria (items). The problem is to find the consensus ranking among the experts, which best agrees with the N rankings offered by the experts.

Consensus ranking finds application in social welfare analysis. In 1950, Kenneth Arrow showed^{Footnote 55} that if a decision-making body consists of at least two members with at least three options to decide among, then it is impossible to design a social welfare function that simultaneously satisfies all the reasonable requirements of a fair system:

• If every voter prefers alternative X over alternative Y, then the group prefers X over Y
• If every voter's preference between X and Y remains unchanged, then the group's preference between X and Y will also remain unchanged (even if voters' preferences between other pairs like X and Z, Y and Z, or Z and W change)
• There is no dictator: no single voter possesses the power to always determine the group's preference

Arrow's impossibility theorem has several technical conditions in its formal statement (see section A3) which defines the "fair" system. While the theorem tells us that no deterministic preferential voting system exists which satisfies the technical fairness requirements, in practice all systems do not work poorly at all times. The impossibility theorem finds application in the study of voting systems and important results can be found in^{Footnote 14Footnote 56}.

Rankings consist of bijections of the set of integers {1,2,3,…, n} onto themselves. We will denote rankings with the symbols π and σ. For example, the ranking π = {2,4,1,3} means that item 1 is ranked second, which we denote as π(1) = 2; item two is ranked fourth, π(2) = 4. Every ranking has an inverse π^-1 which gives the items in terms of the ranks: π x π-1 = e = {1,2,3,…, n}. Given a set of rankings, we would like to find the center or the consensus of the set over some distance measure between rankings. There are many distance metric for rankings, but in this article we will focus on two of the most popular: the Kendall distance and the Hamming distance. For any distance measure d(⋅,⋅) we have d(σ, π) = d(σ π^-1, e). When the reference ranking is the identity ranking e we use the notion d(σ,e) = d(σ).

The Kendall distance between two ranking π and σ is defined by,

Equation 1:

d_k(π,σ) = Σ_{l< πj} 1 [ j < _σ l ]

The notation l <_π j means that item l precedes j in ranking π. The Kendall distance counts the number of pairwise discrepancies between rankings. Withn items, the largest Kendall distance between any two rankings is n(n-1)/2. On the other hand, the Hamming distance d_h(π,σ) counts the number of positions that disagree between two rankings,

Equation 2:

d_k(π,σ) = Σⁿ_j=1(1 [ π (j) ≠ σ (j)]

Thus, the Hamming distance takes values between 2 and n inclusively. The Kendall and Hamming distance measures have the important property that they can be decomposed as a sum over n - 1 and n terms respectively,

Equation 3:

d_k(π,σ) = Σⁿ_j=1 V_j (πσ^-1),

d_k(π,σ) = Σⁿ_j=1 H_j (πσ^-1)

where,

Equation 4:

V_j(σ) = Σ_l>j 1[ l <σ j ],)

Equation 5:

H_j(σ)= { 0, iff σ(j) = j; 1, otherwise }

Given a metric for computing distances between rankings, we can build a probability measure over the space. Mallows model^{Footnote 14} is an exponential location probability model over the rankings defined by a central ranking, σ₀ and a dispersion parameter, θ namely,

Equation 6:

P(π) = ( e^{-θ dk,h(π,σ₀)} ) / ( ψ(θ) )

where ψ(θ) is a normalization constant. In a sense, the Mallows model is the extension of the Gaussian distribution to rankings. When θ > 0 the ranking σ₀ is the mode of the distribution—the consensus—and as θ increases the distribution becomes more sharply peaked around σ₀. If θ < 0, σ₀ becomes the anti-mode.

We see that in the Mallows model, all rankings with the same distance from σ₀ are degenerate in probability. With distance measures that decomposed as a sum like those in equation 3, we can break the degeneracy by attaching θ_j to each component of the sum^{Footnote 57}. For the Kendall and Hamming distance, the Mallows model generalizes using discrepancy measures

Equation 7:

d_k(π,π₀; θ) = Σ^n-1_j=1 θ_jV_j (ππ₀^-1)

d_h(π,π₀; θ) = Σⁿ_j=1 θ_jH_j (ππ₀^-1)

such that

Equation 8:

P(π) = (e ^{dk,h(π,σ₀;θ)} ) / ( ψ(θ))

where θ = ( θ₁, θ₂, …). The central ranking σ₀ and θ can be estimated by maximum likelihood or other approximate techniques.

The value of V_j (σ) in equation 3 gives the number of items in j + 1: n which are ranked before j in σ. Therefore, the parameters θ_j reflect the strength of a ranking around the consensus σ₀ (j) = i in that the larger θ_j the larger the probability that π (j) ≤ i. That is, large θ_j in the generalized Mallows model with the Kendall distance implies that item j is ranking in the first i positions with high probability across all the rankings. Similarly, H_j (σ) of the Hamming distance counts the rank discrepancies. Thus, the parameter θ_j corresponds to the strength of consensus at rank j; large θ_j implies high agreement on the item at the j-th rank.

A2: Hamming parameters

To better see the strength of consensus in the rankings across items, the Hamming θ_j parameters against the cross-entropy of a criterion's rank is shown in Figure A1. The cross-entropy measures the amount of impurity in the ranks,

e_j = -Σ¹⁵_i=1 p_ijlog (p_ij) ,

where j is the item label, i denotes the rank, and p_ij gives the probability of the i-th rank for item j. Criteria with large rank dispersion have high cross-entropy. The Hamming θ_j parameters also measure the strength of consensus at a given rank. Figure A1 shows criteria separating into three basic clusters with increasing consensus appearing towards the southeast corner of the plot. Under the Hamming model, the respondents have particularity strong agreement on the criteria at rank 1, 2 and 15.

A3: Formal conditions of Arrow's Impossibility Theorem

Suppose that we are asked to extract a preference order on a given set of options for society. Each individual provides a preference order on the set of outcomes. We desire a ranked voting electoral system, the preference aggregation rule or social welfare function, which transforms the set of preferences into a single global societal preference order. Arrow's theorem says that if there are at least two members in the society and at least three options to decide among, then it is impossible to design a preference aggregation rule that satisfies all of the conditions below at once (conditions assumed to define a "fair system"):

• Non-dictatorship: The social welfare function should account for the wishes of all voters
• Unrestricted domain, or universality: Each set of individual voter preferences should produce a unique and complete ranking of societal choices from the social welfare function. Thus:
  • It must result in a complete ranking of preferences for society
  • It must be deterministic; each time the preferences are presented in the same way, the welfare function generates the same societal preference order
• Independence of irrelevant alternatives (IIA): The social preference between two choices should depend only on the individual preferences between changes in rankings of irrelevant alternatives should have no impact on the societal ranking
• Monotonicity, or positive association of social and individual values: If any individual changes a preference order by promoting a choice, then the societal preference order should either promote that same choice in the new ranking or leave it at the same position. An individual should not be able to penalize a choice by increasing its preference
• Non-imposition, or citizen sovereignty: Every possible societal preference order should be achievable by some set of individual preferences