Non-pharmaceutical intervention article’s conclusions and response
Download this article as a PDFPublished by: The Public Health Agency of Canada
Issue: Volume 48-10, October 2022: Equity, Diversity and Inclusion in Public Health
Date published: October 2022
ISSN: 1481-8531
Submit a manuscript
About CCDR
Browse
Volume 48-10, October 2022: Equity, Diversity and Inclusion in Public Health
Letter to the Editor
Circular logic and flawed modelling compromises non-pharmaceutical intervention article’s conclusions
Jennifer Grant1, Martha Fulford1, Richard Schabas1
Affiliation
1 Division of Infectious Diseases, University of British Columbia, Vancouver, BC
Correspondence
Suggested citation
Grant JM, Fulford M, Schabas R. Circular logic and flawed modelling compromises non-pharmaceutical intervention article’s conclusions. Can Commun Dis Rep 2022;48(10):492–5.
Keywords: modelling, non-pharmaceutical interventions, COVID-19
Abstract
Assessing the value of non-pharmaceutical interventions (NPIs) in response to coronavirus disease 2019 is a critical exercise to ensure optimal response to future pandemics. To be credible, evaluations should be impartial and rely on robust data and methodologies. Unfortunately, the assessment by Ogden et al. fails on all these accounts and instead further confounds the issue by reliance on models with incorrect underlying assumptions, circular reasoning and inappropriate assignment of causality. Ironically, instead of supporting the argument for NPIs, the authors detract from their argument by making unconvincing points supported by poor analysis.
Introduction
Canada’s early response to coronavirus disease 2019 (COVID-19) was largely based on non-pharmaceutical interventions (NPIs)—school and business closures, stay at home orders, curfews, travel restrictions, mandatory public masking and quarantine—that were initially based on little or no evidence Footnote 1 Footnote 2. These measures were not part of existing pandemic plans and, furthermore, they ignored the overarching principles of pandemic planning to “minimize serious illness and overall deaths” and “minimize societal disruption” Footnote 3. Subsequent publications evaluating NPIs have been observational and ecological with almost no high-quality science. Those randomized, cluster randomized trials and robust case-control studies that have been done show weak effects of most NPIs Footnote 4 Footnote 5 Footnote 6 while robust and growing literature demonstrate the counter-balancing adverse effects of NPIs Footnote 7 Footnote 8 Footnote 9. It is vitally important that we try to assess the effectiveness and the costs of each of these interventions dispassionately, based on real-world data. Unfortunately, the article “Counterfactuals of effects of vaccination and public health measures on COVID-19 cases in Canada: What could have happened?” by Ogden et al. Footnote 10 is superficial, deeply flawed and provides a disservice to the evaluation of these important issues.
1. Confusing case fatality rate with infection fatality rate and reported cases with total infections
The first paragraph claims that the infection fatality rate (IFR) early in the pandemic was 1%. An IFR of 1% is a massive overestimate—infection fatality rates were around 0.2% Footnote 11 Footnote 12 Footnote 13 prior to vaccination, and the less virulent Omicron variant has an estimated IFR of 0.006% Footnote 14. Instead, the number being quoted is closer to the case fatality rate. The error results from reporting 3.3 million cases (8% of the population), when in fact, this number is likely closer to 25 million (60% of the population) Footnote 15 Footnote 16. This means that the authors were either unaware of the distinction between case and infection rates or were intentionally reporting them incorrectly. Either option is concerning and should have been corrected prior to publication.
2. Uncritical reliance on flawed and discredited mathematical models
In this article, Ogden et al. use a model Footnote 17 that presumes efficacy of NPIs to prove that NPIs have efficacy. This circular reasoning alone should have disqualified this article at the stage of peer review. If that were not enough, the authors project “almost a million deaths” in Canada, based on their model. Not only would this be a rate fourteen times higher than that actually experienced in Sweden Footnote 18, it would also have required an IFR of at least 3%—at least an order of magnitude higher than evidence-based estimates pre-vaccine Footnote 13.
3. Attributing causality to temporal correlation where it fits its narrative but ignoring temporal correlations that do not
A brief look at the main graphic of the article (Figure 1) shows arrows that deviate from the vertical, with explanatory arrows off-set horizontally with little explanation as to why the specific distance or angle was chosen. There are also places where, despite no obvious change in stringency, case counts go up or down or there is no obvious temporal correlation between the measure and the change in cases. These are not scientifically valid data without strong numeric evaluation and justification.
4. Failure to consider other explanations
Population mortality rates in British Columbia were 2.5 times lower than Québec and lower than most other parts of the country, yet British Columbia had a lower stringency than most provinces Footnote 19; keeping schools open from June 2020 onwards. In fact, mortality data do not generally follow stringency indexes Footnote 6 and likely have complex explanations such as age structure Footnote 5, obesity rate Footnote 20, population density Footnote 21 and economic disparity Footnote 22.
5. Choosing inappropriate comparators
The authors choose to present specific countries—two isolated islands (New Zealand and Australia) and a country without functional land borders (South Korea)—whose outcomes were favourable early in the pandemic. However, substantial cultural, genetic, geographic and social differences may also explain lower impact early in the pandemic. The authors also conveniently forget that these countries have subsequently had massive outbreaks during the Omicron era. In fact, the heavy impact of the Omicron wave on Pacific Rim countries suggests that factors other than social choices played a role.
6. No consideration of the short and long-term costs of the interventions
Even if deemed effective in preventing disease, an honest evaluation of the impact of NPIs must also consider their costs. The British Columbia Centre for Disease Control has tracked some of these harms, which include extreme social isolation of seniors, worsening both their mental and physical health Footnote 23. For example, there was an increase in falls, which are linked with increased mortality Footnote 24. Another example is the marked increase in substance abuse in younger individuals such that overdoses were a much larger cause of death in this group than COVID-19 Footnote 25. This is also seen in the StatsCan mortality report Footnote 26, which documented an increase in non-COVID-19 deaths in Canadians under the age of 45 years. We are only beginning to understand the impact of the delay in cancer diagnoses and its effect on mortality Footnote 27.
7. Failure to disclose important conflicts of interest
The authors of this article disclose no competing interests; however, two authors are senior scientists at the Public Health Agency of Canada (one is the Chief Public Health Officer for Canada) and four are directly employed by the federal government. As key leaders responsible for decision making, they can hardly be viewed as not having competing interests in the favourable evaluation of pandemic management.
Conclusion
Canada and the world need rigorous analysis of the effectiveness and the costs of the NPI’s used to try to control COVID-19 case-counts. This analysis must be disinterested and based on comprehensive data sets. Unfortunately, this article’s failure to use real-world data, apply scientific rigour and dispassionately consider alternate hypotheses marks it as unscientific. The Canada Communicable Disease Report should not have accepted or published this study because of its lack of scientific merit and its obvious conflict of interest.
Authors’ statement
All authors contributed equally.
Competing interests
Dr. Grant has received remuneration for expert testimony pertaining to COVID-19.
Funding
None.
References
- Footnote 1
-
Smith SMS, Sonego S, Wallen GR, Waterer G, Cheng AC, Thompson P. Use of non-pharmaceutical interventions to reduce the transmission of influenza in adults: A systematic review, Respirology 2015;20(6):896–903. https://doi.org/10.1111/resp.12541
- Footnote 2
-
Jefferson T, Del Mar CB, Dooley L, Ferroni E, Al-Ansary LA, Bawazeer GA, van Driel ML, Jones MA, Thorning S, Beller EM, Clark J, Hoffmann TC, Glasziou PP, Conly JM. Physical interventions to interrupt or reduce the spread of respiratory viruses, Cochrane Database Syst Rev 2020;11(11):CD006207. https://doi.org/10.1002/14651858.CD006207.pub5
- Footnote 3
-
World Health Organization. Non-pharmaceutical public health measures for mitigating the risk and impact of epidemic and pandemic influenza. Geneva, CH: WHO; 2019. https://www.who.int/publications/i/item/non-pharmaceutical-public-health-measuresfor-mitigating-the-risk-and-impact-of-epidemic-and-pandemic-influenza
- Footnote 4
-
Budgaard H, Bundgaard JS, Raaschou-Pedersen DET, von Buchwald C, Todsen T, Norsk JB, Pries-Heje MM, Vissing CR, Nielsen PB, Winsløw UC, Fogh K, Hasselbalch R, Kristensen JH, Ringgaard A, Porsborg Andersen M, Goecke NB, Trebbien R, Skovgaard K, Benfield T, Ullum H, Torp-Pedersen C, Iversen K. Effectiveness of Adding a Mask Recommendation to Other Public Health Measures to Prevent SARS-CoV-2 Infection in Danish Mask Wearers, A Randomized Controlled Trial. Ann Int Med 2021;174(3):335–43. https://doi.org/10.7326/M20-6817
- Footnote 5
-
Mader S, Ruttenauer T. The effects of non-pharmaceutical interventions on COVID-19 mortality: A generalized synthetic control approach across 169 countries. Front Public Health 2022;10:820642. https://doi.org/10.3389/fpubh.2022.820642
- Footnote 6
-
Vickers DM, Baral S, Mishra S, Kwong JC, Sundaram M, Katz A, Calzavara A, Maheu-Giroux M, Buckeridge DL, Williamson T. Stringency of containment and closures on the growth of SARS-COV-2 in Canada prior to accelerated vaccine roll-out. Int J Infect Dis 2022;118:73–82. https://doi.org/10.1016/j.ijid.2022.02.030
- Footnote 7
-
Schirmer CM, Ringer AJ, Arthur AS, Binning MJ, Fox WC, James RF, Levitt MR, Tawk RG, Veznedaroglu E, Walker M, Spiotta AM; Endovascular Research Group (ENRG). Delayed presentation of acute ischemic strokes during the COVID-19 crisis. J Neurointerv Surg 2020;12(7):639–42. https://doi.org/10.1136/neurintsurg-2020-016299
- Footnote 8
-
Wu J, Mamas MA, Mohamed MO, Kwok CS, Roebuck C, Humberstone B, Denwood T, Luescher T, de Belder MA, Deanfield JE, Gale CP. Place and causes of acute cardiovascular mortality during the COVID-19 pandemic. Heart 2021;107:113–19. https://doi.org/10.1136/heartjnl-2020-317912
- Footnote 9
-
BC Centre for Disease Control and BC Children’s Hospital. Impact of School Closures on Learning, Child and Family Well-Being During the COVID-19 Pandemic. Vancouver, BC; BCCDC; 2020. http://www.bccdc.ca/Health-Info-Site/Documents/Public_health_COVID-19_reports/Impact_School_Closures_COVID-19.pdf
- Footnote 10
-
Ogden NH, Turgeon P, Fazil A, Clark J, Gabriele-Rivet V, Tam T, Ng V. Effects of vaccination and public health measures on COVID-19 cases in Canada. Can Commun Dis Rep 2022;48(7-8):292–302. https://doi.org/10.14745/ccdr.v48i78a01
- Footnote 11
-
Ioannidis JPA. Infection fatality rate of COVID-19 inferred from seroprevalence data. Bull World Health Organ 2021;99(1):19–33F. https://doi.org/10.2471/BLT.20.265892
- Footnote 12
-
COVID-19 Forecasting Team. Variation in the COVID-19 infection–fatality ratio by age, time, and geography during the pre-vaccine era: a systematic analysis. Lancet 2022;399(10334):1469–88. https://doi.org/10.1016/S0140-6736(21)02867-1
- Footnote 13
-
Kaspersen KA, Hindhede L, Boldesen JK, Mikkelsen S, Vestergaard LS, Berthelsen AN, Moustsen-Helms IR, Holm DK, Nilsson AC, Sækmose SG, Sørensen E, Harritshøj LH, Aagaard B, Hjalgrim H, Lillevang ST, Jørgensen CS, Krause TG, Ullum H, Pedersen OBV, Ostrowski SR, Erikstrup C. Estimation of SARS-CoV-2 Infection Fatality Rate by Age and Comorbidity Status Using Antibody Screening of Blood Donors During the COVID-19 Epidemic in Denmark. J Infect Dis 2022;225(2):219–28. https://doi.org/10.1093/infdis/jiab566
- Footnote 14
-
Erikstrup C, Laksafoss AD, Gladov J, Kaspersen KA, Mikkelsen S, Hindhede L, Boldsen JK, Jørgensen SW, Ethelberg S, Holm DK, Bruun MT, Nissen J, Schwinn M, Brodersen T, Mikkelsen C, Sækmose SG, Sørensen E, Harritshøj LH, Aagaard B, Dinh KM, Busch MP, Jørgensen CS, Krause TG, Ullum H, Ostrowski SR, Espenhain L, Pedersen OBV. Seroprevalence and infection fatality rate of the SARS-CoV-2 Omicron variant in Denmark: A nationwide serosurveillance study. Lancet Reg Health Eur 2022;21:100479. https://doi.org/10.1016/j.lanepe.2022.100479
- Footnote 15
-
Skowronski DM, Kaweski SE, Ivine MA, Kim S, Chuang ESY, Sabaiduc S, Fraser M, Reyes RC, Henry B, Levett PN, Petric M, Krajden M, Sekirov I. Serial cross-sectional estimation of vaccine and infection-induced SARS-CoV-2 sero-prevalence in children and adults, British Columbia, Canada: March 2020 to August 2022. medRxiv 2022.09.09.22279751. https://doi.org/10.1101/2022.09.09.22279751
- Footnote 16
-
COVID-19 Immunity Task Force. Seroprevalence against SARS-CoV-2 due to infection in Canada CITF; July 5, 2022. (accessed 2022-09-16). https://www.covid19immunitytaskforce.ca/wp-content/uploads/2022/07/CITF_Bespoke-report_Omicron-tsunami_2022_FINAL_ENG.pdf
- Footnote 17
-
Ioannidis JPA, Cripps S, Tanner MA. Forecasting for COVID-19 has failed. Int J Forecasting 2022;38(2):423–38. https://doi.org/10.1016/j.ijforecast.2020.08.004
- Footnote 18
-
Worldometer, Reported Cases and Deaths by Country or Territory. (accessed 2022-09-15). https://www.worldometers.info/coronavirus/
- Footnote 19
-
Bank of Canada. The Bank of Canada COVID‑19 stringency index: measuring policy response across provinces, Bank of Canada. Staff analytical note 2021-1. (accessed 2022-09-15). https://www.bankofcanada.ca/2021/02/staff-analytical-note-2021-1/
- Footnote 20
-
Gardiner J, Oben J, Sutcliffe A. Obesity as a driver of international differences in COVID-19 death rates. Diabetes Obes Metab 2021;23(7):1463–70. https://doi.org/10.1111/dom.14357
- Footnote 21
-
Chaudhry R, Dranitsaris G, Mubashir T, Bartoszko J, Riazi S. A country level analysis measuring the impact of government actions, country preparedness and socioeconomic factors on COVID-19 mortality and related health outcomes, eClinicalMedicine 2020;25:100464. https://doi.org/10.1016/j.eclinm.2020.100464
- Footnote 22
-
Bray I, Gibson A, White J. Coronavirus disease 2019 mortality: a multivariate ecological analysis in relation to ethnicity, population density, obesity, deprivation and pollution. Public Health 2020;185:261–3. https://doi.org/10.1016/j.puhe.2020.06.056
- Footnote 23
-
Office of the Provincial Health Officer and BC Centre for Disease Control. Examining the Societal Consequences of the COVID-19 Pandemic: Social Isolation of Residents in Long-term Care & Assisted Living. Vancouver, BC: BBCDC; July 26, 2021. (accessed 2022-09-15). http://www.bccdc.ca/Health-Professionals-Site/Documents/societal_consequences/Social_Isolation_LTC.pdf
- Footnote 24
-
Spaniolas K, Cheng JD, Gestring ML, Sangosanya A, Stassen NA, Bankey PE. Ground Level Falls Are Associated With Significant Mortality in Elderly Patients. J Trauma 2010;69(4):821–5. https://doi.org/10.1097/TA.0b013e3181efc6c6
- Footnote 25
-
BC Centre for Disease Control. Examining the Societal Consequences of the COVID-19 Pandemic – Increased Overdose Harms and Deaths. Vancouver, BC: BCCDC; July 15, 2021. (accessed 2022-09-15). http://www.bccdc.ca/Health-Professionals-Site/Documents/societal_consequences/Increased-Overdoses.pdf
- Footnote 26
-
Statistics Canada. Provisional death counts and excess mortality, January 2020 to June 2021. Ottawa, ON; StatCan; Sept 9, 2021. (accessed 2022-09-15). https://www150.statcan.gc.ca/n1/daily-quotidien/210909/dq210909b-eng.htm
- Footnote 27
-
Wilkinson AN. Mitigating COVID-19’s impact on missed and delayed cancer diagnoses. Can Fam Physician 2022;68:323–4. https://doi.org/10.46747/cfp.6805323
Response from the Editor-in-Chief
In this “Letter to the Editor”, Grant et al. have purported a “Failure to disclose important conflict of interest” in that the authors of this paper disclose no competing interests when such interests appear to be present.
As part of the editorial process, each author and co-author must submit the International Committee of Medical Journals Editors (ICMJE) Form for Disclosure of Potential Conflicts of Interest. If an author fails to comply with this rule, their name is removed from the author list and put in the “Acknowledgement” section at the end of the article. For this article, all seven authors provided their declaration and none of them had any relevant financial activities outside the submitted work, any patents, whether planned, pending or issued, that were broadly relevant to the work, and no other relationships, conditions, circumstances that present a potential conflict of interest beside being employees at the Public Health Agency of Canada.
As for Grant et al.’s comment “The Canada Communicable Disease Report should not have accepted or published this study because of its lack of scientific merit and its obvious conflict of interest”, the journal ensures scientific rigour through a double-blind review process and, specifically for this article, two reviewers from separate academic institutions provided their comments. Neither reviewer recommended that this study not be published.
Page details
- Date modified: