FluWatchers - Canada’s surveillance of seasonal influenza-like-illness
Published by: The Public Health Agency of Canada
Issue: Volume 47-9: FluWatchers: A Crowdsourcing Approach
Date published: September 2021
Submit a manuscript
Volume 47-9: FluWatchers: A Crowdsourcing Approach
FluWatchers: Evaluation of a crowdsourced influenza-like illness surveillance application for Canadian influenza seasons 2015–2016 to 2018–2019
Liza Lee1, Mireille Desroches1, Shamir Mukhi2, Christina Bancej1
1 Centre for Immunization and Respiratory Infectious Diseases, Public Health Agency of Canada, Ottawa, ON
2 National Microbiology Laboratory, Public Health Agency of Canada, Winnipeg, MB
Lee L, Desroches M, Mukhi S, Bancej C. FluWatchers: Evaluation of a crowdsourced influenza-like illness surveillance application for Canadian influenza seasons 2015–2016 to 2018–2019. Can Commun Dis Rep 2021;47(9):357–63. https://doi.org/10.14745/ccdr.v47i09a02
Keywords: influenza, seasonal, digital participatory surveillance, crowdsourced, Canada, FluWatchers, syndromic, influenza-like illness (ILI)
Background: Sentinel influenza-like illness (ILI) surveillance is an essential component of a comprehensive influenza surveillance program. Community-based ILI surveillance systems that rely solely on sentinel healthcare practices omit important segments of the population, including those who do not seek medical care. Participatory surveillance, which relies on community participation in surveillance, may address some limitations of traditional ILI systems.
Objective: We aimed to evaluate FluWatchers, a crowdsourced ILI application developed to complement and complete ILI surveillance in Canada.
Methods: Using established frameworks for surveillance evaluations, we assessed the acceptability, reliability, accuracy and usefulness of the FluWatchers system 2015–2016, through 2018–2019. Evaluation indicators were compared against national surveillance indicators of ILI and of laboratory confirmed respiratory virus infections.
Results: The acceptability of FluWatchers was demonstrated by growth of 50%–100% in season-over-season participation, and a consistent season-over-season retention of 80%. Reliability was greater for FluWatchers than for our traditional ILI system, although both systems had week-over-week fluctuations in the number of participants responding. FluWatchers' ILI rates had moderate correlation with weekly influenza laboratory detection rates and other winter seasonal respiratory virus detections including respiratory syncytial virus and seasonal coronaviruses. Finally, FluWatchers has demonstrated its usefulness as a source of core FluWatch surveillance information and has the potential to fill data gaps in current programs for influenza surveillance and control.
Conclusion: FluWatchers is an example of an innovative digital participatory surveillance program that was created to address limitations of traditional ILI surveillance in Canada. It fulfills the surveillance system evaluation criteria of acceptability, reliability, accuracy and usefulness.
FluWatch is Canada's national seasonal influenza surveillance program and consists of a network of laboratories, hospitals, physician offices, provincial and territorial ministries of health and CanadiansFootnote 1. FluWatch consists of seven surveillance components (geographical spread, laboratory confirmed detections, syndromic influenza-like illness (ILI) surveillance, outbreak surveillance, severe outcome surveillance, strain characterization and antiviral resistance testing and vaccine monitoring) that work together to allow FluWatch to meet three main program objectives (detect, inform and enable).
The World Health Organization (WHO) defines the global standards for the collection, reporting and analysis of seasonal influenza surveillance data and provides a framework for influenza surveillance for member statesFootnote 2. While the WHO does not mandate the exact surveillance components that every surveillance system must contain, it does recommend the inclusion of community-based surveillance of ILI as part of a comprehensive influenza surveillance systemFootnote 2.
Developed in 1996, the Sentinel Practitioner ILI Reporting System (SPIR) is the primary source for ILI surveillance data for the Public Health Agency of Canada's (PHAC) FluWatch programFootnote 1. The SPIR consists of outpatient influenza data submitted by primary care practitioners or registered nurses. There are, however, three major limitations to SPIR: it is reliant on voluntary reporting from a convenience sample of volunteer sentinel physicians or registered nurses; only data from individuals who seek medical attention are captured; and data submission is highly manual and interrupts practitioner workflow.
A growing trend is the use of hybrid surveillance systems that use digital surveillance to complement traditional surveillanceFootnote 3. One popular digital surveillance trend is participatory surveillance or crowdsourced surveillance. Participatory surveillance systems rely on volunteer members of the community to regularly share and report health information via the internet for disease surveillanceFootnote 3Footnote 4. Relying on volunteers address various limitations of traditional ILI surveillance systems, such as reporting delays, low participation and exclusion of individuals who do not seek medical care.
The need to address the limitations of SPIR and the advantages presented by participatory surveillance prompted the FluWatch program to create FluWatchers, an online participatory syndromic surveillance platform to help improve and complement ILI surveillance in Canada.
The FluWatchers system was developed on the Canadian Network for Public Health Intelligence (CNPHI) platform, an established PHAC initiative developed and managed by the National Microbiology LaboratoryFootnote 5. The CNPHI is a purpose-built scientific public health informatics and biosurveillance platformFootnote 6. Its infrastructure provides a secure, reliable and robust technical environment to facilitate and promote multi-jurisdictional collaboration, supporting the cross-domain and cross-discipline exchange of information, ideas and intelligence. The CNPHI was a natural choice to help develop the FluWatchers program, administer the weekly questionnaire and manage the data.
FluWatchers' participants complete a brief, weekly symptom-based report via an anonymous online questionnaire that asks whether the participant, and/or registered household members, have had a cough and/or fever in the past week and their influenza immunization status. Data on other symptoms, absenteeism and healthcare utilization are also collected from individuals reporting cough and fever. The weekly questionnaire is typically administered from October through May.
The objective of the present study is to present a formal evaluation of the FluWatchers program against four surveillance metrics that were adapted from the Centers for Disease Control and Prevention's Framework for Evaluating Public Health Surveillance Systems for Early Detection of OutbreaksFootnote 7:
- Acceptability—Are Canadians willing to participate in FluWatchers?
- Reliability—Are participants providing data consistently?
- Accuracy—How well does the FluWatchers data track influenza patterns in Canada?
- Usefulness—Is FluWatchers adding value to the FluWatch program?
FluWatch Sentinel Practitioner Influenza-like Illness Reporting System
The SPIR program consists of sentinel practitioners who report the total number of patient visits and the number of patient visits presenting with ILI on a weekly basis. Influenza-like illness is defined as a sudden onset of fever and cough and with one or more of the following: sore throat, joint pain, muscle aches, fatigue, which could be due to the influenza virusFootnote 1.
The weekly percentage of visits for ILI is defined as the number of patient visits to healthcare providers presenting with ILI symptoms in a given week divided by the total number of weekly patient visits to healthcare providers as a whole for that same week.
Respiratory Virus Detection Surveillance System
The Respiratory Virus Detection Surveillance System (RVDSS) is FluWatch's primary source for laboratory-based data on influenza and other seasonal respiratory viruses (adenovirus, coronavirus, enterovirus/rhinovirus, human metapneumovirus [hMPV], parainfluenza and respiratory syncytial virus [RSV])Footnote 8. The RVDSS collects weekly data from provincial, regional and some hospital labs across Canada. Laboratories report on the number of tests performed and the number of tests positive for influenza and other respiratory viruses.
The weekly percentage of tests positive for influenza and all other respiratory viruses were used for this analysis. The weekly percentage of tests positive is defined as the number of positive tests for a given virus in a given week divided by the number of tests performed for a given virus for that same week.
FluWatchers data consist of self-reported weekly episodes of cough and/or fever. For any participant reporting cough and fever, data on other symptoms experienced, absenteeism and healthcare utilization are collected. For the FluWatchers program, ILI is defined as a report of fever and cough.
The weekly percentage of FluWatchers reporting ILI is defined as the number of reports of cough and fever in a given week divided by the total number reports received by participants for that same week.
The four evaluation components were assessed as outlined in Table 1.
|Evaluation component||Indicator||Estimation method/calculation|
|Acceptability||Participation rate||Median number of weekly participants for a given season
Average weekly response rate
|Retention rate||Number of baseline participants who participated in the subsequent season|
|Reliability||Proportion of registrants who report in a given week||Percentage of weeks within ±5%, ±10% or ±15% of the median number of weekly participants|
|Accuracy||Association between FluWatchers data compared with ILI and laboratory data||Pearson correlation for FluWatchers data and the weekly percentage of tests for influenza, other respiratory viruses and the SPIR data|
|Usefulness||Contribution to detection of cases and the program's impact and value-added applications||Qualitative assessment of other applications and the additional data variables|
Analyses used data from epidemiological weeks 44 to 18 in the pilot 2015–2016 season, weeks 41 to 18 in 2016–2017 and weeks 40 to 18 in 2017–2018 and 2018–2019 to correspond to the weeks when the FluWatchers surveillance program was active. Analyses were performed in SAS 9.4 and Excel 2016.
The number of FluWatchers participants increased from a weekly median of 500 participants in season 2015–2016 to just over 3,200 participants in season 2018–2019 (Figure 1). This represents a percent increase of 98% (from seasons 2015–2016 to 2016–2017), 112% (from 2016–2017 to 2017–2018) and 52% (from 2017–2018 to 2018–2019).
Figure 1: Number of FluWatcher participants and the median number of weekly participants by season, Canada, influenza seasons 2015–2016 to 2018–2019
Text description: Figure 1
|Participants by season||505||998||2,115||3,211|
|Median weekly participants||398||749||1,422||2,188|
There was a high retention rate among participants, with 79%–80% of participants continuing their participation to the following season: approximately 60% of participants who started in the 2015–2016 season were still participating in the 2018–2019 season.
The median number of participants also increased from 398 in 2015–2016 to 2,188 in 2018–2019. The average weekly response rate was 78% in 2015–2016, 78% in 2016–2017, 74% in 2017–2018 and 74% in 2018–2019.
Across four seasons, FluWatchers was consistently more reliable than SPIR (i.e. the denominator was more consistent week to week). The percentage of weeks where the denominator (number of weekly FluWatcher participants) was found to be within ±5% of a season median ranged from 55%–64% (Table 2). This range is higher than the denominator (weekly number of patients seen) reported by SPIR sentinels, where only 26%–41% of reporting weeks were within ±5% of a respective season median. The percentage of weeks where the denominator was within ±10% or ±15% of a respective season median was always higher in the FluWatchers data. In the season 2018–2019 (Figure 2), the percentage of weeks where the denominator was found to be within ±5% of a season median was 65% in the FluWatchers data compared with 26% in the SPIR data. The percentage of weeks where the denominator was found to be within ±15% of the season median improved to 100% in the FluWatchers data and 65% in the SPIR data.
|Season||Program||Percentage of reporting weeks within given percentage of the median|
Figure 2: Number of FluWatchers participantsFootnote a and the number of patients seen by sentinel practitioners in Sentinel Practitioner Influenza-like Illness Reporting system, Canada, season 2018–2019
Text description: Figure 2
|Week||Number of participants||Median||Plus 5%||Minus 5%||Plus 10%||Minus 10%||Plus 15%||Minus 15%||Number of patients seen||Median||Plus 5%||Minus 5%||Plus 10%||Minus 10%||Plus 15%||Minus 15%|
Across four seasons, when the weekly FluWatchers ILI rates were compared with the positivity rate of influenza from national surveillance system, there was a significant and strong correlation between the two datasets (Figure 3).
Figure 3: Percentage of FluWatchers reporting cough and fever and national influenza positivity rate, Canada, seasons 2015–2016 to 2018–2019
Text description: Figure 3
|Epidemiological week||FluWatchers||Percent positive influenza|
Furthermore, when comparing the FluWatchers data to the positivity rate of other respiratory viruses across four seasons, there was either a weak or a negative correlation with adenovirus, enterovirus/rhinovirus, hMPV and parainfluenza (Table 3). There was a moderate to strong correlation between the FluWatchers data and seasonal coronavirus and RSV in all seasons except the 2015–2016 season.
|Correlation with a given virus||Season|
|RSV||0.171||0.651Footnote *||0.555Footnote *||0.598Footnote *|
|Parainfluenza||-0.451Footnote *||-0.459Footnote *||-0.323||-0.179|
|Adenovirus||-0.515Footnote *||-0.252||-0.666Footnote *||-0.374Footnote *|
|Rhinovirus/enterovirus||-0.403Footnote *||-0.521Footnote *||-0.609Footnote *||-0.296|
|Coronavirus||0.278||0.501Footnote *||0.738Footnote *||0.499Footnote *|
Weekly FluWatchers ILI rates were also compared to SPIR’s weekly percentage of visits for ILI. The Pearson correlation coefficient varied across the four seasons ranging from moderate to strong (Figure 4). As a validation measure, the weekly percentage of visits for ILI was compared with the percentage of tests positive for influenza. During the four seasons, the correlation was variable between SPIR ILI and laboratory data, with a Pearson correlation coefficient (r) of 0.858, 0.685, 0.738 and 0.501 in seasons 2015–2016, 2016–2017, 2017–2018 and 2018–2019, respectively, all with statistically significant p values.
Figure 4: Percentage of FluWatchers reporting cough and fever and percentage of visits due to influenza-like illnesses reported by sentinels, Canada, seasons 2015–2016 to 2018–2019
Text description: Figure 4
|Epidemiological week||FluWatchers||Percent positive influenza|
The FluWatchers program provided other data that had not previously been collected by PHAC or was not available weekly. These data include healthcare utilization, laboratory testing, vaccination status, absenteeism and demographic information such as age, gender, regular contact with patients and location (first three characters of a postal code—forward sortation area). Additionally, the FluWatchers program collected surveillance data from individuals who did not seek medical care or get tested.
A more detailed analysis on healthcare utilization, vaccination status and absenteeism within the FluWatchers population can be found in the publication by Desroches et al. in this issueFootnote 9.
Our analyses show that the FluWatchers program fulfills the four surveillance evaluation areas assessed.
Acceptability—Canadians are willing to participate in FluWatchers, as reflected by an increase in uptake over the four seasons and a high retention rate. In its fourth year of surveillance, the number of participants was comparable to that seen in mature participatory ILI surveillance systems, some of which have been established as early as 2005 and in countries with populations larger than CanadaFootnote 10Footnote 11. Some programs from countries with smaller populations than Canada, such as the Flutracking in Australia and the De Grote Griepmeting system in Belgium and the Netherlands, have between 15,000 and 50,000 registered users. The United States’ Flu Near You has over 50,000 users from a national population of over 327 millionFootnote 10Footnote 11. FluWatchers is still a small and relatively new program and has the potential to attract and retain more participants.
Reliability—FluWatcher participants have been consistently providing data. The percentage of weeks where the number of FluWatcher participants (denominator) was within either ±5%, ±10% or ±15% from the season median was always higher than that of the SPIR system. Influenza season in Canada often peaks around Christmas and New Years (late December, early January), when data providers such as practitioners and laboratories may be at reduced capacities. This affects the timing and the quality of data around peak influenza season. FluWatchers participation consistently dropped in late December and early January; however, the drop was not as drastic as that seen for SPIR participants. Receiving consistent and reliable data is important in surveillance to interpret trends.
Accuracy—The FluWatchers data appeared to track influenza in Canada with a moderate to strong positive correlation to our main influenza activity indicator, the percentage of laboratory tests positive for influenza. The timing of the peaks suggest that FluWatchers ILI data peaks before the influenza laboratory data. This is not unexpected because one of the aims of syndromic surveillance is to identify an increase illness activity before formal diagnoses are confirmed and reported to public health agenciesFootnote 12. The observed moderate to strong positive correlation between the FluWatchers data and seasonal coronavirus and RSV in all seasons (except the 2015–2016 season) and the weak or negative correlation with adenovirus, enterovirus/rhinovirus, hMPV and parainfluenza is also not unexpected. Seasonal coronavirus and RSV often circulate at the same time as influenza in Canada, while viruses such as enterovirus/rhinovirus often circulate outside the FluWatchers surveillance seasonFootnote 8. The FluWatchers’ ILI case definition of cough and fever could identify activity of other respiratory viruses such as RSV and seasonal coronavirus. Since FluWatchers collects data on other symptoms, the FluWatchers case definition for ILI could be tailored to be more specific to influenza.
Usefulness—FluWatchers added value to the FluWatch surveillance program by filling gaps in data that is either not collected by PHAC or not available in a timely manner. Traditional surveillance programs within FluWatch typically capture the “tip of the iceberg” of influenza cases in Canada since only reports of positive laboratory confirmed cases are collected. FluWatchers may give us a better idea about the burden of influenza in Canada by capturing cases who did not seek medical attention or get tested for influenza, and by providing data on absenteeism and healthcare utilization. Additionally, the FluWatchers data can be used to inform work on initiatives such as the WHO’s Pandemic Influenza Severity Assessment (PISA)Footnote 13. For example, “impact” is one of PISA’s three main indicators, where school and work absenteeism due to influenza is recommended as a measurement of how an influenza epidemic affects society. Currently, data on the impact indicator is not currently available for Canada, and FluWatchers could potentially fill this gap with the weekly absenteeism data it collects.
Strengths and limitations
FluWatchers does have its limitations. The FluWatchers population differs from the Canadian population as seen in the 2016 Canadian Census: FluWatchers participants, while coming from all provinces and territories, from urban and rural settings, and all age groups, genders and influenza vaccine acceptance, under-represent the tails of Canada’s age distribution and over-represent females, urban-dwelling Canadians and those who engage in health promoting behaviors as indicated by high influenza vaccine coverageFootnote 9. This is not unique to Canada as other participatory surveillance systems around the world experience the same limitations but are still able demonstrate similar trends as traditional ILI sourcesFootnote 3Footnote 4Footnote 14Footnote 15. Despite this limitation, the FluWatchers data demonstrated positive attributes of other participatory surveillance systems, such as accuracy and sensitivity and being able to measure burden of illnessFootnote 4Footnote 15. While the data provided by FluWatchers was comprehensive, further exploration of the data must be done and biases need to be quantified before using the data for other purposes than that of surveillance for ILI. The FluWatchers program is still in its infancy and public health practitioners can work towards using these data for other purposes, such as estimating vaccination coverage and effectiveness, informing disease transmission models, and supplying information for cost-benefit analyses of public health measures such as vaccination, as has been done by similar programs in other countriesFootnote 4.
FluWatchers is an example of an effective and innovative surveillance program that was created to address limitations of traditional ILI surveillance in Canada. Currently, FluWatchers ILI rates are a formal indicator under syndromic surveillance and have been incorporated into Canada’s weekly FluWatch report.
- LL — Writing, review, editing
- MD — Review, editing
- SM — Review, editing
- CB — Review, editing
The FluWatchers program would like to thank all their volunteer participants. Fifteen seconds of your time each week can make a difference in public health. For more information on the program and sign-up information, visit the following site: https://www.canada.ca/en/public-health/services/diseases/flu-influenza/fluwatcher.html.
This work was supported by the Public Health Agency of Canada.
- Footnote 1
Health Canada. Overview of influenza monitoring in Canada. Ottawa (ON): HC; 2020 (updated 2020-12-10, accessed 2020-01-29). https://www.canada.ca/en/public-health/services/diseases/flu-influenza/influenza-surveillance/about-fluwatch.html#a2.3
- Footnote 2
World Health Organization. WHO Global Epidemiological Surveillance Standards for Influenza. Geneva (Switzerland): WHO; 2013. https://www.who.int/influenza/resources/documents/WHO_Epidemiological_Influenza_Surveillance_Standards_2014.pdf
- Footnote 3
Ali ST, Cowling BJ. Influenza Virus: Tracking, Predicting, and Forecasting. Annu Rev Public Health 2021;42:43–57. https://doi.org/10.1146/annurev-publhealth-010720-021049
- Footnote 4
Wójcik OP, Brownstein JS, Chunara R, Johansson MA. Public health for the people: participatory infectious disease surveillance in the digital age. Emerg Themes Epidemiol 2014;11:7. https://doi.org/10.1186/1742-7622-11-7
- Footnote 5
Health Canada. (2020). FluWatchers. Ottawa (ON): HC; 2020 (updated 2020-09-25; accessed 2020-01-29). https://www.canada.ca/en/public-health/services/diseases/flu-influenza/fluwatcher.html
- Footnote 6
Mukhi S, Aramini J, Kabani A. Contributing to communicable diseases intelligence management in Canada: CACMID meeting, March 2007, Halifax, Nova Scotia. Can J Infect Dis Med Microbiol 2007;18(6):353–6. https://doi.org/10.1155/2007/386481
- Footnote 7
Buehler JW, Hopkins RS, Overhage JM, Sosin DM, Ton V. Framework for Evaluating Public Health Surveillance Systems for Early Detections of Outbreaks. Morb Mortal Wkly Rep (MMWR) 53(RR05):1-11. https://www.cdc.gov/mmwr/preview/mmwrhtml/rr5305a1.htm
- Footnote 8
Health Canada. Respiratory Virus Detections in Canada. Ottawa (ON): HC; 2020 (accessed 2020-01-29). https://www.canada.ca/en/public-health/services/surveillance/respiratory-virus-detections-canada.html
- Footnote 9
Desroches M, Lee L, Bancej C, Mukhi S. Representativeness of the FluWatchers Participatory Disease Surveillance Program 20152016 to 20182019: How do participants compare with the Canadian population? Can Commun Dis Rep 2021;47(9):36471. https://doi.org/10.14745/ccdr.v47i09a03
- Footnote 10
Smolinski MS, Crawley AW, Olsen JM, Jayaraman T, Libel M. Participatory Disease Surveillance: Engaging Communities Directly in Reporting, Monitoring, and Responding to Health Threats. JMIR Public Health Surveill 2017;3(4):e62. https://doi.org/10.2196/publichealth.7540
- Footnote 11
The World Bank. Population, total | Data. World Bank; 2020 (accessed 2020-01-29). https://data.worldbank.org/indicator/SP.POP.TOTL
- Footnote 12
Henning KJ. Overview of Syndromic Surveillance What is Syndromic Surveillance? Morb Mort Wkly Rep (MMWR). 2004;53(Suppl):5-11 (accessed 2020-01-29). https://www.cdc.gov/mmwr/preview/mmwrhtml/su5301a3.htm
- Footnote 13
World Health Organization. Pandemic Influenza Severity Assessment (PISA) – A WHO guide to assess the severity of influenza in seasonal epidemics and pandemics. Geneva (Switzerland): WHO; 2017. http://apps.who.int/iris/handle/10665/259392
- Footnote 14
Koppeschaar CE, Colizza V, Guerrisi C, Turbelin C, Duggan J, Edmunds WJ, Kjelsø C, Mexia R, Moreno Y, Meloni S, Paolotti D, Perrotta D, van Straten E, Franco AO. Influenzanet: Citizens Among 10 Countries Collaborating to Monitor Influenza in Europe. JMIR Public Health Surveill 2017;3(3):e66. https://doi.org/10.2196/publichealth.7429
- Footnote 15
Baltrusaitis K, Santillana M, Crawley AW, Chunara R, Smolinski M, Brownstein JS. Determinants of Participants’ Follow-Up and Characterization of Representativeness in Flu Near You, A Participatory Disease Surveillance System. JMIR Public Health Surveill. 2017;3(2):e.18. https://doi.org/10.2196/publichealth.7304
This work is licensed under a Creative Commons Attribution 4.0 International License
Report a problem or mistake on this page
- Date modified: