Evaluation of the National COVID-19 Exposure Notification App
Organization: Health Canada
Prepared by the Office of Audit and Evaluation
Health Canada and the Public Health Agency of Canada
Table of contents
- List of acronyms
- Executive summary
- Evaluation purpose and context
- Evaluation findings
- Key findings: Adherence to privacy principles
- Key findings: Governance
- Internal processes in place (within the federal community)
- External processes in place
- Key findings: Effectiveness
- Lack of OTKs issued
- Technical issues
- Burden on app users
- Privacy and trust concerns
- Perceived lack of effectiveness, interest or need by Canadians
- Conclusions and lessons learned
List of acronyms
- Assistant Deputy Minister
- Amazon Web Services
- Canadian Centre for Cybersecurity
- Canadian Digital Service
- Deputy Minister
- Federal, Provincial and Territorial
- Google/Apple Exposure Notification
- Government of Canada
- Health Canada
- Information and Privacy Commissioner of Ontario
- Innovation, Science and Economic Development Canada
- Minister’s Office
- Memorandum of Understanding
- National Health Service
- Office of the Privacy Commissioner
- One-Time Key
- Public Health Agency of Canada
- Privacy Management Division
- Public Opinion Research
- Provinces and Territories
- Quick Response
- Short Message Service
- Treasury Board Secretariat
- Terms of Reference
On March 11, 2020, the World Health Organization declared COVID-19 a global pandemic. Governments around the world quickly implemented widespread public health measures to prevent further transmission of this new virus, including self-isolation, hand washing and use of hand sanitizers, wearing non-medical face masks, and physical distancing.
At the same time, several governments started exploring innovative ways in which digital applications (apps) could complement existing public health measures, including manual contact tracing. In Canada, the federal government launched the National COVID-19 Exposure Notification app (COVID Alert app) on July 31, 2020, to help limit the spread of the virus.
The COVID Alert app was not intended to replace manual contact tracing, but rather to be “one tool in the toolbox” to help limit the spread of COVID-19. The app was designed to notify users who had been in close contact with another app user who had tested positive for COVID-19 and reported it through the app, thereby potentially reaching more people and doing so faster than through manual contact tracing.
Similar to other nations, the app was voluntary and designed to provide a high level of privacy protection. Nine provinces and territories (PTs) and the Canadian Armed Forces chose to onboard the national app; Alberta, British Columbia (BC), Yukon, and Nunavut chose not to, moreover, Alberta developed and used its own contact tracing app.
What we found
Overall, there is evidence that the design and implementation of the COVID Alert app, as well as subsequent changes, adhered to all FPT privacy principles (that is, consent and trust, legal authority, necessity and proportionality, purpose limitation, de-identification, time limitation, transparency, accountability, safeguards). Health Canada ensured these principles were upheld by:
- carrying out privacy assessments through the Privacy Management Division (PMD) and
- ensuring early and consistent engagement with the Office of the Privacy Commissioner (OPC)
To help support the design and implementation of the app, a large number of governance mechanisms were established. Health Canada engaged with a wide variety of internal partners and external stakeholders to share information and seek advice. They worked together to quickly introduce a technology that was new to all involved and, while these governance mechanisms generally worked well, some challenges persisted. In particular, there was a need for greater clarity with respect to roles and responsibilities between Health Canada and the Canadian Digital Service (CDS).
There are some indications that the app had an impact on helping to limit the spread of the virus; however, given its novelty, this was difficult to quantify in the absence of pre-determined indicators of effectiveness, such as benchmarks, targets, and public health impacts. Furthermore, a number of challenges, including issues with distribution of one-time keys (OTKs), technical issues, a burden on app users, privacy and trust concerns, and a perceived lack of effectiveness have limited the app’s effectiveness and, as a result, it may not have reached its full potential.
The evaluation reviewed a number of lines of evidence, including document and public opinion research (POR) reviews, and interviews with internal and external key informants. As a result, a few lessons learned emerged and should be considered to ensure the successful development and launch of a similar app by the Government of Canada in the future:
- streamline the process so there is less of a burden on app users to achieve the desired outcome
- when possible, consult with PTs earlier in the process to determine their needs and gain their input into the app’s design to ensure greater buy-in and engagement with all PTs
- while keeping privacy at the core of app design, explore options to increase the ability to collect data, including personal information, to aid public health measures any discussion about the collection of personal information would benefit from early engagement of internal privacy experts, as well as the OPC
- establish clear leadership among federal partners, and clearly articulate, communicate, and update the roles and responsibilities and agreements of all parties
- to the extent possible, determine what effectiveness might look like in advance, and set targets and goals accordingly
- also, take more steps to demonstrate effectiveness earlier in the process, as this may help address the public’s hesitancy to use apps in the future
Evaluation purpose and context
The purpose of the evaluation was to assess Health Canada’s activities related to the National COVID-19 Exposure Notification app (herein referred to as the COVID Alert app).
This evaluation fulfilled a commitment made in response to the Privacy Review of the COVID Alert app conducted by the Office of the Privacy Commissioner (OPC).
Evaluation scope and approach
The evaluation covered Health Canada activities related to the app from April 2020 to July 2021. These activities were carried out by Health Canada’s Testing Secretariat, Privacy Management Division (PMD), and the Communications and Public Affairs Branch, with the assistance of the Treasury Board Secretariat’s Canadian Digital Service (CDS), Innovation, Science and Economic Development Canada (ISED), and the Public Health Agency of Canada (PHAC). The activities of other government departments and other jurisdictions were not examined.
Multiple lines of evidence were used in conducting the evaluation, in order to address questions focusing on privacy principles, governance, and effectiveness. For more details on evaluation questions, and data collection and analysis, see Appendices 1 and 2.
Adherence to the privacy principles outlined in a joint statement issued by the FPT Privacy Commissioners.
Effectiveness of governance mechanisms supporting the design and implementation of the app.
Effectiveness of the app in contributing to efforts to limit the spread of COVID-19.
On March 11, 2020, the World Health Organization declared COVID-19 a global pandemic. Governments around the world quickly implemented widespread public health measures to prevent further transmission of this new virus. These included personal protective measures, such as self-isolation, hand washing and use of hand sanitizers, wearing non-medical face masks, and physical distancing.
Many governments also introduced mandatory lockdowns and stay-at-home orders. At the same time, several governments started exploring innovative ways in which digital apps could complement existing public health measures, including manual contact tracing. It is important to note that these digital apps used technology that was new to most governments at a time when knowledge of the virus was changing quickly. This situation evolved rapidly, which in turn influenced how the app was designed and implemented, as well as how its effectiveness was measured.
In Canada, the federal government launched the COVID Alert app on July 31, 2020 to help limit the spread of the virus. The app was built for Health Canada by CDS using the COVID Shield application source code developed by Shopify volunteers, and included a security review by Blackberry. As in many other countries and jurisdictions, the app operated with an exposure notification application programming interface (API) developed by Apple and Google.
The aim of the app was to complement rather than replace manual contact tracing by being “one tool in the toolbox” to help limit the spread of COVID-19. This exposure notification app would notify users if they had been in close contact with another user who had tested positive for COVID-19 and reported it through the app, potentially reaching more people and doing so faster than manual contact tracing. This notification gave people a chance to self-isolate and get tested before they started exhibiting symptoms. This has proven to be quite important, as a January 2021 study estimated that 59% of COVID-19 cases came from asymptomatic transmissions (35% pre-symptomatic and 24% from persons who never developed symptoms).Footnote 1
Similar to other nations (for example, Australia, England and Wales, Germany, Ireland, Italy, New Zealand, Switzerland), the app was voluntary and designed to be highly protective of users’ privacy, meaning that it did not collect any personal information (See Appendix 3 for further information on other countries and jurisdictions). Nine PTs and the Canadian Armed Forces chose to onboard the national app; Alberta, BC, Yukon, and Nunavut chose not to, moreover, Alberta developed and used its own contact tracing app.
The app involves several steps, as seen below in the diagram outlining the process for the COVID Alert app user process.
COVID Alert App User Process - Text description
This process flow chart illustrates the user process for the COVID Alert app, outlining the steps an app user must navigate when using the app. An icon accompanies each step.
The user process begins with awareness: Individuals are aware of the app and where to find it. The icon for this step is a megaphone.
The next step in the process is to download app: Individuals take action to find the app through an app store and download it to their device. The icon for this step is the COVID Alert symbol with a white maple leaf on a red circle background surrounded by eight grey triangles.
Following this step, the app is active on the device meaning the app is updated and actively operating on the device. Next are notifications. Individuals receive an exposure notification in the app if they were within two metres of a positive case for fifteen minutes or longer. The icon for this step is two phones exchanging information.
Above the notification received icon is the symptoms icon: Individuals experience COVID-19 symptoms. The icon for this step is a picture of a person wiping their nose with a tissue.
Symptoms and notification received lead to the next step, testing. Individuals take action to be tested following symptoms or notification of contact with a positive case. The icon for this step is a test tube.
After testing, is the step to acquire a one-time key. Following a positive test result, individuals will acquire a one-time key from the provincial or local public health authority. The icon for this step is a ten-digit alphanumeric code surrounded by a rectangular box.
The next step in the process is to enter the one-time key into the app. Individuals use the COVID Alert app by entering a one-time key into the app. This is the end of the process for an individual user. The icon for this step shows a hand holding a cell phone with an image containing COVID Alert app details.
The final step in the figure shows the feedback loop from an individual entering their one-time key and notification of individuals who have been in contact with a positive case. This icon for this step is a radio wave symbol with three half circles. This final step loops back to the notification received step.
Key findings: Adherence to privacy principles
FPT privacy principles were respected, even as changes to the app were implemented over the last year. Health Canada ensured that privacy principles were upheld by:
- conducting a privacy assessment of the app and
- updating the privacy assessment or
- conducting stand-alone assessments for changes or new features and
- engaging early and consistently with the Office of the Privacy Commissioner
Various reviews found that the design and implementation of the app adhered to FPT privacy principles.
The evaluation did not include an examination of the code or underlying IT infrastructure of the app itself to determine adherence to FPT privacy principles. Instead, we relied on the results of:
- Health Canada’s privacy analyses
- the OPC’s initial review
- security assessments by the Canadian Centre for Cybersecurity (CCCS) and
- Health Canada’s response to the OPC review
Elements of these reviews included:
- privacy assessments
- security and vulnerability assessments
- audit log review procedures
- incident management protocols
- stand-alone assessments when changes or new features were introduced to the app
- documented or communicated changes to the app and
- communications between Health Canada management and the OPC
This analysis was supplemented by reviews conducted by other independent external parties (that is, the Information and Privacy Commissioner of Ontario), and key informant interviews.
The evaluation found the Google/Apple Exposure Notification (GAEN) framework was inherently privacy preserving; it was designed with privacy in mind and did not allow for the collection of personally identifiable information. Reviews conducted in support of obtaining Authority to Operate for the app confirm that individual privacy is preserved within the app. CDS has instituted an automated audit log review procedure and incident management protocol to investigate and remedy any instances that fall outside of predetermined thresholds. Of the forty incidents investigated resulting from the audit log reviews and other sources (for example, helpdesk calls), none have resulted in instances of personal information being shared or safeguards not functioning as intended. Additionally, most internal and external key informants also confirmed that the design and implementation of the app respected the privacy principles (for example, the app was voluntary, there was no collection of personal information, it was very transparent) and the principles were integrated into every aspect of the app.
While not required, Health Canada recognized that privacy could have been a source of concern for Canadians and requested a privacy review from the OPC. Health Canada also engaged the OPC early and often to ensure the privacy principles were upheld.
In July 2020, the OPC conducted an initial privacy review of the app according to the nine FPT privacy principles associated with the COVID Alert app and the findings were shared in writing with Health Canada’s Deputy Minister. Overall, this review and various other assessments and reviews concluded that the app met the FPT privacy principles, as outlined belowFootnote 2:
|Privacy principle||Adherence to privacy principles|
|Consent and trust||The Health Canada privacy analysis and the OPC review confirmed that the use of the app was voluntary. When individuals downloaded the COVID Alert app, they were provided with an overview of how the app works, written in clear and accessible language, which included a clear statement that the app is voluntary.|
|Legal authority||The Health Canada privacy analysis and the OPC review confirmed that sufficient legal authority existed for the app in Section 4 of the Department of Health Act.|
|Necessity and proportionality||As per the Health Canada privacy analysis, the OPC review, and most internal and external key informants, the government made a conscious effort to minimize data being collected. In addition, the app responded to a clear public health need. As per the OPC, the governments of Canada and Ontario had sufficiently demonstrated that COVID Alert was likely to be effective in reducing the spread of the virus, as part of a larger set of measures, and subject to close monitoring for effectiveness once the app was in use. The OPC recommended that the app be decommissioned if it is shown to be ineffective. The effectiveness of the app is also addressed in more detail in Section 4.3.|
|Purpose limitation||As per the Health Canada privacy analysis and the OPC review, Health Canada established a clear and limited purpose for the app, which was to limit the spread of the virus by notifying users of potential exposures so they could take appropriate actions.|
|De-identification||As per the Health Canada privacy analysis, the OPC review, and the CCCS Security Assessment, there were few, if any, concerns about the app sharing personally identifiable information. The OPC review in particular stated that “the COVID Alert app contains important and very strong measures to protect users’ identity”, while the CCCS review indicated “a very high degree of confidence that the process and results yielded a secure system”. Most key informants also confirmed that the design and implementation of the app does not collect or disclose personal information of the users. The process of exchanging random codes takes place on the phone and no personal information ever leaves the phone. The CDS audit log review process and incident management protocol confirmed the results of previous reviews, as they did not identify any instances of personally identifiable information being shared.|
|Time limitation||As per the Health Canada privacy analysis and the OPC review, Health Canada committed to decommissioning the app within 30 days of the pandemic being declared over. This commitment was built into the Memorandums of Understanding (MoUs) with participating PTs. Of note, as of April 2022, the World Health Organization had not declared the pandemic over.|
|Transparency||As per the OPC review and the Health Canada response, Health Canada made the results of their own privacy assessment as well as any updates to the app publicly available at: https://www.canada.ca/en/public-health/services/diseases/coronavirus-disease-covid-19/covid-alert/privacy-policy/assessment.html. The code for the app is open source and a list of app metrics are available online. In addition to the in-app metrics, CDS collects and uses technical performance data and shares it with Health Canada. For the purposes of this evaluation, it was determined that these metrics may not be critical to the effectiveness story; however, this has no bearing on the transparency of them being collected along with other metrics and how the program is using them. A list of those being collected is available online.|
|Accountability||The Advisory Council played an accountability role by providing advice and working with the GoC to help improve the effectiveness of the app. The Advisory Council publishes its reports on the app. Health Canada considered all advice and determined what the GoC could action as a result, and then reported back to the Council to provide an update. The evaluation of the app also played an accountability role by assessing adherence to the privacy principles and the app’s effectiveness.|
|Safeguards||The Health Canada privacy analysis, the CCCS Security Analysis, and the OPC review noted several very strong safeguards in place for the app. Specifically, they mentioned that data at rest and in transit are encrypted using strong encryption methods. CDS has also implemented an automated audit log review process and a cyber event management plan to ensure that expected safeguards continue to work as planned. The one-time key process also relies on one of the strongest cryptographic hashing functions, and supports an anti-spam mechanism to ensure that fake diagnosis keys are not accidentally or maliciously uploaded. In addition, access to data on the CDS server is limited to staff with a “need-to-know” role. The CDS audit log review identified several incidents where controls and safeguards worked as intended (for example, one thwarted key claim attempt, incidents of access not working because the technologist moved to another section of the organization and had their privileges revoked accordingly). The Cyber Event Management Plan was testedFootnote 3 following its implementation, and was found to be effective in a simulated incident response. The Plan also highlighted that formalizing decision and approval processes around releasing coordinated communications should be considered as an area for continued improvement across the participating organizations. The Cyber Event Management Plan was updated based on a post-mortem from the table top testing exercise.|
Source: Internal Health Canada and CDS documents, and the OPC privacy review
Health Canada responded to the OPC’s review with a letter from the Deputy Minister to the Privacy Commissioner, outlining changes made to the app in response to the OPC review. These changes included:
- removed the word “anonymous” from the app’s privacy statement and screens, and added a clear statement that the app is voluntary
- shared OPC concerns with relevant officials at Justice Canada and ISED for consideration in future updates of the Privacy Act
- agreed to conduct an evaluation involving Health Canada and the OPC to assess respect for the FPT privacy principles in the design and implementation of the app, as well as to assess effectiveness of the app and of the various governance mechanisms supporting its design and implementation
- continued to monitor changes made by Google and Apple to their operating systems and committed to reviewing and communicating potential privacy risks to the public, where appropriate
- updated the privacy assessment as new features were considered
- made several changes to the Canada-Ontario agreement
- an assessment of the COVID Alert app by the Information and Privacy Commissioner of Ontario (IPC) confirmed that the revised Canada-Ontario agreement had addressed both the OPC concerns, and any IPC concerns
Source: Internal Health Canada and OPC documents
Changes to the app adhered to the privacy principles.
As with the implementation of the app, Health Canada’s PMD engaged with the OPC early and regularly through emails and weekly calls to highlight changes to the app. There were a few changes made to the app over the course of the last year, the main ones included the following:
- the ability to turn the app on and off which was particularly important for health care staff
- the ability to enter the date of symptom onset and the test date
- both were helpful for narrowing down the time a person was most infectious and
- the introduction of in-app metrics, starting in February 2021
PMD assessed the privacy implications of the new features and updated its privacy assessment, or conducted a separate assessment where it was warranted.Footnote 4 These assessments were shared with OPC and Health Canada’s Testing Secretariat for review, and were approved by both the Testing Secretariat’s ADM and PMD’s Director. The OPC was consulted on all changes to the app and provided advice on various aspects including the privacy notice which Health Canada then updated based on their advice. In addition, most internal key informants reported that the OPC agreed that these changes maintained adherence to the privacy principles, as previously outlined. Furthermore, several internal and external key informants reported that the new changes to the app upheld the strict privacy principles. For example, separate privacy assessments were conducted for in-app metrics and QR Codes,Footnote 5 and were submitted to the OPC and to the Testing Secretariat.
There has been widespread support for the privacy preserving nature of the app.
Several prominent Canadians, including privacy experts, publicly endorsed or were satisfied with the COVID Alert app due to its privacy safeguards, including:
- The Prime Minister of Canada, Justin Trudeau, announced publicly that the COVID Alert app does not collect personal information; he also stated that he had downloaded the app and encouraged others to do so.
- Every onboarded PT issued similar news releases when the app was launched within their jurisdiction, reassuring residents that the app would not collect location, personal, or health information. In addition, most PTs discussed the app’s privacy-preserving nature on their websites.
- The Privacy Commissioner of Canada released a statement saying, “Canadians can opt to use this technology knowing it includes very significant privacy protections”; he went on to say “I will use it”.
- The Ontario Information and Privacy Commissioner agreed with the Privacy Commissioner of Canada and provided the following statement, “This app will only work if people trust their personal information will be protected and choose to use the technology. Based on our review of the app and acceptance of our recommendations, I am satisfied that there are strong measures built in to help protect individual privacy”.
- The Office of the Information and Privacy Commissioner of Newfoundland also tweeted their support by saying, “The COVID Alert app has strong privacy protections. From a privacy perspective, it may be one of the safest apps on your phone. Newfoundland and Labrador Information and Privacy Commissioner Michael Harvey has personally downloaded the app and recommends others consider doing so also”.
Privacy experts and critics
- Michael Geist, a University of Ottawa law professor, posted a blog in which he noted that the app was voluntary, did not collect personal information, nor provide the government (or anyone else) with location information. In this endorsement, he also indicated that, from a privacy perspective, the app was very low risk. This was also confirmed by David Fraser, a Halifax-based internet and privacy lawyer, who encouraged Canadians to download the app stating, "I don't think I've ever seen a piece of software more thoroughly scrutinized and also more easily available for others to scrutinize than this particular application“.
There were minimal concerns with adherence to privacy principles.
While most internal and external key informants, including a few privacy experts, felt the design and implementation of the app respected the privacy principles, some concerns remain. For example, certain apps such as COVID Alert, which does not collect personal information, do not fall under current privacy legislation (that is, the Privacy Act). While this concern is beyond Health Canada’s control, it is worth noting that a couple of internal key informants felt adherence to the privacy principles helped fill this gap.
With its privacy-centric focus, the collection of in-app metrics was quite limited, and, at the same time, there was a concern that this initial lack of in-app metricsFootnote 6 made it difficult to fully assess and demonstrate effectiveness and accountability, which are two of the FPT Privacy Principles.
While the QR codes were never launched, discussions on their potential use also raised some concernsFootnote 7 by PTs because it was felt that it would be up to individual jurisdictions to determine whether the QR codes should be voluntary or mandatory. The concern was that QR codes would become a condition of entry, and while it would allow public health to notify a larger amount of people through venue-based notifications, some provincial representatives felt that this might undermine the voluntary nature of the app and broaden its original scope.
Key findings: Governance
Governance processes were established to help support the design and implementation of the COVID Alert app. Health Canada engaged with a wide variety of internal and external stakeholders to share information and seek advice. While these governance mechanisms are generally working well, some challenges persist. In particular, there was a need for greater clarity with respect to roles and responsibilities between Health Canada and the CDS.
Governance processes were in place to support the design and implementation of the app.
Health Canada carried out a large number of discussions and engagements with internal partners and external stakeholders to support the design and implementation of the app. Health Canada’s Testing Secretariat is the lead for this initiative and chairs many internal and external discussions. The Testing Secretariat provides co-secretariat support to the Advisory Council, along with ISED, and co-chairs the FPT working group. Several processes were in place to engage with a wide variety of stakeholders. These included sixteen different working level and senior management meetings held with internal and external representatives. These meetings were held anywhere from a daily to a monthly basis. (See Appendix 4 for further details)
Internal processes in place (within the federal community)
Internal key informants (from Health Canada and other federal partners) noted that Health Canada regularly engaged internal partners, such as CDS (technical), ISED (governance), and PHAC (public health) through meetings, calls, and emails. Health Canada’s PMD was the point of contact for privacy-related issues and they used a case management system to track all interactions with the OPC, including dates that documents were sent, and when comments were received. Even though PMD conducted its own privacy analyses with respect to the COVID Alert app, collaboration between PMD and the OPC ensured the privacy regulator was also engaged throughout various stages of the implementation of the app including a privacy review of the app’s design. OPC reported a consistent channel of communication between themselves and PMD, which resulted in meaningful and timely updates on the development of the app throughout the process.
Formal MoUs were established between Health Canada and CDS, Health Canada and the Canadian Armed Forces, and Health Canada and PTs who adopted the app. These MoUs outlined a number of areas including the timeframe for the agreements and roles and responsibilities.
In addition to the OPC and ISED, Health Canada engaged PHAC throughout the process for its public health expertise. Some internal key informants noted that PHAC’s public health advice was taken into account. For example, exposures are recorded within 2 metres, for fifteen minutes, or longer.
External processes in place
Formal governance mechanisms were established with groups outside of the federal government to support the rollout of the app, and provide advice and public health expertise. The most prominent groups in this process were the Advisory Council and the FPT Working Group.
Terms of reference (ToRs) with clear mandates and objectives were developed for both the Advisory Council and the FPT Working Group. In addition to the objectives and mandates of the two groups, the ToRs outline meeting frequency, organizational structure, and terms of service.
Formal Advisory Council meetings started on August 5, 2020 and were held, on average, every two weeks, with over two-thirds of members attending each meeting. In addition to regularly scheduled meetings, Advisory Council members attended four technical briefings between August 20 and September 10, 2020, on topics ranging from technology and privacy, to security and public health measurement. Advisory Council meetings were originally managed by ISED (who continues to serve as co-secretariat for the Council) before being co-managed by Health Canada and ISED.
The Advisory Council was mandated to provide expert advice and guidance to ensure the app met the highest standards with respect to public health outcomes, privacy, and technology.Footnote 8 To carry out its mandate, the Advisory Council established a formal work plan to help guide its work, which focused on three pillars:
- Social and Economic Determinants of App Adoption, Retention and Use
- COVID Alert as a Government Service and
- COVID Alert as a Public Health Tool
The Advisory Council committed to releasing a report for each of these three pillars to provide advice and suggestions for improvements to increase the effectiveness of the app.
In terms of the FPT working group, PTs were engaged through bilateral meetings beginning in early summer 2020 (June-July) to determine interest in adoption, to provide demonstrations of the app, and to share draft agreements. PTs were also engaged through FPT working group meetings that started on August 6, 2020. Meetings were held biweekly, on average. Attendance was strongest in the fall, with at least one representative from each of the member PTs in attendance, whereas more recent meetings have had an average of five PT representatives per meeting.
The FPT working group was established to provide public health advice and to ensure the app adds value to existing public health measures. Health Canada provided updates to the FPT Special Advisory Committee on COVID-19, including its Technical Advisory Committee and Public Health Working Group on Remote and Isolated Communities, and, at the same time, sought their advice, which has been useful given that public health is a key component of the app. Health Canada sought advice on various areas, such as:
- the length of exposure window
- user fatigue with public health measures
- impact of QR codes on public health units and
- impact of variants on app notifications
In addition to these working groups, ISED currently serves as co-secretariat for the Advisory Council and helped recruit Council members and draft the ToRs for the Advisory Council governance mechanism. A few internal key informants noted that ISED’s experience in this area was particularly helpful in setting up the Council.
Partners and stakeholders were satisfied with the level of engagement.
Most external key informants were satisfied with their level of engagement with Health Canada, stating that staff were professional and responsive (that is, Health Canada took stakeholders’ advice into consideration, acted upon it where they could or where it was warranted, and responded quickly to requests for clarification). The relationship between Health Canada and partners and stakeholders has been collaborative and respectful, with ample communication. Most Advisory Council and FPT working group key informants were satisfied with the frequency of Advisory Council and FPT Working Group meetings.
External key informants noted that Health Canada and its partners (for example, CDS, ISED) all showed a high degree of collaboration and willingness to help. CDS’ technical briefings were felt to be particularly helpful for improving partners’ understanding of the app and its features.
The Advisory Council was actively engaged and transparent in its activities.
A review of the Advisory Council member bios showed that there is a good mix of expertise on the Council, covering three main areas: privacy, technology, and public health.
A review of meeting summaries and key informant interviews found that, in addition to information sharing by Health Canada and its partners through updates and presentations, there was active discussion on various topics at each of the Council’s meetings. In addition to providing advice through meetings, the Council produced three reports. The Council has been transparent in its activities by posting its ToRs and high-level meeting summaries online. The Advisory Council provided independent external advice and guidance on potential improvements and changes on a number of issues, including:
- the ability to turn the app on and off
- increasing delivery of one-time keys (OTKs)
- increasing accessibility through additional languages
- potential use of QR codes and
- increasing uptake among Indigenous communities
This advice was taken into account by Health Canada, who explored increasing the availability of the app in several more languages, and whose Minister wrote a letter to provincial Ministers of Health to garner support for the OTK issue.
Even though the FPT Working Group was mainly an information sharing body, its advice was used to help inform new features.
A review of meeting minutes and most internal and external key informants confirmed there is good information shared at the FPT Working Group meetings. While not an exhaustive list, some topics discussed included public health measures, PT onboarding experiences, new features, OTKs and notifications, as well as app metrics.
A review of FPT working group documents, including ToRs, agendas, and meeting minutes, indicate the group has good representation from Health Canada, CDS, PHAC, the Department of National Defence, and the PTs. Representatives from all onboarded PTs regularly attended meetings. Representatives from Alberta and BC (who did not adopt the app) also attended until the beginning of March 2021.
Some internal and external key informants noted that PTs provided feedback on potential new features, such as short message service (SMS) texts and QR codes, and PTs felt that Health Canada took their advice into account in terms of whether or not, or how, to move forward with these features.
Areas for improvement
Internal roles and responsibilities should be clarified.
There was a MoU in place between Health Canada and CDS outlining roles and responsibilities between the two partners. In particular, as the business owner, Health Canada was responsible for the overall implementation and distribution of the app, and CDS provided Health Canada with support by leading the design and implementation of the technical components of the app. This MoU expired at the end of December 2020 and an updated MoU was shared and discussed with CDS; however, it was never finalized.
As reported by internal key informants, there was a perceived lack of clear direction regarding roles and responsibilities between Health Canada and CDS, which led to confusion and challenges for moving things forward efficiently. In summer 2020, CDS led the project, when the focus was on getting the app up and running. By fall and winter 2020, Health Canada assumed its business ownership role, which appears to have led to some tensions and disagreement on the approach to service delivery and timelines.
PTs could have been engaged earlier in the process to determine their needs and capacity to implement the app.
The FPT working group was mandated to advise on the public health aspects of the COVID Alert app and, while most external key informants saw the value in the information that was shared at these working group meetings, most also agreed that it was primarily a one-way exchange of information from the federal government. A few mentioned that, in the beginning, the focus was on the technical aspects of the app and less on public health issues, but this changed following those initial meetings. Also, given the rapidly evolving situation and the need for national governments to establish exposure notification technology to help limit the transmission of the virus, time was limited to consult with PTs to determine their needs. Some internal and external key informants felt that the lack of early consultation to determine the PT needs hindered collaboration and buy-in moving forward. As a result, the app was introduced without a full understanding of the needs of individual PTs and each jurisdiction’s capacity to fulfill their needed role (that is, giving out OTKs to people who tested positive for COVID-19). This could have influenced provincial and territorial buy-in for the app, especially for jurisdictions that were more interested in having a contact tracing app, as well as for those jurisdictions who were overwhelmed by the pandemic.
Key findings: Effectiveness
There are some indications the app has had an impact; however, given the novelty of this tool, it was difficult to quantify due to a lack of data and the absence of pre-determined indicators of effectiveness, such as benchmarks, targets, and public health impacts. Furthermore, a number of challenges including privacy and trust concerns, technical issues, perceived lack of effectiveness, and issues with the distribution of one-time keys (OTKs) have limited the app’s effectiveness and, as a result, it may not have reached its full potential.
As previously mentioned, app users and the app itself must undergo multiple steps to achieve desired outcomes, including:
- raising awareness
- downloading the app
- exchanging random codes with nearby phones
- receiving a notification
- getting tested
- obtaining an OTK
- entering it into the system and
- sending a notification
This section outlines how the app’s effectiveness was assessed against these steps.
There were numerous efforts to increase awareness and encourage app usage.
An extensive communications campaign was launched to increase awareness and understanding of the COVID Alert app to encourage Canadians to download and use the app. These efforts included the following:
|Advertising||Numerous campaigns, (for example, for each onboarded PT, national, Indigenous, and multilingual campaigns) ran from July 31, 2020 to March 23, 2021, resulting in 863.9 million digital impressions, 919.8 million out-of-home impressions, and up to 6.3 million link clicks.|
|Partnerships||Multiple private sector partnerships (for example, Google/Apple, Loblaws, Shoppers Drug Mart, Metro, Best Buy, Air Canada, Via Rail, Cineplex, Maple Leaf Sports and Entertainment Ltd., various financial institutions, First Nations Associations, and a number of ski associations) were established to help promote the app through social media, emails, audio and video ads, digital flyers, and posters. In addition to those listed above, internal key informants noted that telecommunications companies also helped promote awareness of the app through push notifications to let their customers know when the app was available in their jurisdiction. Apple and Google highlighted the app through their app stores.|
Health Canada conducted numerous outreach efforts to a wide variety of groups, some of which included:
Source: Internal documents
Health Canada launched organic social media campaigns to promote downloading the app. Social media campaigns focused on the best practices that app users should follow. Other government department social media accounts amplified Health Canada’s social media, as well as creating their own content to encourage Canadians to download the COVID Alert app. Top officials, such as the Prime Minister and Canada’s Chief Public Health Officer encouraged Canadians to download the app on social media and also spoke about the app in press conferences. Additionally, Health Canada engaged with Canadians on social media to answer questions regarding the app.
Health Canada also helped raise awareness through social media influencers. Health Canada contacted more than 1,000 notable Canadians to help promote the app through their social media channels. These influencers, representing all PTs but one, came from a variety of backgrounds such as:
- athletes (including entire athletic teams)
- social media content creators
- TV radio and media personalities
- other government departments
- scientists and
- medical professionals
More than 95 individual influencers actively promoted the app on their social media platforms.
Health Canada and the Canadian Digital Service also provided support to the Service Canada Call Centre by providing question and answers on the app. There was also a short video produced for use in all 210 Service Canada locations.
All of these communications efforts likely contributed to raising awareness among Canadians, which steadily increased over the last year, as demonstrated below:
- Public opinion research (POR) found that awareness of the COVID Alert app has been relatively high since its launch, ranging from 60%-64% between August and September 2020, and rising to between 73%-80% during the period of October 2020 to January 2021. Another survey in November 2020 tagged awareness of the app at nine in ten Canadians (90%).
- In late January/early February 2021, approximately seven in ten Canadians (69%) reported that they had seen, heard, or read Government of Canada advertising about the COVID Alert app.
- Among those who had seen COVID Alert app advertising, almost one in five (17%) said that they talked about the app with others as a result, and another one in ten (10%) say they downloaded the app as a result of exposure to this advertising, while six percent decided not to download the app.
Furthermore, some internal and external key informants noted that awareness of the app was also promoted through the social media platforms of certain Chief Medical Officers of Health. For example, between February 10 and 19, 2021, Newfoundland’s Chief Medical Officer of Health tweeted about the app 13 times to her nearly 30,000 followers. However, as there is no data available regarding individual PT downloads of the app, it is not possible to determine the extent to which this promotion influenced downloads. However, during the week of February 1 to 7, 2021, 12.5% of positive cases in Newfoundland entered an OTK, as compared to 30.6% the week of February 8 to 14, 2021 and 32.75% the week of February 15 to 21, 2021. This appears to demonstrate that the promotion of the app on the Chief Medical Officer of Health’s twitter account has had an impact.
While Health Canada communications and marketing efforts have likely contributed to increased awareness and the 6 million downloads of the app between July and December 2020, downloads then plateaued due to other factors limiting use (See section below on barriers for further details).
Factors such as awareness, privacy protection, usability, and perceived value of the app helped encourage downloads and use of the app.
As described in the previous section, extensive promotion of the app helped reach Canadians of all ages and in various locations. A few internal and external key informants felt that, as additional PTs onboarded the app, awareness also grew in those PTs because more individuals outside of government were talking about it. In addition, downloads of the app spiked as new PTs adopted it.
The privacy preserving nature of the app was another factor that helped encourage downloads and use. Several internal and external key informants felt that privacy aspects helped generate public confidence that the app would maintain people’s privacy, thereby encouraging downloads and easing provincial concerns. As previously mentioned, the app received multiple public endorsements and support from privacy experts and critics. These were also seen as a contributing factor to downloads of the app.
Some internal and external key informants highlighted the usability of the app as another key factor in facilitating its use. The app has been described as simple and easy to access and download. For example, the app won a plain language award from the Center for Plain Language in the United States.Footnote 9
Furthermore, POR conducted in the fall of 2020 found most COVID Alert app users were satisfied with the app (73% Sept 2020; 71% Oct 2020), citing the following reasons: it provides valuable information, is easy to use, and provides peace of mind. Similarly, seven in ten Canadians (69%) felt the COVID Alert app was useful.
Key informants also identified key factors encouraging downloads and use of the app. A few internal and external key informants noted that one of the drivers for app use was the perceived value of the app, meaning that there was an action that could be taken if a notification was received (for example, self-isolate, get tested). POR results also reported the app’s perceived value and effectiveness as driving factors for its use. Canadians who indicated having downloaded the COVID Alert app identified the following reasons for doing so:
- be notified if they had been exposed (79%)
- help reduce the spread of COVID-19 (61%)
- support contact tracing efforts (57%)
- considered it their civic duty (52%)
- tell others they had tested positive (44%) and
- peace of mind (35%)
Similarly, in an earlier study of those who downloaded the app, most did so to:
- know if they had been exposed (30%)
- protect themselves and others (18%)
- help stop the spread of the virus (15%) and
- be safe and to keep others safe (13%)
The app had the potential to have a greater impact; however, due to various challenges, it may not have reached its full potential.
Interestingly, in an August 2020 survey, while nearly three-quarters of Canadians (73%) thought the COVID Alert mobile app was a good idea, they were much less convinced that it would work; just over one-quarter (27%) thought the COVID Alert app would work, whereas 39% disagreed and one-third (34%) did not know.
A number of studies indicate Canadians’ willingness to get tested or enter their code into the app should they receive a notification or a positive COVID test result, as demonstrated by the following:
- Almost every Canadian who has downloaded or is likely to download the COVID Alert app (94%) would get tested if they received a notification on their app.
- Among those who downloaded the app, over eight in ten (82%) said they would or had entered their unique 8-digit code into the system if they were to or had tested positive for COVID.
- In another survey, two-thirds of respondents (67%) indicated they were likely to use the app to notify others if they tested positive by entering the unique code they received from public health.
However, while this POR indicates a willingness on the part of Canadians to download the app, get tested, and enter an OTK, as of July 28, 2021, 46,056 OTKs have been generated in participating PTs, of which 33,404 have been entered. Evidence shows that even though the number of OTKs entered is low, when offered an OTK, the majority of Canadians (approximately 72%) were using it to notify others (See Graph 1 below). The challenge is that not enough OTKs were distributed for a variety of reasons, such as:
- competing PT priorities
- varying systems in place
- the burden being on the person to get the OTK and
- enter it into the app
Graph 1 - Text description
Graph 1 is a line graph showing both the total number of one-time keys generated and the total number of one-time keys entered monthly from August 2020 through July 2021. The results shown in the graph are as follows:
|Month||One-time keys generated||One-time keys entered|
Between April and July 2021, more than 10,000 Canadians who tested positive for COVID-19 entered an OTK in their app, triggering more than 66,000 notifications issued to Canadians, which resulted in 640 people receiving a notification, testing positive and entering an OTK into the systemFootnote 10 (See Graph 2 below). This represented nearly seven notifications per OTK entered. Between April and July, the number of notifications per OTK entered ranged from a low of 5.4 in April 2021 to a high of 8.0 in July 2021Footnote 11, meaning that anywhere from 1 to 8 persons were reached with each notification. This ratio is higher than international comparisons, wherein data from February 2021 shows that OTKs claimed in England and Wales represented 4.41 notifications per OTK claimed, and Switzerland’s OTKs entered represented 4.22 notifications per OTK claimed.
Graph 2 - Text description
Graph 2 is a line graph showing the number of one-time keys entered, notifications issued, and number of one-time keys entered as a result of notifications from April to July 2021.
The results are as follows:
|Month||One-time keys entered||Notifications issued||Notifications with subsequent positive test and one-time key entered|
A McGill study of the app’s effectiveness used in-app metrics and app data to model averted cases and deaths in six Canadian provinces:
- Newfoundland and Labrador
- Nova Scotia
- Quebec and Saskatchewan.
The study estimated that over the period of March to July 2021, the app had averted between 6,284 and 10,894 (or 0.44% and 0.76%) of total recorded infections and between 57 and 101 deaths (or 0.004% and 0.01% of COVID-induced fatalities) in the six provinces covered by the study. This is significantly lower than the estimates for the United Kingdom, with between 284,000 and 594,000 (or 4.94% and 10.34%) cases averted, and between 4,100 and 8,700 (or 0.07% and 0.15%) deaths. The study also found that in provinces where there was more widespread adoption of the app, such as Newfoundland and Labrador and Nova Scotia, the number of cases and deaths averted were higher than for Canada as a whole. For example, in Newfoundland and Labrador, the number of cases averted was more than 60% of total cases. The study concluded that this ratio shows that the app can be an effective tool when adopted by a sufficient proportion of the population.Footnote 13
This is only a part of the app’s overall effectiveness story, since app users may be taking other measures to mitigate exposure; however, the data on these other measures is not available or tends to be anecdotal. For example, an article published in Ottawa in September 2020 highlighted the city’s first instance where the COVID Alert app was used to alert a person of a close contact, which led to that person being tested, and thereby preventing the spread of the virus.Footnote 14 In addition, several internal and external key informants mentioned the story of a teacher in Toronto who was alerted through the app, leading that person to avoid the classroom, potentially ensuring that a number of children, staff, and families not being exposed to the virus. Another such story comes from New Brunswick, where a high school principal mailed out a notice to parents encouraging their kids to download the app after an exposure occurred at the school.
A few internal and external key informants felt that, if more positive stories like these had been highlighted more often, it may have resulted in an increase in the use of the app, especially since as previously seen, one of the biggest barriers to use was that some people did not think the app worked. Publicizing these stories may have led to the release of additional stories, which could have helped tell a better effectiveness story.
The majority of internal and external key informants felt that the app did not meet its full potential as a public health tool due to a number of challenges, which are outlined in the section below.
Several barriers, such as privacy and trust concerns, technical issues, lack of one-time keys issued, burden on app users, and perceived lack of effectiveness, affected downloads and use of the app.
Lack of OTKs issued
The Advisory Council’s Pillar 2 report concluded that the app could be an effective public health tool, as long as the OTKs are widely distributed and used.Footnote 15 As previously mentioned, when comparing to cases in onboarded PTs, only 46,056 (5.0%) OTKs were generated, and even fewer (33,404 or 3.7%) were entered into the COVID Alert app by Canadians, thereby triggering a notification to other app users of a possible exposure. Not all OTKs reached Canadians for the following reasons:
- Some OTKs were generated as a test, for training purposes, or by accident.
- The pandemic did not evolve at the same pace from one jurisdiction to another. Also, each PT developed its own process for generating and distributing OTKs. The Advisory Council found that the variation in public health scripts, information technology systems, and secure online portals had an impact on the generation and distribution of OTKs.
- The Advisory Council’s Pillar 3 report recognized that introducing a new technological process (that is, OTKs) when contact tracers and case managers were often overwhelmed by COVID-19 cases represented a significant challenge for public health units.Footnote 16
If more OTKs could have been distributed and entered, the app could potentially have had a much bigger impact on reducing the spread of COVID-19. The Advisory Council stated in its Pillar 3 report that the OTK distribution issue was the biggest missed opportunity to use the app more effectively.
The Council’s Pillar 2 report concluded that technological challenges also limited the app’s ability to reach its full potential.Footnote 17 Initial technical issues, such as challenges with the Google and Apple framework, and the ability to download the app on Apple iPhone 6s or lower, may have prevented some people from downloading the app. A couple of internal key informants noted that these early frustrations with the app may have deterred Canadians from attempting to download the app a second time. Another technical issue was that some Android phones (approximately 33%) would shut down the COVID Alert app as it was working in the background in order to save battery power. This issue was addressed for some devices, but there are still devices for which the problem remains. It is possible that these glitches created barriers to downloading and properly using the app. Furthermore, the lack of overall, and particularly in-app metrics at the beginning hindered reporting capability, as a number of phones did not have apps updated to versions that included in-app metrics, as well as the specifics of the GAEN framework and opt-in reporting.
Burden on app users
Furthermore, several internal key informants noted that:
- some users did not know how to get an OTK and
- the responsibility was on the person who had just tested positive for COVID-19 to get the OTK and enter it
Some internal and external key informants noted this was challenging because, in addition to dealing with a positive diagnosis, the person could be quite ill or have family members who are ill. They may have other concerns related to their diagnosis, which means that the burden of getting an OTK, writing it down, then entering it into the system may not have been their top priority.
Similarly, the Advisory Council concluded that the number of OTKs being entered into the app was low compared to the number of positive COVID-19 cases across Canada, and for the app to be an effective tool, it must be easy for users to receive and upload an OTK following a positive diagnosis.
While the lack of standardized OTK processes across PTs created challenges in Canada, other countries such as England and Wales each have their own national health system that could send out one set of instructions, thereby facilitating the ‘OTK’ distribution process. In addition, Colorado automated its ‘OTK’ process by texting a one-time verification code to anyone who tests positive for COVID-19. Since automating its process, Colorado saw the daily average of codes sent increase from 20 to 5,000.Footnote 18
Privacy and trust concerns
Although the app strictly adhered to privacy principles and was seen by many internal and external key informants as one of its key strengths, Canadians' privacy concerns and distrust of the government potentially collecting personal or location information through the app was also seen as one of its biggest challenges.
According to one POR study, privacy (35%) was the primary concern of Canadians prior to the launch of the COVID Alert app. Similarly, in another study, when asked why they did not intend to download a contact tracing app, privacy was more of a concern immediately after the launch of the COVID Alert app, but declined over time (23% in August 2020, 15% in October 2020). However, more recent POR (February 2021) specifically looking at the COVID Alert app had privacy concerns at the top of its reasons for not downloading it.
Trust in government is also clearly an issue. When asked if they believe the government does not collect their personal information and that it does not allow the GoC to determine their location, more than one-half (52%) of survey respondents did not believe the government. In another study, Canadians who indicated that they had not downloaded the app cited not trusting the privacy of the app or the government (34%), and not wanting government to access location data (29%) among the top three reasons for not downloading the app. A number of internal and external key informants also reported that there are certain segments of the population who do not trust the government and will not download the app.
Perceived lack of effectiveness, interest or need by Canadians
A perceived lack of effectiveness of the app was cited in POR as one of the biggest reported barriers to downloading the app:
- When asked why they did not intend to download a contact tracing app, including the COVID Alert app and other provincial apps, lack of interest/need (36% August 2020 - 40% October 2020) was the primary reason, followed by an inability to do so (22%-25%).
- A number of other surveys on the COVID Alert app also found that lack of interest/no need was a main reason for not being interested in downloading the app.
- Those dissatisfied with the app (18% September 2020; 14% October 2020) felt that it was not being used by enough people to make it effective (22% and 13%, respectively), or reported they were not receiving any notifications (19% and 5%, respectively). Another study reported the main reason for the lack of confidence in the effectiveness of the COVID Alert app to reduce the spread of COVID-19 in Canada was the feeling that that not enough people were using it (55%).
- In addition, one-third (33%) of those who removed the app after originally downloading it reported that they did so because they did not think it was working or had received no notifications.
- During November 2020 and January 2021, Canadians were more confident in the ability of the COVID Alert app to protect privacy (56% and 60%, respectively) than in its effectiveness to reduce the spread of COVID-19 in Canada (44% and 38%, respectively).
Furthermore, a few internal and external key informants noted there was limited access to certain segments of the population, such as seniorsFootnote 19 and those for whom neither English nor French is their first language, which could have limited the perceived effectiveness of the app for these groups.
There were mixed views on the effectiveness of the app.
The goal of the COVID Alert app was to help contribute to a reduction in the spread of COVID-19; however, without any benchmarks or targets, it is difficult to adequately measure effectiveness. The majority of internal and external key informants had differing views regarding effectiveness of the app; some felt that one life saved made it a success, while others believed that much greater uptake of the app was required for it to be considered effective.
Canada is not the only country struggling to measure the effectiveness of its app. For example, an Oxford study, released in April 2020, was widely reported as stating that COVID-19 apps would need 60% of the country’s population to download the app if no other public health measures (for example, wearing masks, physical distancing, washing and sanitizing hands) were taken. The authors of that study attempted to correct this assumption by reporting that apps start to protect people at much lower levels of uptake (some as low as 14%). Based on these models, Canada’s app, with uptake levels of approximately 17% of the population (6,612,141 downloads as of July 28, 2021), should theoretically have had an impact.
A New York Times article (May 2021) questioned the efficacy of COVID apps and found that some public health officials and researchers in the United States had noted modest but important benefits of the app. The Chief Information Officer of University of California San Diego Health, who is responsible for California’s app, stated, “Whether it’s hundreds of lives saved or dozens or a handful, if we save lives, that’s a big deal.” Other health agencies predicted that apps would be able to notify users of virus exposures faster than manual contact tracing.Footnote 20
As mentioned, some key informants felt greater uptake was needed to increase effectiveness, but after initial spikes in downloads in July and August 2020, followed by another in October 2020, there has been minimal but continued growth in downloads since the end of December 2020 and beginning of January 2021. Based on publicly available download rates versus smartphone users, the COVID Alert app has uptake levels lower than comparable countries with similar systems and privacy regimesFootnote 21 (for example, Canada: 28%, England and Wales: 37.8%, Germany: 45.6%). The same is seen when comparing download rates versus population (for example, Canada: 17.3%, Italy: 20.2%, Switzerland: 27.1%, Germany: 36.9%, England and Wales: 37.4%, New Zealand: 59.7%).
However, the number of downloads is only part of the picture. According to in-app metrics data from July 28, 2021 of the 6,612,141 downloads of the app, there were 3.23 million active usersFootnote 22 of COVID Alert app, meaning that fewer devices had attempted to do exposure checks. This can be for a variety of reasons:
- technical issues preventing older devices from synching with COVID Alert servers and functioning as intended
- devices not connected to Wi-Fi
- users temporarily turned off the app
- users who disabled the Bluetooth or turned off the data on their phone)
All of these limit the app’s effectiveness. A McGill study (July 2021) compared the number of active users across a number of countries and found that Canada had a lower percentage of active users versus downloads (48.8%) than both Switzerland and the United Kingdom (68.4% and 78.6% respectively).Footnote 23
A December 2020 article attempted to identify the reasons some countries were successful at controlling the spread of COVID-19 and highlighted the contact tracing methods (that is, isolating infected people and their contacts, as well as using personal data such as mobile-phone signals to track adherence to the quarantine) used in countries such as South Korea as exemplary methods of preventing transmission. While these methods work in South Korea, they would not work in countries like Canada, the United Kingdom, and Germany, among others, who have chosen a much more privacy-centric digital app to complement manual contact tracing.Footnote 24
This privacy-first approach helped encourage some Canadians to download the app, but at the same time, it affected Health Canada’s ability to tell a more complete effectiveness story because of the limited information it collected. Furthermore, several internal and external key informants and studies suggested privacy may have limited the app’s effectiveness from a public health perspective. For example, the lack of demographic and location information hindered the ability to implement more targeted public health strategies.
As previously mentioned, some changes have been made to improve the app since its launch. These included:
- tightening the exposure window that allowed individuals to more accurately identify the period in which they were most infectious
- ability to turn the app on and off, which was particularly helpful for health care staff who would have registered a large number of potential exposures even though they were wearing full personal protective equipment
- technical fixes for the bucketing issue to save power:
- prior to the fix, some phones would essentially shut down apps that were working in the background
- the fix enabled the app to remain on and working to more accurately identify potential exposures
- however, this issue was not resolved for all devices
- starting in February 2021, the inclusion of in-app metrics to help tell the effectiveness story for the app
In addition to these changes, the Advisory Council, as well as most internal and external key informants, suggested the following changes as ways to further improve the effectiveness of the app:
- the inclusion of additional languages in the app
- the deployment of venue-based notifications through QR codes
- the use of tokens or wearables
- the inclusion of more information in the app (other than just notifications) and
- the use of SMS texts to streamline the OTK process
Additional languages would have ensured more people could access the app and QR codes, and would have allowed for venue-based exposure notification to notify multiple users of potential exposure. The use of tokens or wearables was explored as a solution for individuals who did not have access to a smartphone. Finally, SMS text messages with an OTK could have been sent to those who had tested positive for COVID-19 so they could enter it in the app.Footnote 25
It appears that the QR code function could have presented an opportunity to increase use of the app. Other countries like Australia, Germany, New Zealand, and England and Wales have used QR codes. For example, in England and Wales, 963,032 National Health Service (NHS) QR code posters were created; and there have been 248,004,976 venue check-ins using NHS QR code posters. According to a May 2021 survey, it appears that a notable percentage of Canadians have expressed a willingness to use QR codes to enable venue-based notifications when available, and moreover, felt this function would have had the potential to help limit the spread of COVID-19:
- Half of smartphone users (50%) say they would use the COVID Alert App to scan QR codes upon entering a business or venue.
- Approximately half of Canadians (49%) think the QR code function will be very (15%) or somewhat (34%) helpful in slowing the spread of COVID-19.
Conclusions and lessons learned
Overall, there is evidence that the design and implementation of the COVID Alert app, as well as subsequent changes, adhered to all FPT privacy principles:
- consent and trust
- legal authority
- necessity and proportionality
- purpose limitation
- time limitation
- accountability and
Health Canada ensured these principles were upheld by:
- carrying out privacy assessments conducted by the Privacy Management Division (PMD)
- ensuring early and consistent engagement with the Office of the Privacy Commissioner (OPC)
To help support the design and implementation of the app, a large number of governance mechanisms were established. Health Canada engaged with a wide variety of internal partners and external stakeholders to share information and seek advice. They worked together to quickly introduce a technology that was new to all involved and, while these governance mechanisms were generally working well, some challenges persisted. In particular, there was a need for greater clarity with respect to roles and responsibilities between Health Canada and the Canadian Digital Service (CDS).
There are some indications that the app had an impact on helping to limit the spread of the virus; however, given its novelty, it was difficult to quantify in the absence of pre-determined indicators of effectiveness, such as benchmarks, targets, and public health impacts. Furthermore, a number of challenges, including issues with the distribution of one-time keys (OTKs), technical issues, the burden on app users, privacy and trust concerns, and a perceived lack of effectiveness have limited the app’s effectiveness and, as a result, it may not have reached its full potential.
The evaluation reviewed a number of lines of evidence, including document and POR reviews, and interviews with internal and external key informants. As a result, a few lessons learned emerged and should be considered to ensure the successful development and launch of a similar app by the Government of Canada in the future:
- Streamline the process so there is less of a burden on app users to achieve the desired outcome.
- When possible, consult with PTs earlier in the process to determine their needs and gain their input into the app’s design to ensure greater buy-in and engagement with all PTs.
- While keeping privacy at the core of app design, explore options to increase the ability to collect data, including personal information, to aid public health measures. Any discussion about the collection of personal information would benefit from early engagement of internal privacy experts, as well as the OPC.
- Establish clear leadership among federal partners, and clearly articulate, communicate, and update the roles and responsibilities and agreements of all parties.
- To the extent possible, determine what effectiveness might look like in advance, and set targets and goals accordingly. Also, take more steps to demonstrate effectiveness earlier in the process, as this may help address the public’s hesitancy to use apps in the future.
Appendix 1 – Evaluation issues and questions
This evaluation covered the period from April 1, 2020 to July 31, 2021, and its scope included Health Canada’s activities related to the COVID Alert app. It did not assess the activities of partners, such as the Treasury Board Secretariat’s (TBS) Canadian Digital Services (CDS) and Innovation, Science and Economic Development Canada (ISED), nor the activities of provinces and territories (PTs).
The evaluation examined questions in relation to the adherence to privacy principles, effectiveness of the governance mechanisms, and effectiveness of the app, as shown in the table below.
Evaluation issues and questions
|Core issues||Evaluation questions||Issue type|
Adherence to privacy principles
Effectiveness of governance mechanisms
||Performance (effectiveness, economy and efficiency)|
Effectiveness of app
||Performance (effectiveness, economy and efficiency)|
Appendix 2 – Data collection and analysis methods
Evaluators collected and analyzed data from multiple sources. Data collection started in March 2021 and ended in July 2021. Data were analyzed by triangulating information gathered from the different methods listed below. The use of multiple lines of evidence and triangulation were intended to increase the reliability and credibility of the evaluation findings and conclusions.
Program document and performance data review
The evaluation reviewed a series of documents to inform findings related to the privacy principles, governance, and effectiveness of the app. Approximately 160 documents were reviewed. In addition to these documents, we reviewed a number of POR studies that examined areas such as awareness, drivers of use, barriers to use, and views on effectiveness.
Key informant interviews
Key informant interviews were conducted to gather in-depth information related to the privacy principles, governance, and effectiveness of the app. Interviews were conducted based on a predetermined interview guide. In total, 31 interviews were conducted with 39 respondents. Respondents included:
- Internal: program staff (n=10 interviews with 13 staff)
- Internal: other federal partners (n=five interviews with seven partners)*
- Other jurisdiction representatives (n=four interviews with five representatives)
- Expert groups: Advisory Council and FPT Working Group (n=six interviews with seven representatives)
- Experts and academia (n= six interviews with seven representatives).
*This includes one written response
Emerging themes from interviews were identified and quantified using NVIVO qualitative analysis software.
International and jurisdictional comparison
An international and jurisdictional comparison was conducted with eleven comparator countries and jurisdictions. Findings from a literature review (academic, peer-reviewed publications, and grey literature) helped inform the international comparison. Approximately 35 articles were reviewed. Country and jurisdiction websites and relevant effectiveness studies were also reviewed.
Data collected by these various methods was analyzed by triangulation to increase the reliability and credibility of the evaluation findings and conclusions. Still, most evaluations face constraints that may affect the validity and reliability of evaluation findings and conclusions.
The table below outlines the limitations encountered during the implementation of the selected methods for this evaluation and mitigation strategies put in place to ensure that the evaluation findings are sufficiently robust.
|Lack of benchmarks or targets||Without pre-determined indicators of effectiveness such as benchmarks, targets and public health impacts, it was challenging to measure the app’s effectiveness.||Triangulation with other lines of evidence provided as clear a picture as possible as to the impact of the app.|
|In-app metrics were only collected as of February 2021||Not having this additional data from the date of the app’s launch (July 2020) made it more challenging to fully assess the app’s effectiveness||Triangulation of other lines of evidence was used to augment available information.|
Appendix 3 – International and jurisdictional comparison
|Name of app and
|Platform used||Voluntary vs mandatory||Privacy considerations and country context||Uptake and notifications|
|Canada||COVID Alert -
July 31, 2020
|Google/Apple Exposure Notification (GAEN)||Voluntary||Federated nation that conducted a privacy review to ensure the app adheres to the nine FPT privacy principles.||As of July 28, 2021, the app has been onboarded by nine of 13 provinces and territories, and 6,612,141 users have downloaded the app and entered 33,404 OTKs into the system.|
|BlueTrace||Voluntary||Province with a privacy statement that describes the type of personal information it will collect.||More than 300,000 people have downloaded the app, and according to the most recent numbers (March 2021), the app had been used to track 32 positive COVID-19 cases and 141 close contacts.|
|Amazon Web Services (AWS)||Voluntary||Federated nation with a Privacy Act, privacy principles and a privacy assessment.||As of April 6, 2021, the COVIDSafe application was downloaded 7 million times.|
|Colorado||CO Exposure Notifications -
October 25, 2020
|NHS COVID-19 app -
|GAEN||Voluntary||Unitary constitutional monarch with a privacy notice and data protection principles.||As of August 3, 2021, the app has been downloaded 27,423,906 times, 963,032 NHS QR code posters have been created, and there have been 248,004,976 venue check-ins using NHS QR code posters.|
|GAEN||Voluntary||Federated nation with a privacy assessment.||The Corona-Warn-App has over 27 million downloads.|
|Ireland||COVID Tracker Ireland - July 7, 2020||GAEN||Voluntary||Constitutional Republic with a data protection information notice and a data protection impact assessment.||As of 27 April 2021, over 2,510,000 people have downloaded the app. There is an active user base of 1.3 million.|
June 15, 2020
|GAEN||Voluntary||Federated nation whose app does not collect personal information, nor does it track the user’s location.||As of August 11, 2021, the app has been downloaded 13,772,014 times. COVID positive users who uploaded their keys: 21,255|
|New Zealand||NZ COVID Tracer -
May 20, 2020
|AWS||Voluntary||Federated nation with a privacy assessment.||As of June 8 2021, there were 2,844,910 app registrations, 587,731 QR Codes generated, and 138,673,630 active devices.|
March 20, 2020
|BlueTrace||Partially voluntary||Independent Republic with a privacy assessment.||As of 11 May 2021, the adoption rate had exceeded 92%, with 4,923,054 individuals aged above six. The app helped to identify about 25,000 close contacts of COVID-19 cases, of which 160 eventually tested positive.|
|South Korea||There is no contact tracing or exposure notification app||SMS text messages||Mandatory||The legal framework for the Korean government’s contact tracing methods allows relevant authorities to collect information on confirmed cases or suspected cases.||Text messages are sent to cellphones with information every time there is a positive case in a particular region.|
|Switzerland||SwissCovid app -
June 25, 2020
|GAEN||Voluntary||Federated nation with a risk assessment and high data protection requirements||As of August 2, 2021, the SwissCovid app was downloaded 3,292,200 times.|
Appendix 4 – Governance mechanisms
|Advisory Council||Biweekly||External advisory body|
|FPT Working Group||Biweekly||ADM co-chair, working level PT reps|
|DM Briefs||Daily||HC DM and Testing Secretariat DM|
|QR Code Prime||Daily||HC and CDS|
|COVID Alert TSK Stand Up||Twice a week||HC and ISED (working level)|
|Interdepartmental COVID Alert Check-in||Twice a week||HC, PHAC, CDS, ISED (working level)|
|MINO||Weekly||MINO, HC ADM, DGs and Testing Secretariat DM|
|Executive||Weekly||Testing task force executive|
|CDS Generic Inbox Sync||Weekly||HC, Service Canada, and CDS|
|QR Code Discussion||Weekly
|HC and CDS
|CDS Partnership||Weekly||HC and CDS|
|Four Corners||Weekly||MO, PMO, HC, CDS, TBS, and ISED|
|Testing Secretariat Executive||Biweekly||FPTs and private sector|
|ADM FPT Working Group||Biweekly||FPT ADMs in charge of testing, contact tracing, and data management|
|PMD and OPC||Biweekly (weekly at first)||PMD and OPC|
|Monthly PT Touchpoint||Monthly||PT Health Ministries – add in weekly PMD and CDS meetings|
- Footnote 1
Johansson, M., Quandelacy, T., Kada, S. et al. (2021-01-07). SARS-CoV-2 Transmission From People Without COVID-19 Symptoms. Journal of the American Medical Association, 4(1). Retrieved from https://jamanetwork.com/journals/jamanetworkopen/fullarticle/2774707
- Footnote 2
Office of the Privacy Commissioner of Canada. (2020-05-07). Supporting public health, building public trust: Privacy principles for contact tracing and similar apps: Joint Statement by Federal, Provincial and Territorial Privacy Commissioners. Retrieved from: https://www.priv.gc.ca/en/opc-news/speeches/2020/s-d_20200507/
- Footnote 3
The plan was tested through a virtual tabletop exercise organized by the Treasury Board Secretariat’s (TBS) Office of the Chief Information Officer’s cybersecurity division and including participants from TBS, CDS, Health Canada and the CCCS.
- Footnote 4
If privacy implications were minimal, rather than conducting a full assessment, PMD updated the main privacy assessment (for example, symptom onset, testing date).
- Footnote 5
QR codes stand for quick response codes. These are a type of barcode or pattern containing various forms of data that can be scanned. While this feature was never launched, PMD drafted a complete privacy analysis.
- Footnote 6
Health Canada began collecting in-app metrics in February 2021.
- Footnote 7
Under this proposal, people would be required to scan a QR code as a condition of entry to a venue.
- Footnote 8
Innovation, Science and Economic Development Canada. (2020-07-31). Terms of Reference of the Advisory Council to Federal, Provincial and Territorial Governments on the National COVID-19 Exposure Notification App. Retrieved from: https://www.ic.gc.ca/eic/site/icgc.nsf/eng/07688.html
- Footnote 9
Center for Plain Language. (2021). 2021 ClearMark Winners. Retrieved from: https://centerforplainlanguage.org/2021-clearmark-winners/
- Footnote 10
While the number of people who tested positive for COVID-19 and entered an OTK has been collected since October 2020 (for a total of more than 33,000), the number of notifications issued and the number of people receiving a notification, testing positive and entering an OTK into the system has only been collected since February and April respectively.
- Footnote 11
Of note, there were fewer than 11,000 cases in participating PTs in July 2021.
- Footnote 12
Government of Canada. (2021-07-28). COVID Alert Performance Metrics. Retrieved from https://www.canada.ca/en/public-health/services/diseases/coronavirus-disease-covid-19/covid-alert/performance-metrics.html
- Footnote 13
Ruths, D., Moodie, E., Shaw, M., Sun, S. (2021-08-24). COVID Alert Application Effectiveness. Retrieved from: http://derekruths.com/posts/covid_alert/
- Footnote 14
CTV News. (2020-09-02). Ottawa resident tests positive for COVID-19 after notification from COVID Alert App. Retrieved from: https://ottawa.ctvnews.ca/ottawa-resident-tests-positive-for-covid-19-after-notification-from-covid-alert-app-1.5089953
- Footnote 15
COVID-19 Exposure Notification App Advisory Council. (2021-08). Interim report: COVID Alert as a Government Service. Retrieved from: https://www.ic.gc.ca/eic/site/icgc.nsf/eng/07740.html
- Footnote 16
COVID-19 Exposure Notification App Advisory Council. (2021-08). COVID Alert as a Public Health Tool. Retrieved from: https://www.ic.gc.ca/eic/site/icgc.nsf/eng/07744.html
- Footnote 17
COVID-19 Exposure Notification App Advisory Council. (2021-08). Interim report: COVID Alert as a Government Service. Retrieved from: https://www.ic.gc.ca/eic/site/icgc.nsf/eng/07740.html
- Footnote 18
Colorado Department of Public Health and Environment. (2020-11-13). CO Exposure Notifications enabled by 1 million Coloradans. Retrieved from: https://covid19.colorado.gov/press-release/co-exposure-notifications-enabled-by-1-million-coloradans
- Footnote 19
However, it should be noted that some seniors may include long-term care individuals who would not have the app, nor the ability or need to enter an OTK.
- Footnote 20
Singer, Natasha. (2021-05-27). Why Apple and Google’s Virus Alert Apps Had Limited Success. Retrieved from: https://www.nytimes.com/2021/05/27/business/apple-google-virus-tracing-app.html
- Footnote 21
Of note, Canada could have had lower uptake levels partially due to the fact that there were four PTs who did not use the app.
- Footnote 22
The active user metric indicates the number of devices that have attempted to do an exposure check.
- Footnote 23
Ruths, D., Moodie, E., Shaw, M., Sun, S. (2021-08-24). COVID Alert Application Effectiveness. Retrieved from: http://derekruths.com/posts/covid_alert/
- Footnote 24
Lewis, Dyani. (2020-12-17). Why many countries failed at COVID contact-tracing — but some got it right. Nature. Retrieved from: https://www.nature.com/articles/d41586-020-03518-4
- Footnote 25
COVID-19 Exposure Notification App Advisory Council. (2021-08). Interim report: COVID Alert as a Government Service. Retrieved from: https ://www.ic.gc.ca/eic/site/icgc.nsf/eng/07740.html
Report a problem or mistake on this page
- Date modified: