Language selection

Government of Canada / Gouvernement du Canada

Search

Evaluation of the National COVID-19 Exposure Notification App

Download in PDF format
(1.1 MB, 45 pages)

Organization: Health Canada

Published: 2022-06-20

ISBN: 978-0-660-44028-6

Prepared by the Office of Audit and Evaluation
Health Canada and the Public Health Agency of Canada
June 2022

Table of contents

List of acronyms

ADM
Assistant Deputy Minister
AWS
Amazon Web Services
CCCS
Canadian Centre for Cybersecurity
CDS
Canadian Digital Service
DM
Deputy Minister
FPT
Federal, Provincial and Territorial
GAEN
Google/Apple Exposure Notification
GoC
Government of Canada
HC
Health Canada
IPC
Information and Privacy Commissioner of Ontario
ISED
Innovation, Science and Economic Development Canada
MINO
Minister’s Office
MoU
Memorandum of Understanding
NHS
National Health Service
OPC
Office of the Privacy Commissioner
OTK
One-Time Key
PHAC
Public Health Agency of Canada
PMD
Privacy Management Division
POR
Public Opinion Research
PTs
Provinces and Territories
QR
Quick Response
SMS
Short Message Service
TBS
Treasury Board Secretariat
ToRs
Terms of Reference

Executive summary

Context

On March 11, 2020, the World Health Organization declared COVID-19 a global pandemic. Governments around the world quickly implemented widespread public health measures to prevent further transmission of this new virus, including self-isolation, hand washing and use of hand sanitizers, wearing non-medical face masks, and physical distancing.

At the same time, several governments started exploring innovative ways in which digital applications (apps) could complement existing public health measures, including manual contact tracing. In Canada, the federal government launched the National COVID-19 Exposure Notification app (COVID Alert app) on July 31, 2020, to help limit the spread of the virus.

The COVID Alert app was not intended to replace manual contact tracing, but rather to be “one tool in the toolbox” to help limit the spread of COVID-19. The app was designed to notify users who had been in close contact with another app user who had tested positive for COVID-19 and reported it through the app, thereby potentially reaching more people and doing so faster than through manual contact tracing.

Similar to other nations, the app was voluntary and designed to provide a high level of privacy protection. Nine provinces and territories (PTs) and the Canadian Armed Forces chose to onboard the national app; Alberta, British Columbia (BC), Yukon, and Nunavut chose not to, moreover, Alberta developed and used its own contact tracing app.

What we found

Overall, there is evidence that the design and implementation of the COVID Alert app, as well as subsequent changes, adhered to all FPT privacy principles (that is, consent and trust, legal authority, necessity and proportionality, purpose limitation, de-identification, time limitation, transparency, accountability, safeguards). Health Canada ensured these principles were upheld by:

To help support the design and implementation of the app, a large number of governance mechanisms were established. Health Canada engaged with a wide variety of internal partners and external stakeholders to share information and seek advice. They worked together to quickly introduce a technology that was new to all involved and, while these governance mechanisms generally worked well, some challenges persisted. In particular, there was a need for greater clarity with respect to roles and responsibilities between Health Canada and the Canadian Digital Service (CDS).

There are some indications that the app had an impact on helping to limit the spread of the virus; however, given its novelty, this was difficult to quantify in the absence of pre-determined indicators of effectiveness, such as benchmarks, targets, and public health impacts. Furthermore, a number of challenges, including issues with distribution of one-time keys (OTKs), technical issues, a burden on app users, privacy and trust concerns, and a perceived lack of effectiveness have limited the app’s effectiveness and, as a result, it may not have reached its full potential.

Lessons learned

The evaluation reviewed a number of lines of evidence, including document and public opinion research (POR) reviews, and interviews with internal and external key informants. As a result, a few lessons learned emerged and should be considered to ensure the successful development and launch of a similar app by the Government of Canada in the future:

Evaluation purpose and context

Evaluation purpose

The purpose of the evaluation was to assess Health Canada’s activities related to the National COVID-19 Exposure Notification app (herein referred to as the COVID Alert app).

This evaluation fulfilled a commitment made in response to the Privacy Review of the COVID Alert app conducted by the Office of the Privacy Commissioner (OPC).

Evaluation scope and approach

The evaluation covered Health Canada activities related to the app from April 2020 to July 2021. These activities were carried out by Health Canada’s Testing Secretariat, Privacy Management Division (PMD), and the Communications and Public Affairs Branch, with the assistance of the Treasury Board Secretariat’s Canadian Digital Service (CDS), Innovation, Science and Economic Development Canada (ISED), and the Public Health Agency of Canada (PHAC). The activities of other government departments and other jurisdictions were not examined.

Multiple lines of evidence were used in conducting the evaluation, in order to address questions focusing on privacy principles, governance, and effectiveness. For more details on evaluation questions, and data collection and analysis, see Appendices 1 and 2.

Privacy principles

Adherence to the privacy principles outlined in a joint statement issued by the FPT Privacy Commissioners.

Governance

Effectiveness of governance mechanisms supporting the design and implementation of the app.

Effectiveness

Effectiveness of the app in contributing to efforts to limit the spread of COVID-19.

Context

Program context

On March 11, 2020, the World Health Organization declared COVID-19 a global pandemic. Governments around the world quickly implemented widespread public health measures to prevent further transmission of this new virus. These included personal protective measures, such as self-isolation, hand washing and use of hand sanitizers, wearing non-medical face masks, and physical distancing.

Many governments also introduced mandatory lockdowns and stay-at-home orders. At the same time, several governments started exploring innovative ways in which digital apps could complement existing public health measures, including manual contact tracing. It is important to note that these digital apps used technology that was new to most governments at a time when knowledge of the virus was changing quickly. This situation evolved rapidly, which in turn influenced how the app was designed and implemented, as well as how its effectiveness was measured.

In Canada, the federal government launched the COVID Alert app on July 31, 2020 to help limit the spread of the virus. The app was built for Health Canada by CDS using the COVID Shield application source code developed by Shopify volunteers, and included a security review by Blackberry. As in many other countries and jurisdictions, the app operated with an exposure notification application programming interface (API) developed by Apple and Google.

The aim of the app was to complement rather than replace manual contact tracing by being “one tool in the toolbox” to help limit the spread of COVID-19. This exposure notification app would notify users if they had been in close contact with another user who had tested positive for COVID-19 and reported it through the app, potentially reaching more people and doing so faster than manual contact tracing. This notification gave people a chance to self-isolate and get tested before they started exhibiting symptoms. This has proven to be quite important, as a January 2021 study estimated that 59% of COVID-19 cases came from asymptomatic transmissions (35% pre-symptomatic and 24% from persons who never developed symptoms).Footnote 1

Similar to other nations (for example, Australia, England and Wales, Germany, Ireland, Italy, New Zealand, Switzerland), the app was voluntary and designed to be highly protective of users’ privacy, meaning that it did not collect any personal information (See Appendix 3 for further information on other countries and jurisdictions). Nine PTs and the Canadian Armed Forces chose to onboard the national app; Alberta, BC, Yukon, and Nunavut chose not to, moreover, Alberta developed and used its own contact tracing app.

The app involves several steps, as seen below in the diagram outlining the process for the COVID Alert app user process.

COVID Alert App User Process
COVID Alert App User Process - Text description

This process flow chart illustrates the user process for the COVID Alert app, outlining the steps an app user must navigate when using the app. An icon accompanies each step.

The user process begins with awareness: Individuals are aware of the app and where to find it. The icon for this step is a megaphone.

The next step in the process is to download app: Individuals take action to find the app through an app store and download it to their device. The icon for this step is the COVID Alert symbol with a white maple leaf on a red circle background surrounded by eight grey triangles.

Following this step, the app is active on the device meaning the app is updated and actively operating on the device. Next are notifications. Individuals receive an exposure notification in the app if they were within two metres of a positive case for fifteen minutes or longer. The icon for this step is two phones exchanging information.

Above the notification received icon is the symptoms icon: Individuals experience COVID-19 symptoms. The icon for this step is a picture of a person wiping their nose with a tissue.

Symptoms and notification received lead to the next step, testing. Individuals take action to be tested following symptoms or notification of contact with a positive case. The icon for this step is a test tube.
After testing, is the step to acquire a one-time key. Following a positive test result, individuals will acquire a one-time key from the provincial or local public health authority. The icon for this step is a ten-digit alphanumeric code surrounded by a rectangular box.

The next step in the process is to enter the one-time key into the app. Individuals use the COVID Alert app by entering a one-time key into the app. This is the end of the process for an individual user. The icon for this step shows a hand holding a cell phone with an image containing COVID Alert app details.

The final step in the figure shows the feedback loop from an individual entering their one-time key and notification of individuals who have been in contact with a positive case. This icon for this step is a radio wave symbol with three half circles. This final step loops back to the notification received step.

Evaluation findings

Key findings: Adherence to privacy principles

FPT privacy principles were respected, even as changes to the app were implemented over the last year. Health Canada ensured that privacy principles were upheld by:

Various reviews found that the design and implementation of the app adhered to FPT privacy principles.

The evaluation did not include an examination of the code or underlying IT infrastructure of the app itself to determine adherence to FPT privacy principles. Instead, we relied on the results of:

Elements of these reviews included:

This analysis was supplemented by reviews conducted by other independent external parties (that is, the Information and Privacy Commissioner of Ontario), and key informant interviews.

The evaluation found the Google/Apple Exposure Notification (GAEN) framework was inherently privacy preserving; it was designed with privacy in mind and did not allow for the collection of personally identifiable information. Reviews conducted in support of obtaining Authority to Operate for the app confirm that individual privacy is preserved within the app. CDS has instituted an automated audit log review procedure and incident management protocol to investigate and remedy any instances that fall outside of predetermined thresholds. Of the forty incidents investigated resulting from the audit log reviews and other sources (for example, helpdesk calls), none have resulted in instances of personal information being shared or safeguards not functioning as intended. Additionally, most internal and external key informants also confirmed that the design and implementation of the app respected the privacy principles (for example, the app was voluntary, there was no collection of personal information, it was very transparent) and the principles were integrated into every aspect of the app.

While not required, Health Canada recognized that privacy could have been a source of concern for Canadians and requested a privacy review from the OPC. Health Canada also engaged the OPC early and often to ensure the privacy principles were upheld.

In July 2020, the OPC conducted an initial privacy review of the app according to the nine FPT privacy principles associated with the COVID Alert app and the findings were shared in writing with Health Canada’s Deputy Minister. Overall, this review and various other assessments and reviews concluded that the app met the FPT privacy principles, as outlined belowFootnote 2:

Table 1: Adherence to the privacy principles
Privacy principle Adherence to privacy principles
Consent and trust The Health Canada privacy analysis and the OPC review confirmed that the use of the app was voluntary. When individuals downloaded the COVID Alert app, they were provided with an overview of how the app works, written in clear and accessible language, which included a clear statement that the app is voluntary.
Legal authority The Health Canada privacy analysis and the OPC review confirmed that sufficient legal authority existed for the app in Section 4 of the Department of Health Act.
Necessity and proportionality As per the Health Canada privacy analysis, the OPC review, and most internal and external key informants, the government made a conscious effort to minimize data being collected. In addition, the app responded to a clear public health need. As per the OPC, the governments of Canada and Ontario had sufficiently demonstrated that COVID Alert was likely to be effective in reducing the spread of the virus, as part of a larger set of measures, and subject to close monitoring for effectiveness once the app was in use. The OPC recommended that the app be decommissioned if it is shown to be ineffective. The effectiveness of the app is also addressed in more detail in Section 4.3.
Purpose limitation As per the Health Canada privacy analysis and the OPC review, Health Canada established a clear and limited purpose for the app, which was to limit the spread of the virus by notifying users of potential exposures so they could take appropriate actions.
De-identification As per the Health Canada privacy analysis, the OPC review, and the CCCS Security Assessment, there were few, if any, concerns about the app sharing personally identifiable information. The OPC review in particular stated that “the COVID Alert app contains important and very strong measures to protect users’ identity”, while the CCCS review indicated “a very high degree of confidence that the process and results yielded a secure system”. Most key informants also confirmed that the design and implementation of the app does not collect or disclose personal information of the users. The process of exchanging random codes takes place on the phone and no personal information ever leaves the phone. The CDS audit log review process and incident management protocol confirmed the results of previous reviews, as they did not identify any instances of personally identifiable information being shared.
Time limitation As per the Health Canada privacy analysis and the OPC review, Health Canada committed to decommissioning the app within 30 days of the pandemic being declared over. This commitment was built into the Memorandums of Understanding (MoUs) with participating PTs. Of note, as of April 2022, the World Health Organization had not declared the pandemic over.
Transparency As per the OPC review and the Health Canada response, Health Canada made the results of their own privacy assessment as well as any updates to the app publicly available at: https://www.canada.ca/en/public-health/services/diseases/coronavirus-disease-covid-19/covid-alert/privacy-policy/assessment.html. The code for the app is open source and a list of app metrics are available online. In addition to the in-app metrics, CDS collects and uses technical performance data and shares it with Health Canada. For the purposes of this evaluation, it was determined that these metrics may not be critical to the effectiveness story; however, this has no bearing on the transparency of them being collected along with other metrics and how the program is using them. A list of those being collected is available online.
Accountability The Advisory Council played an accountability role by providing advice and working with the GoC to help improve the effectiveness of the app. The Advisory Council publishes its reports on the app. Health Canada considered all advice and determined what the GoC could action as a result, and then reported back to the Council to provide an update. The evaluation of the app also played an accountability role by assessing adherence to the privacy principles and the app’s effectiveness.
Safeguards The Health Canada privacy analysis, the CCCS Security Analysis, and the OPC review noted several very strong safeguards in place for the app. Specifically, they mentioned that data at rest and in transit are encrypted using strong encryption methods. CDS has also implemented an automated audit log review process and a cyber event management plan to ensure that expected safeguards continue to work as planned. The one-time key process also relies on one of the strongest cryptographic hashing functions, and supports an anti-spam mechanism to ensure that fake diagnosis keys are not accidentally or maliciously uploaded. In addition, access to data on the CDS server is limited to staff with a “need-to-know” role. The CDS audit log review identified several incidents where controls and safeguards worked as intended (for example, one thwarted key claim attempt, incidents of access not working because the technologist moved to another section of the organization and had their privileges revoked accordingly). The Cyber Event Management Plan was testedFootnote 3 following its implementation, and was found to be effective in a simulated incident response. The Plan also highlighted that formalizing decision and approval processes around releasing coordinated communications should be considered as an area for continued improvement across the participating organizations. The Cyber Event Management Plan was updated based on a post-mortem from the table top testing exercise.

Source: Internal Health Canada and CDS documents, and the OPC privacy review

Health Canada responded to the OPC’s review with a letter from the Deputy Minister to the Privacy Commissioner, outlining changes made to the app in response to the OPC review. These changes included:

Source: Internal Health Canada and OPC documents

Changes to the app adhered to the privacy principles.

As with the implementation of the app, Health Canada’s PMD engaged with the OPC early and regularly through emails and weekly calls to highlight changes to the app. There were a few changes made to the app over the course of the last year, the main ones included the following:

PMD assessed the privacy implications of the new features and updated its privacy assessment, or conducted a separate assessment where it was warranted.Footnote 4 These assessments were shared with OPC and Health Canada’s Testing Secretariat for review, and were approved by both the Testing Secretariat’s ADM and PMD’s Director. The OPC was consulted on all changes to the app and provided advice on various aspects including the privacy notice which Health Canada then updated based on their advice. In addition, most internal key informants reported that the OPC agreed that these changes maintained adherence to the privacy principles, as previously outlined. Furthermore, several internal and external key informants reported that the new changes to the app upheld the strict privacy principles. For example, separate privacy assessments were conducted for in-app metrics and QR Codes,Footnote 5 and were submitted to the OPC and to the Testing Secretariat.

There has been widespread support for the privacy preserving nature of the app.

Several prominent Canadians, including privacy experts, publicly endorsed or were satisfied with the COVID Alert app due to its privacy safeguards, including:

Politicians

Privacy Commissioners

Privacy experts and critics

There were minimal concerns with adherence to privacy principles.

While most internal and external key informants, including a few privacy experts, felt the design and implementation of the app respected the privacy principles, some concerns remain. For example, certain apps such as COVID Alert, which does not collect personal information, do not fall under current privacy legislation (that is, the Privacy Act). While this concern is beyond Health Canada’s control, it is worth noting that a couple of internal key informants felt adherence to the privacy principles helped fill this gap.

With its privacy-centric focus, the collection of in-app metrics was quite limited, and, at the same time, there was a concern that this initial lack of in-app metricsFootnote 6 made it difficult to fully assess and demonstrate effectiveness and accountability, which are two of the FPT Privacy Principles.

While the QR codes were never launched, discussions on their potential use also raised some concernsFootnote 7 by PTs because it was felt that it would be up to individual jurisdictions to determine whether the QR codes should be voluntary or mandatory. The concern was that QR codes would become a condition of entry, and while it would allow public health to notify a larger amount of people through venue-based notifications, some provincial representatives felt that this might undermine the voluntary nature of the app and broaden its original scope.

Key findings: Governance

Governance processes were established to help support the design and implementation of the COVID Alert app. Health Canada engaged with a wide variety of internal and external stakeholders to share information and seek advice. While these governance mechanisms are generally working well, some challenges persist. In particular, there was a need for greater clarity with respect to roles and responsibilities between Health Canada and the CDS.

Governance processes were in place to support the design and implementation of the app.

Health Canada carried out a large number of discussions and engagements with internal partners and external stakeholders to support the design and implementation of the app. Health Canada’s Testing Secretariat is the lead for this initiative and chairs many internal and external discussions. The Testing Secretariat provides co-secretariat support to the Advisory Council, along with ISED, and co-chairs the FPT working group. Several processes were in place to engage with a wide variety of stakeholders. These included sixteen different working level and senior management meetings held with internal and external representatives. These meetings were held anywhere from a daily to a monthly basis. (See Appendix 4 for further details)

Internal processes in place (within the federal community)

Internal key informants (from Health Canada and other federal partners) noted that Health Canada regularly engaged internal partners, such as CDS (technical), ISED (governance), and PHAC (public health) through meetings, calls, and emails. Health Canada’s PMD was the point of contact for privacy-related issues and they used a case management system to track all interactions with the OPC, including dates that documents were sent, and when comments were received. Even though PMD conducted its own privacy analyses with respect to the COVID Alert app, collaboration between PMD and the OPC ensured the privacy regulator was also engaged throughout various stages of the implementation of the app including a privacy review of the app’s design. OPC reported a consistent channel of communication between themselves and PMD, which resulted in meaningful and timely updates on the development of the app throughout the process.

Formal MoUs were established between Health Canada and CDS, Health Canada and the Canadian Armed Forces, and Health Canada and PTs who adopted the app. These MoUs outlined a number of areas including the timeframe for the agreements and roles and responsibilities.

In addition to the OPC and ISED, Health Canada engaged PHAC throughout the process for its public health expertise. Some internal key informants noted that PHAC’s public health advice was taken into account. For example, exposures are recorded within 2 metres, for fifteen minutes, or longer.

External processes in place

Formal governance mechanisms were established with groups outside of the federal government to support the rollout of the app, and provide advice and public health expertise. The most prominent groups in this process were the Advisory Council and the FPT Working Group.

Terms of reference (ToRs) with clear mandates and objectives were developed for both the Advisory Council and the FPT Working Group. In addition to the objectives and mandates of the two groups, the ToRs outline meeting frequency, organizational structure, and terms of service.

Formal Advisory Council meetings started on August 5, 2020 and were held, on average, every two weeks, with over two-thirds of members attending each meeting. In addition to regularly scheduled meetings, Advisory Council members attended four technical briefings between August 20 and September 10, 2020, on topics ranging from technology and privacy, to security and public health measurement. Advisory Council meetings were originally managed by ISED (who continues to serve as co-secretariat for the Council) before being co-managed by Health Canada and ISED.

The Advisory Council was mandated to provide expert advice and guidance to ensure the app met the highest standards with respect to public health outcomes, privacy, and technology.Footnote 8 To carry out its mandate, the Advisory Council established a formal work plan to help guide its work, which focused on three pillars:

The Advisory Council committed to releasing a report for each of these three pillars to provide advice and suggestions for improvements to increase the effectiveness of the app.

In terms of the FPT working group, PTs were engaged through bilateral meetings beginning in early summer 2020 (June-July) to determine interest in adoption, to provide demonstrations of the app, and to share draft agreements. PTs were also engaged through FPT working group meetings that started on August 6, 2020. Meetings were held biweekly, on average. Attendance was strongest in the fall, with at least one representative from each of the member PTs in attendance, whereas more recent meetings have had an average of five PT representatives per meeting.

The FPT working group was established to provide public health advice and to ensure the app adds value to existing public health measures. Health Canada provided updates to the FPT Special Advisory Committee on COVID-19, including its Technical Advisory Committee and Public Health Working Group on Remote and Isolated Communities, and, at the same time, sought their advice, which has been useful given that public health is a key component of the app. Health Canada sought advice on various areas, such as:

In addition to these working groups, ISED currently serves as co-secretariat for the Advisory Council and helped recruit Council members and draft the ToRs for the Advisory Council governance mechanism. A few internal key informants noted that ISED’s experience in this area was particularly helpful in setting up the Council.

Partners and stakeholders were satisfied with the level of engagement.

Most external key informants were satisfied with their level of engagement with Health Canada, stating that staff were professional and responsive (that is, Health Canada took stakeholders’ advice into consideration, acted upon it where they could or where it was warranted, and responded quickly to requests for clarification). The relationship between Health Canada and partners and stakeholders has been collaborative and respectful, with ample communication. Most Advisory Council and FPT working group key informants were satisfied with the frequency of Advisory Council and FPT Working Group meetings.

External key informants noted that Health Canada and its partners (for example, CDS, ISED) all showed a high degree of collaboration and willingness to help. CDS’ technical briefings were felt to be particularly helpful for improving partners’ understanding of the app and its features.

The Advisory Council was actively engaged and transparent in its activities.

A review of the Advisory Council member bios showed that there is a good mix of expertise on the Council, covering three main areas: privacy, technology, and public health.

A review of meeting summaries and key informant interviews found that, in addition to information sharing by Health Canada and its partners through updates and presentations, there was active discussion on various topics at each of the Council’s meetings. In addition to providing advice through meetings, the Council produced three reports. The Council has been transparent in its activities by posting its ToRs and high-level meeting summaries online. The Advisory Council provided independent external advice and guidance on potential improvements and changes on a number of issues, including:

This advice was taken into account by Health Canada, who explored increasing the availability of the app in several more languages, and whose Minister wrote a letter to provincial Ministers of Health to garner support for the OTK issue.

Even though the FPT Working Group was mainly an information sharing body, its advice was used to help inform new features.

A review of meeting minutes and most internal and external key informants confirmed there is good information shared at the FPT Working Group meetings. While not an exhaustive list, some topics discussed included public health measures, PT onboarding experiences, new features, OTKs and notifications, as well as app metrics.

A review of FPT working group documents, including ToRs, agendas, and meeting minutes, indicate the group has good representation from Health Canada, CDS, PHAC, the Department of National Defence, and the PTs. Representatives from all onboarded PTs regularly attended meetings. Representatives from Alberta and BC (who did not adopt the app) also attended until the beginning of March 2021.

Some internal and external key informants noted that PTs provided feedback on potential new features, such as short message service (SMS) texts and QR codes, and PTs felt that Health Canada took their advice into account in terms of whether or not, or how, to move forward with these features.

Areas for improvement

Internal roles and responsibilities should be clarified.

There was a MoU in place between Health Canada and CDS outlining roles and responsibilities between the two partners. In particular, as the business owner, Health Canada was responsible for the overall implementation and distribution of the app, and CDS provided Health Canada with support by leading the design and implementation of the technical components of the app. This MoU expired at the end of December 2020 and an updated MoU was shared and discussed with CDS; however, it was never finalized.

As reported by internal key informants, there was a perceived lack of clear direction regarding roles and responsibilities between Health Canada and CDS, which led to confusion and challenges for moving things forward efficiently. In summer 2020, CDS led the project, when the focus was on getting the app up and running. By fall and winter 2020, Health Canada assumed its business ownership role, which appears to have led to some tensions and disagreement on the approach to service delivery and timelines.

PTs could have been engaged earlier in the process to determine their needs and capacity to implement the app.

The FPT working group was mandated to advise on the public health aspects of the COVID Alert app and, while most external key informants saw the value in the information that was shared at these working group meetings, most also agreed that it was primarily a one-way exchange of information from the federal government. A few mentioned that, in the beginning, the focus was on the technical aspects of the app and less on public health issues, but this changed following those initial meetings. Also, given the rapidly evolving situation and the need for national governments to establish exposure notification technology to help limit the transmission of the virus, time was limited to consult with PTs to determine their needs. Some internal and external key informants felt that the lack of early consultation to determine the PT needs hindered collaboration and buy-in moving forward. As a result, the app was introduced without a full understanding of the needs of individual PTs and each jurisdiction’s capacity to fulfill their needed role (that is, giving out OTKs to people who tested positive for COVID-19). This could have influenced provincial and territorial buy-in for the app, especially for jurisdictions that were more interested in having a contact tracing app, as well as for those jurisdictions who were overwhelmed by the pandemic.

Key findings: Effectiveness

There are some indications the app has had an impact; however, given the novelty of this tool, it was difficult to quantify due to a lack of data and the absence of pre-determined indicators of effectiveness, such as benchmarks, targets, and public health impacts. Furthermore, a number of challenges including privacy and trust concerns, technical issues, perceived lack of effectiveness, and issues with the distribution of one-time keys (OTKs) have limited the app’s effectiveness and, as a result, it may not have reached its full potential.

As previously mentioned, app users and the app itself must undergo multiple steps to achieve desired outcomes, including:

This section outlines how the app’s effectiveness was assessed against these steps.

There were numerous efforts to increase awareness and encourage app usage.

An extensive communications campaign was launched to increase awareness and understanding of the COVID Alert app to encourage Canadians to download and use the app. These efforts included the following:

Table 2: Communications campaigns: Awareness building efforts
Type Description
Advertising Numerous campaigns, (for example, for each onboarded PT, national, Indigenous, and multilingual campaigns) ran from July 31, 2020 to March 23, 2021, resulting in 863.9 million digital impressions, 919.8 million out-of-home impressions, and up to 6.3 million link clicks.
Partnerships Multiple private sector partnerships (for example, Google/Apple, Loblaws, Shoppers Drug Mart, Metro, Best Buy, Air Canada, Via Rail, Cineplex, Maple Leaf Sports and Entertainment Ltd., various financial institutions, First Nations Associations, and a number of ski associations) were established to help promote the app through social media, emails, audio and video ads, digital flyers, and posters. In addition to those listed above, internal key informants noted that telecommunications companies also helped promote awareness of the app through push notifications to let their customers know when the app was available in their jurisdiction. Apple and Google highlighted the app through their app stores.
Outreach

Health Canada conducted numerous outreach efforts to a wide variety of groups, some of which included:

  • Social media add-ons: the #COVIDAlert campaign was awarded Twitter Canada’s Best Campaign for Driving Positive Change;
  • Mail-outs with postcards and posters were sent to 7,700 schools and to public health units of onboarded PTs (except Québec, due to Bill M-30);
  • Promotional inserts: in December and January, Canada Revenue Agency mail-outs reached more than 900,000 households;
  • Articles, videos and radio clips for use in community papers, radio stations, and other media outlets across the country, reaching a total of 2,702,324 Canadians;
  • Resources, including web buttons, canned parent letters, and e-newsletter banners were sent via email to more than 170 school boards;
  • Posters, postcards, digital ads and videos were translated in up to 25 alternate languages each (10 of which were Indigenous languages), and distributed to groups disproportionately affected by the pandemic. Languages were selected based on the ‘Language most often spoken in the home’ (Census 2016) with a focus on those more recently immigrated;
  • Worked with 51 regional health authorities in the onboarded provinces to provide them with resources, such as social media plans, posters, shareable visual content, and key messages; and
  • Outreach to 57 municipalities across Canada. This involved several adding web content to their website and posting social media.

Source: Internal documents

Health Canada launched organic social media campaigns to promote downloading the app. Social media campaigns focused on the best practices that app users should follow. Other government department social media accounts amplified Health Canada’s social media, as well as creating their own content to encourage Canadians to download the COVID Alert app. Top officials, such as the Prime Minister and Canada’s Chief Public Health Officer encouraged Canadians to download the app on social media and also spoke about the app in press conferences. Additionally, Health Canada engaged with Canadians on social media to answer questions regarding the app.

Health Canada also helped raise awareness through social media influencers. Health Canada contacted more than 1,000 notable Canadians to help promote the app through their social media channels. These influencers, representing all PTs but one, came from a variety of backgrounds such as:

More than 95 individual influencers actively promoted the app on their social media platforms.

Health Canada and the Canadian Digital Service also provided support to the Service Canada Call Centre by providing question and answers on the app. There was also a short video produced for use in all 210 Service Canada locations.

All of these communications efforts likely contributed to raising awareness among Canadians, which steadily increased over the last year, as demonstrated below:

Furthermore, some internal and external key informants noted that awareness of the app was also promoted through the social media platforms of certain Chief Medical Officers of Health. For example, between February 10 and 19, 2021, Newfoundland’s Chief Medical Officer of Health tweeted about the app 13 times to her nearly 30,000 followers. However, as there is no data available regarding individual PT downloads of the app, it is not possible to determine the extent to which this promotion influenced downloads. However, during the week of February 1 to 7, 2021, 12.5% of positive cases in Newfoundland entered an OTK, as compared to 30.6% the week of February 8 to 14, 2021 and 32.75% the week of February 15 to 21, 2021. This appears to demonstrate that the promotion of the app on the Chief Medical Officer of Health’s twitter account has had an impact.

While Health Canada communications and marketing efforts have likely contributed to increased awareness and the 6 million downloads of the app between July and December 2020, downloads then plateaued due to other factors limiting use (See section below on barriers for further details).

Factors such as awareness, privacy protection, usability, and perceived value of the app helped encourage downloads and use of the app.

As described in the previous section, extensive promotion of the app helped reach Canadians of all ages and in various locations. A few internal and external key informants felt that, as additional PTs onboarded the app, awareness also grew in those PTs because more individuals outside of government were talking about it. In addition, downloads of the app spiked as new PTs adopted it.

The privacy preserving nature of the app was another factor that helped encourage downloads and use. Several internal and external key informants felt that privacy aspects helped generate public confidence that the app would maintain people’s privacy, thereby encouraging downloads and easing provincial concerns. As previously mentioned, the app received multiple public endorsements and support from privacy experts and critics. These were also seen as a contributing factor to downloads of the app.

Some internal and external key informants highlighted the usability of the app as another key factor in facilitating its use. The app has been described as simple and easy to access and download. For example, the app won a plain language award from the Center for Plain Language in the United States.Footnote 9

Furthermore, POR conducted in the fall of 2020 found most COVID Alert app users were satisfied with the app (73% Sept 2020; 71% Oct 2020), citing the following reasons: it provides valuable information, is easy to use, and provides peace of mind. Similarly, seven in ten Canadians (69%) felt the COVID Alert app was useful.

Key informants also identified key factors encouraging downloads and use of the app. A few internal and external key informants noted that one of the drivers for app use was the perceived value of the app, meaning that there was an action that could be taken if a notification was received (for example, self-isolate, get tested). POR results also reported the app’s perceived value and effectiveness as driving factors for its use. Canadians who indicated having downloaded the COVID Alert app identified the following reasons for doing so:

Similarly, in an earlier study of those who downloaded the app, most did so to:

The app had the potential to have a greater impact; however, due to various challenges, it may not have reached its full potential.

Interestingly, in an August 2020 survey, while nearly three-quarters of Canadians (73%) thought the COVID Alert mobile app was a good idea, they were much less convinced that it would work; just over one-quarter (27%) thought the COVID Alert app would work, whereas 39% disagreed and one-third (34%) did not know.

A number of studies indicate Canadians’ willingness to get tested or enter their code into the app should they receive a notification or a positive COVID test result, as demonstrated by the following:

However, while this POR indicates a willingness on the part of Canadians to download the app, get tested, and enter an OTK, as of July 28, 2021, 46,056 OTKs have been generated in participating PTs, of which 33,404 have been entered. Evidence shows that even though the number of OTKs entered is low, when offered an OTK, the majority of Canadians (approximately 72%) were using it to notify others (See Graph 1 below). The challenge is that not enough OTKs were distributed for a variety of reasons, such as:

Graph 1: Number of OTKs generated and entered – August 2020-July 2021
Graph 1 - Text description

Graph 1 is a line graph showing both the total number of one-time keys generated and the total number of one-time keys entered monthly from August 2020 through July 2021. The results shown in the graph are as follows:

One-time keys generated and entered
Month One-time keys generated One-time keys entered
September 2020 260 524
October 2020 2,928 2,480
November 2020 4,852 3,857
December 2020 6,841 5,456
January 2021 7,203 5,610
February 2021 3,206 2,387
March 2021 4,796 3,393
April 2021 9,813 6,494
May 2021 5,305 3,149
June 2021 935 504
July 2021 267 150

Between April and July 2021, more than 10,000 Canadians who tested positive for COVID-19 entered an OTK in their app, triggering more than 66,000 notifications issued to Canadians, which resulted in 640 people receiving a notification, testing positive and entering an OTK into the systemFootnote 10 (See Graph 2 below). This represented nearly seven notifications per OTK entered. Between April and July, the number of notifications per OTK entered ranged from a low of 5.4 in April 2021 to a high of 8.0 in July 2021Footnote 11, meaning that anywhere from 1 to 8 persons were reached with each notification. This ratio is higher than international comparisons, wherein data from February 2021 shows that OTKs claimed in England and Wales represented 4.41 notifications per OTK claimed, and Switzerland’s OTKs entered represented 4.22 notifications per OTK claimed.

Graph 2: Number of OTKs entered, notifications issued and OTKs entered as a result of notifications – April-July 2021

Source: COVID Alert Performance MetricsFootnote 12

Graph 2 - Text description

Graph 2 is a line graph showing the number of one-time keys entered, notifications issued, and number of one-time keys entered as a result of notifications from April to July 2021.

The results are as follows:

Number of OTKs entered, notifications issued and OTKs entered as a result of notifications – April-July 2021
Month One-time keys entered Notifications issued Notifications with subsequent positive test and one-time key entered
April 2021 6,495 35,204 416
May 2021 3,149 26,409 198
June 2021 504 3,910 24
July 2021 138 1,090 1

A McGill study of the app’s effectiveness used in-app metrics and app data to model averted cases and deaths in six Canadian provinces:

The study estimated that over the period of March to July 2021, the app had averted between 6,284 and 10,894 (or 0.44% and 0.76%) of total recorded infections and between 57 and 101 deaths (or 0.004% and 0.01% of COVID-induced fatalities) in the six provinces covered by the study. This is significantly lower than the estimates for the United Kingdom, with between 284,000 and 594,000 (or 4.94% and 10.34%) cases averted, and between 4,100 and 8,700 (or 0.07% and 0.15%) deaths. The study also found that in provinces where there was more widespread adoption of the app, such as Newfoundland and Labrador and Nova Scotia, the number of cases and deaths averted were higher than for Canada as a whole. For example, in Newfoundland and Labrador, the number of cases averted was more than 60% of total cases. The study concluded that this ratio shows that the app can be an effective tool when adopted by a sufficient proportion of the population.Footnote 13

This is only a part of the app’s overall effectiveness story, since app users may be taking other measures to mitigate exposure; however, the data on these other measures is not available or tends to be anecdotal. For example, an article published in Ottawa in September 2020 highlighted the city’s first instance where the COVID Alert app was used to alert a person of a close contact, which led to that person being tested, and thereby preventing the spread of the virus.Footnote 14 In addition, several internal and external key informants mentioned the story of a teacher in Toronto who was alerted through the app, leading that person to avoid the classroom, potentially ensuring that a number of children, staff, and families not being exposed to the virus. Another such story comes from New Brunswick, where a high school principal mailed out a notice to parents encouraging their kids to download the app after an exposure occurred at the school.

A few internal and external key informants felt that, if more positive stories like these had been highlighted more often, it may have resulted in an increase in the use of the app, especially since as previously seen, one of the biggest barriers to use was that some people did not think the app worked. Publicizing these stories may have led to the release of additional stories, which could have helped tell a better effectiveness story.

The majority of internal and external key informants felt that the app did not meet its full potential as a public health tool due to a number of challenges, which are outlined in the section below.

Several barriers, such as privacy and trust concerns, technical issues, lack of one-time keys issued, burden on app users, and perceived lack of effectiveness, affected downloads and use of the app.

Lack of OTKs issued

The Advisory Council’s Pillar 2 report concluded that the app could be an effective public health tool, as long as the OTKs are widely distributed and used.Footnote 15 As previously mentioned, when comparing to cases in onboarded PTs, only 46,056 (5.0%) OTKs were generated, and even fewer (33,404 or 3.7%) were entered into the COVID Alert app by Canadians, thereby triggering a notification to other app users of a possible exposure. Not all OTKs reached Canadians for the following reasons:

If more OTKs could have been distributed and entered, the app could potentially have had a much bigger impact on reducing the spread of COVID-19. The Advisory Council stated in its Pillar 3 report that the OTK distribution issue was the biggest missed opportunity to use the app more effectively.

Technical issues

The Council’s Pillar 2 report concluded that technological challenges also limited the app’s ability to reach its full potential.Footnote 17 Initial technical issues, such as challenges with the Google and Apple framework, and the ability to download the app on Apple iPhone 6s or lower, may have prevented some people from downloading the app. A couple of internal key informants noted that these early frustrations with the app may have deterred Canadians from attempting to download the app a second time. Another technical issue was that some Android phones (approximately 33%) would shut down the COVID Alert app as it was working in the background in order to save battery power. This issue was addressed for some devices, but there are still devices for which the problem remains. It is possible that these glitches created barriers to downloading and properly using the app. Furthermore, the lack of overall, and particularly in-app metrics at the beginning hindered reporting capability, as a number of phones did not have apps updated to versions that included in-app metrics, as well as the specifics of the GAEN framework and opt-in reporting.

Burden on app users

Furthermore, several internal key informants noted that:

Some internal and external key informants noted this was challenging because, in addition to dealing with a positive diagnosis, the person could be quite ill or have family members who are ill. They may have other concerns related to their diagnosis, which means that the burden of getting an OTK, writing it down, then entering it into the system may not have been their top priority.

Similarly, the Advisory Council concluded that the number of OTKs being entered into the app was low compared to the number of positive COVID-19 cases across Canada, and for the app to be an effective tool, it must be easy for users to receive and upload an OTK following a positive diagnosis.

While the lack of standardized OTK processes across PTs created challenges in Canada, other countries such as England and Wales each have their own national health system that could send out one set of instructions, thereby facilitating the ‘OTK’ distribution process. In addition, Colorado automated its ‘OTK’ process by texting a one-time verification code to anyone who tests positive for COVID-19. Since automating its process, Colorado saw the daily average of codes sent increase from 20 to 5,000.Footnote 18

Privacy and trust concerns

Although the app strictly adhered to privacy principles and was seen by many internal and external key informants as one of its key strengths, Canadians' privacy concerns and distrust of the government potentially collecting personal or location information through the app was also seen as one of its biggest challenges.

According to one POR study, privacy (35%) was the primary concern of Canadians prior to the launch of the COVID Alert app. Similarly, in another study, when asked why they did not intend to download a contact tracing app, privacy was more of a concern immediately after the launch of the COVID Alert app, but declined over time (23% in August 2020, 15% in October 2020). However, more recent POR (February 2021) specifically looking at the COVID Alert app had privacy concerns at the top of its reasons for not downloading it.

Trust in government is also clearly an issue. When asked if they believe the government does not collect their personal information and that it does not allow the GoC to determine their location, more than one-half (52%) of survey respondents did not believe the government. In another study, Canadians who indicated that they had not downloaded the app cited not trusting the privacy of the app or the government (34%), and not wanting government to access location data (29%) among the top three reasons for not downloading the app. A number of internal and external key informants also reported that there are certain segments of the population who do not trust the government and will not download the app.

Perceived lack of effectiveness, interest or need by Canadians

A perceived lack of effectiveness of the app was cited in POR as one of the biggest reported barriers to downloading the app:

Furthermore, a few internal and external key informants noted there was limited access to certain segments of the population, such as seniorsFootnote 19 and those for whom neither English nor French is their first language, which could have limited the perceived effectiveness of the app for these groups.

There were mixed views on the effectiveness of the app.

The goal of the COVID Alert app was to help contribute to a reduction in the spread of COVID-19; however, without any benchmarks or targets, it is difficult to adequately measure effectiveness. The majority of internal and external key informants had differing views regarding effectiveness of the app; some felt that one life saved made it a success, while others believed that much greater uptake of the app was required for it to be considered effective.

Canada is not the only country struggling to measure the effectiveness of its app. For example, an Oxford study, released in April 2020, was widely reported as stating that COVID-19 apps would need 60% of the country’s population to download the app if no other public health measures (for example, wearing masks, physical distancing, washing and sanitizing hands) were taken. The authors of that study attempted to correct this assumption by reporting that apps start to protect people at much lower levels of uptake (some as low as 14%). Based on these models, Canada’s app, with uptake levels of approximately 17% of the population (6,612,141 downloads as of July 28, 2021), should theoretically have had an impact.

A New York Times article (May 2021) questioned the efficacy of COVID apps and found that some public health officials and researchers in the United States had noted modest but important benefits of the app. The Chief Information Officer of University of California San Diego Health, who is responsible for California’s app, stated, “Whether it’s hundreds of lives saved or dozens or a handful, if we save lives, that’s a big deal.” Other health agencies predicted that apps would be able to notify users of virus exposures faster than manual contact tracing.Footnote 20

As mentioned, some key informants felt greater uptake was needed to increase effectiveness, but after initial spikes in downloads in July and August 2020, followed by another in October 2020, there has been minimal but continued growth in downloads since the end of December 2020 and beginning of January 2021. Based on publicly available download rates versus smartphone users, the COVID Alert app has uptake levels lower than comparable countries with similar systems and privacy regimesFootnote 21 (for example, Canada: 28%, England and Wales: 37.8%, Germany: 45.6%). The same is seen when comparing download rates versus population (for example, Canada: 17.3%, Italy: 20.2%, Switzerland: 27.1%, Germany: 36.9%, England and Wales: 37.4%, New Zealand: 59.7%).

However, the number of downloads is only part of the picture. According to in-app metrics data from July 28, 2021 of the 6,612,141 downloads of the app, there were 3.23 million active usersFootnote 22 of COVID Alert app, meaning that fewer devices had attempted to do exposure checks. This can be for a variety of reasons:

All of these limit the app’s effectiveness. A McGill study (July 2021) compared the number of active users across a number of countries and found that Canada had a lower percentage of active users versus downloads (48.8%) than both Switzerland and the United Kingdom (68.4% and 78.6% respectively).Footnote 23

A December 2020 article attempted to identify the reasons some countries were successful at controlling the spread of COVID-19 and highlighted the contact tracing methods (that is, isolating infected people and their contacts, as well as using personal data such as mobile-phone signals to track adherence to the quarantine) used in countries such as South Korea as exemplary methods of preventing transmission. While these methods work in South Korea, they would not work in countries like Canada, the United Kingdom, and Germany, among others, who have chosen a much more privacy-centric digital app to complement manual contact tracing.Footnote 24

This privacy-first approach helped encourage some Canadians to download the app, but at the same time, it affected Health Canada’s ability to tell a more complete effectiveness story because of the limited information it collected. Furthermore, several internal and external key informants and studies suggested privacy may have limited the app’s effectiveness from a public health perspective. For example, the lack of demographic and location information hindered the ability to implement more targeted public health strategies.

As previously mentioned, some changes have been made to improve the app since its launch. These included:

In addition to these changes, the Advisory Council, as well as most internal and external key informants, suggested the following changes as ways to further improve the effectiveness of the app:

Additional languages would have ensured more people could access the app and QR codes, and would have allowed for venue-based exposure notification to notify multiple users of potential exposure. The use of tokens or wearables was explored as a solution for individuals who did not have access to a smartphone. Finally, SMS text messages with an OTK could have been sent to those who had tested positive for COVID-19 so they could enter it in the app.Footnote 25

It appears that the QR code function could have presented an opportunity to increase use of the app. Other countries like Australia, Germany, New Zealand, and England and Wales have used QR codes. For example, in England and Wales, 963,032 National Health Service (NHS) QR code posters were created; and there have been 248,004,976 venue check-ins using NHS QR code posters. According to a May 2021 survey, it appears that a notable percentage of Canadians have expressed a willingness to use QR codes to enable venue-based notifications when available, and moreover, felt this function would have had the potential to help limit the spread of COVID-19:

Conclusions and lessons learned

Conclusions

Overall, there is evidence that the design and implementation of the COVID Alert app, as well as subsequent changes, adhered to all FPT privacy principles:

Health Canada ensured these principles were upheld by:

To help support the design and implementation of the app, a large number of governance mechanisms were established. Health Canada engaged with a wide variety of internal partners and external stakeholders to share information and seek advice. They worked together to quickly introduce a technology that was new to all involved and, while these governance mechanisms were generally working well, some challenges persisted. In particular, there was a need for greater clarity with respect to roles and responsibilities between Health Canada and the Canadian Digital Service (CDS).

There are some indications that the app had an impact on helping to limit the spread of the virus; however, given its novelty, it was difficult to quantify in the absence of pre-determined indicators of effectiveness, such as benchmarks, targets, and public health impacts. Furthermore, a number of challenges, including issues with the distribution of one-time keys (OTKs), technical issues, the burden on app users, privacy and trust concerns, and a perceived lack of effectiveness have limited the app’s effectiveness and, as a result, it may not have reached its full potential.

Lessons learned

The evaluation reviewed a number of lines of evidence, including document and POR reviews, and interviews with internal and external key informants. As a result, a few lessons learned emerged and should be considered to ensure the successful development and launch of a similar app by the Government of Canada in the future:

Appendices

Appendix 1 – Evaluation issues and questions

This evaluation covered the period from April 1, 2020 to July 31, 2021, and its scope included Health Canada’s activities related to the COVID Alert app. It did not assess the activities of partners, such as the Treasury Board Secretariat’s (TBS) Canadian Digital Services (CDS) and Innovation, Science and Economic Development Canada (ISED), nor the activities of provinces and territories (PTs).

The evaluation examined questions in relation to the adherence to privacy principles, effectiveness of the governance mechanisms, and effectiveness of the app, as shown in the table below.

Evaluation issues and questions

Table 2: Number of OTKs entered, notifications issued and OTKs entered as a result of notifications – April-July 2021
Core issues Evaluation questions Issue type
Issue #1:
Adherence to privacy principles
  • Have the following federal-provincial-territorial (FPT) privacy principles for exposure notification and similar apps been respected in both the design and implementation of the app?
  • consent and trust
  • legal authority
  • necessity and proportionality
  • purpose limitation
  • de-identification
  • time limitation
  • transparency
  • accountability
  • safeguards
Privacy
Issue #2:
Effectiveness of governance mechanisms
  • How have various governance mechanisms (for example, FPT Public Health Working Group, External Advisory Council and work between Office of the Privacy Commissioner and Health Canada) contributed to the app’s design, implementation and effectiveness?
  • Have key governance processes been established and followed?
  • Has advice and guidance been taken into consideration for decision-making?
 Performance (effectiveness, economy and efficiency)
Issue #3:
Effectiveness of app
  • How effective has the app been in contributing to efforts to reduce the spread of COVID-19?
  • How do actual take-up rates compare to any established benchmarks?
  • What are the factors affecting people’s use of the app?
  • What public health impacts can be demonstrated?
  • Is the app functioning correctly or as intended?
 Performance (effectiveness, economy and efficiency)

Appendix 2 – Data collection and analysis methods

Evaluators collected and analyzed data from multiple sources. Data collection started in March 2021 and ended in July 2021. Data were analyzed by triangulating information gathered from the different methods listed below. The use of multiple lines of evidence and triangulation were intended to increase the reliability and credibility of the evaluation findings and conclusions.

Program document and performance data review

The evaluation reviewed a series of documents to inform findings related to the privacy principles, governance, and effectiveness of the app. Approximately 160 documents were reviewed. In addition to these documents, we reviewed a number of POR studies that examined areas such as awareness, drivers of use, barriers to use, and views on effectiveness.

Key informant interviews

Key informant interviews were conducted to gather in-depth information related to the privacy principles, governance, and effectiveness of the app. Interviews were conducted based on a predetermined interview guide. In total, 31 interviews were conducted with 39 respondents. Respondents included:

Emerging themes from interviews were identified and quantified using NVIVO qualitative analysis software.

International and jurisdictional comparison

An international and jurisdictional comparison was conducted with eleven comparator countries and jurisdictions. Findings from a literature review (academic, peer-reviewed publications, and grey literature) helped inform the international comparison. Approximately 35 articles were reviewed. Country and jurisdiction websites and relevant effectiveness studies were also reviewed.

Data collected by these various methods was analyzed by triangulation to increase the reliability and credibility of the evaluation findings and conclusions. Still, most evaluations face constraints that may affect the validity and reliability of evaluation findings and conclusions.

The table below outlines the limitations encountered during the implementation of the selected methods for this evaluation and mitigation strategies put in place to ensure that the evaluation findings are sufficiently robust.

Table 3: Mitigation strategies for limitations of data analysis in this evaluation
Limitation Impact Mitigation strategy
Lack of benchmarks or targets Without pre-determined indicators of effectiveness such as benchmarks, targets and public health impacts, it was challenging to measure the app’s effectiveness. Triangulation with other lines of evidence provided as clear a picture as possible as to the impact of the app.
In-app metrics were only collected as of February 2021 Not having this additional data from the date of the app’s launch (July 2020) made it more challenging to fully assess the app’s effectiveness Triangulation of other lines of evidence was used to augment available information.

Appendix 3 – International and jurisdictional comparison

Table 4: International and jurisdictional comparison
Country or
jurisdiction		 Name of app and
Launch date		 Platform used Voluntary vs mandatory Privacy considerations and country context Uptake and notifications
Canada COVID Alert -
July 31, 2020		 Google/Apple Exposure Notification (GAEN) Voluntary Federated nation that conducted a privacy review to ensure the app adheres to the nine FPT privacy principles. As of July 28, 2021, the app has been onboarded by nine of 13 provinces and territories, and 6,612,141 users have downloaded the app and entered 33,404 OTKs into the system.
Alberta ABTraceTogether -
May 2020		 BlueTrace Voluntary Province with a privacy statement that describes the type of personal information it will collect. More than 300,000 people have downloaded the app, and according to the most recent numbers (March 2021), the app had been used to track 32 positive COVID-19 cases and 141 close contacts.
Australia COVIDSafe -
April 2020		 Amazon Web Services (AWS) Voluntary Federated nation with a Privacy Act, privacy principles and a privacy assessment. As of April 6, 2021, the COVIDSafe application was downloaded 7 million times.
Colorado CO Exposure Notifications -
October 25, 2020		 GAEN Voluntary U.S. State with a Privacy Policy that discusses what information can be collected, used and disclosed. As of January 5, 2021, approximately 1.4 million citizens opted into the app, and as of May 27, 2021, more than 28,000 people have used the technology to notify contacts of possible virus exposures.
England/
Wales		 NHS COVID-19 app -
September 2020		 GAEN Voluntary Unitary constitutional monarch with a privacy notice and data protection principles. As of August 3, 2021, the app has been downloaded 27,423,906 times, 963,032 NHS QR code posters have been created, and there have been 248,004,976 venue check-ins using NHS QR code posters.
Germany Corona-Warn-App -
June 2020		 GAEN Voluntary Federated nation with a privacy assessment. The Corona-Warn-App has over 27 million downloads.
Ireland COVID Tracker Ireland - July 7, 2020 GAEN Voluntary Constitutional Republic with a data protection information notice and a data protection impact assessment. As of 27 April 2021, over 2,510,000 people have downloaded the app. There is an active user base of 1.3 million.
Italy Immuni -
June 15, 2020		 GAEN Voluntary Federated nation whose app does not collect personal information, nor does it track the user’s location. As of August 11, 2021, the app has been downloaded 13,772,014 times. COVID positive users who uploaded their keys: 21,255
New Zealand NZ COVID Tracer -
May 20, 2020		 AWS Voluntary Federated nation with a privacy assessment. As of June 8 2021, there were 2,844,910 app registrations, 587,731 QR Codes generated, and 138,673,630 active devices.
Singapore TraceTogether -
March 20, 2020		 BlueTrace Partially voluntary Independent Republic with a privacy assessment. As of 11 May 2021, the adoption rate had exceeded 92%, with 4,923,054 individuals aged above six. The app helped to identify about 25,000 close contacts of COVID-19 cases, of which 160 eventually tested positive.
South Korea There is no contact tracing or exposure notification app SMS text messages Mandatory The legal framework for the Korean government’s contact tracing methods allows relevant authorities to collect information on confirmed cases or suspected cases. Text messages are sent to cellphones with information every time there is a positive case in a particular region.
Switzerland SwissCovid app -
June 25, 2020		 GAEN Voluntary Federated nation with a risk assessment and high data protection requirements As of August 2, 2021, the SwissCovid app was downloaded 3,292,200 times.

Appendix 4 – Governance mechanisms

Table 5: Governance mechanisms
Meetings Frequency Members
Advisory Council Biweekly External advisory body
FPT Working Group Biweekly ADM co-chair, working level PT reps
DM Briefs Daily HC DM and Testing Secretariat DM
QR Code Prime Daily HC and CDS
COVID Alert TSK Stand Up Twice a week HC and ISED (working level)
Interdepartmental COVID Alert Check-in Twice a week HC, PHAC, CDS, ISED (working level)
MINO Weekly MINO, HC ADM, DGs and Testing Secretariat DM
Executive Weekly Testing task force executive
CDS Generic Inbox Sync Weekly HC, Service Canada, and CDS
QR Code Discussion Weekly
Daily		 HC and CDS
Function leads
CDS Partnership Weekly HC and CDS
Four Corners Weekly MO, PMO, HC, CDS, TBS, and ISED
Testing Secretariat Executive Biweekly FPTs and private sector
ADM FPT Working Group Biweekly FPT ADMs in charge of testing, contact tracing, and data management
PMD and OPC Biweekly (weekly at first) PMD and OPC
Monthly PT Touchpoint Monthly PT Health Ministries – add in weekly PMD and CDS meetings

Endnotes

Footnotes

Footnote 1

Johansson, M., Quandelacy, T., Kada, S. et al. (2021-01-07). SARS-CoV-2 Transmission From People Without COVID-19 Symptoms. Journal of the American Medical Association, 4(1). Retrieved from https://jamanetwork.com/journals/jamanetworkopen/fullarticle/2774707

Return to footnote 1 referrer

Footnote 2

Office of the Privacy Commissioner of Canada. (2020-05-07). Supporting public health, building public trust: Privacy principles for contact tracing and similar apps: Joint Statement by Federal, Provincial and Territorial Privacy Commissioners. Retrieved from: https://www.priv.gc.ca/en/opc-news/speeches/2020/s-d_20200507/

Return to footnote 2 referrer

Footnote 3

The plan was tested through a virtual tabletop exercise organized by the Treasury Board Secretariat’s (TBS) Office of the Chief Information Officer’s cybersecurity division and including participants from TBS, CDS, Health Canada and the CCCS.

Return to footnote 3 referrer

Footnote 4

If privacy implications were minimal, rather than conducting a full assessment, PMD updated the main privacy assessment (for example, symptom onset, testing date).

Return to footnote 4 referrer

Footnote 5

QR codes stand for quick response codes. These are a type of barcode or pattern containing various forms of data that can be scanned. While this feature was never launched, PMD drafted a complete privacy analysis.

Return to footnote 5 referrer

Footnote 6

Health Canada began collecting in-app metrics in February 2021.

Return to footnote 6 referrer

Footnote 7

Under this proposal, people would be required to scan a QR code as a condition of entry to a venue.

Return to footnote 7 referrer

Footnote 8

Innovation, Science and Economic Development Canada. (2020-07-31). Terms of Reference of the Advisory Council to Federal, Provincial and Territorial Governments on the National COVID-19 Exposure Notification App. Retrieved from: https://www.ic.gc.ca/eic/site/icgc.nsf/eng/07688.html

Return to footnote 8 referrer

Footnote 9

Center for Plain Language. (2021). 2021 ClearMark Winners. Retrieved from: https://centerforplainlanguage.org/2021-clearmark-winners/

Return to footnote 9 referrer

Footnote 10

While the number of people who tested positive for COVID-19 and entered an OTK has been collected since October 2020 (for a total of more than 33,000), the number of notifications issued and the number of people receiving a notification, testing positive and entering an OTK into the system has only been collected since February and April respectively.

Return to footnote 10 referrer

Footnote 11

Of note, there were fewer than 11,000 cases in participating PTs in July 2021.

Return to footnote 11 referrer

Footnote 12

Government of Canada. (2021-07-28). COVID Alert Performance Metrics. Retrieved from https://www.canada.ca/en/public-health/services/diseases/coronavirus-disease-covid-19/covid-alert/performance-metrics.html

Return to footnote 12 referrer

Footnote 13

Ruths, D., Moodie, E., Shaw, M., Sun, S. (2021-08-24). COVID Alert Application Effectiveness. Retrieved from: http://derekruths.com/posts/covid_alert/

Return to footnote 13 referrer

Footnote 14

CTV News. (2020-09-02). Ottawa resident tests positive for COVID-19 after notification from COVID Alert App. Retrieved from: https://ottawa.ctvnews.ca/ottawa-resident-tests-positive-for-covid-19-after-notification-from-covid-alert-app-1.5089953

Return to footnote 14 referrer

Footnote 15

COVID-19 Exposure Notification App Advisory Council. (2021-08). Interim report: COVID Alert as a Government Service. Retrieved from: https://www.ic.gc.ca/eic/site/icgc.nsf/eng/07740.html

Return to footnote 15 referrer

Footnote 16

COVID-19 Exposure Notification App Advisory Council. (2021-08). COVID Alert as a Public Health Tool. Retrieved from: https://www.ic.gc.ca/eic/site/icgc.nsf/eng/07744.html

Return to footnote 16 referrer

Footnote 17

COVID-19 Exposure Notification App Advisory Council. (2021-08). Interim report: COVID Alert as a Government Service. Retrieved from: https://www.ic.gc.ca/eic/site/icgc.nsf/eng/07740.html

Return to footnote 17 referrer

Footnote 18

Colorado Department of Public Health and Environment. (2020-11-13). CO Exposure Notifications enabled by 1 million Coloradans. Retrieved from: https://covid19.colorado.gov/press-release/co-exposure-notifications-enabled-by-1-million-coloradans

Return to footnote 18 referrer

Footnote 19

However, it should be noted that some seniors may include long-term care individuals who would not have the app, nor the ability or need to enter an OTK.

Return to footnote 19 referrer

Footnote 20

Singer, Natasha. (2021-05-27). Why Apple and Google’s Virus Alert Apps Had Limited Success. Retrieved from: https://www.nytimes.com/2021/05/27/business/apple-google-virus-tracing-app.html

Return to footnote 20 referrer

Footnote 21

Of note, Canada could have had lower uptake levels partially due to the fact that there were four PTs who did not use the app.

Return to footnote 21 referrer

Footnote 22

The active user metric indicates the number of devices that have attempted to do an exposure check.

Return to footnote 22 referrer

Footnote 23

Ruths, D., Moodie, E., Shaw, M., Sun, S. (2021-08-24). COVID Alert Application Effectiveness. Retrieved from: http://derekruths.com/posts/covid_alert/

Return to footnote 23 referrer

Footnote 24

Lewis, Dyani. (2020-12-17). Why many countries failed at COVID contact-tracing — but some got it right. Nature. Retrieved from: https://www.nature.com/articles/d41586-020-03518-4

Return to footnote 24 referrer

Footnote 25

COVID-19 Exposure Notification App Advisory Council. (2021-08). Interim report: COVID Alert as a Government Service. Retrieved from: https ://www.ic.gc.ca/eic/site/icgc.nsf/eng/07740.html

Return to footnote 25 referrer

Page details

Date modified: