Evaluation of the Digital Citizen Initiative 2018-19 to 2020-21

Evaluation Services Directorate
June 9, 2023

On this page

• Executive summary
• 1. Introduction
• 2. Program profile
  • 2.1 Program activities, objectives and expected outcomes
  • 2.2 Program management and governance
  • 2.3 Program resources
• 3. Approach and methodology
  • 3.1 Scope, timeline and quality control
  • 3.2 Calibration
  • 3.3 Evaluation questions
  • 3.4 Data collection methods
  • 3.5. Evaluation limits and mitigation
• 4. Findings
  • 4.1 Relevance
  • 4.2 Effectiveness
  • 4.3 Efficiency
• 5. Conclusion
• 6. Recommendations, management response and action plan
• Annex A: evaluation framework
• Annex B: DCI performance indicators according to expected results
• Annex C: bibliography

List of tables

• Table 1: DCI logic model
• Table 2: DCI total planned spending ($) per fiscal year, 2019-20 and 2020-21
• Table 3: DCI total actual spending ($) per fiscal year, 2019-20 and 2020-21
• Table 4: evaluation Issues and Questions
• Table 5: limitations and mitigation strategies
• Table 6: number of funded applications per province and total funding approved, from 2019-20 to 2020-21
• Table 7: recurrent applicants to DCI and PCH by DCI approval status (#)
• Table 8: recurrent applicants to DCI and PCH by DCI approval status (%)
• Table 10: number of applications submitted by award or grant name to SSHRC-DCI Joint Initiative
• Table 11: adherence to DCI Service Standards – Acknowledgment of Receipt of Application
• Table 12: adherence to DCI Service Standards – Funding Decision
• Table 13: DCI Planned Spending ($), 2019-20 to 2020-21
• Table 14: DCI Actual Spending and Administrative Ratio of Efficiency ($), 2019-20 to 2020-21
• Table 15: recommendation 1 – action plan
• Table 16: recommendation 2 – action plan
• Table 17: recommendation 3 – action plan

List of acronyms

ADM

Assistant Deputy Minister

DCCP

Digital Citizenship Contribution Program

DCI

Digital Citizen Initiative

DCRP

Digital Citizen Research Program

DoCO

Diversity of Content Online

EBP

Employee Benefit Plan

ESD

Evaluation Services Directorate

FAA

Financial Administration Act

G&Cs

Grants and Contributions

GBA Plus

Gender-Based Analysis Plus

GoC

Government of Canada

NDMLW

National Digital Media Literacy Week

O&M

Operations and Maintenance

OL

Official Language

OLMC

Official Language Minority Community

PCH

Canadian Heritage

PCO

Privy Council Office

PIP

Performance Information Profile

PPF

Public Policy Forum

SSHRC

Social Sciences and Humanities Research Council

T&Cs

Terms and Conditions

Alternate format

Evaluation of the Digital Citizen Initiative 2018-19 to 2020-21 [PDF version - 552 KB]

Executive summary

The Digital Citizen Initiative (DCI) is a multi-component strategy that aims to support democracy and social cohesion in Canada by building citizen resilience against online disinformation and building partnerships to support a healthy information ecosystem. The DCI has two components:

1. the Digital Citizen Research Program (DCRP) which includes the Digital Citizenship Contribution Program (DCCP); and
2. the Diversity of Content Online (DoCO Initiative).

The DCI was established and initially funded through Budget 2019 with a focus on projects related to election misinformation but has evolved over the evaluation period to include a broader focus, including COVID-19 specific project funding.

This evaluation is the first for DCI and its scope includes activities undertaken between 2018-19 and 2020-21. This evaluation used a mixed-method approach including administrative data and file review, document review, literature review, and key informant interviews. The evaluation focused primarily on the DCRP.

Relevance

Overall, the evaluation finds that both components of the DCI is relevant to many key needs, to government priorities, and to the Department of Canadian Heritage (PCH)’s mandate. DCI fills an important need by funding research to help define and understand the problem of disinformation; connecting researchers and others in a community of practice; and supporting citizen-focused activities designed to help Canadians become more resilient to disinformation and associated harms.

Disinformation impacts Canadians’ health and safety, civic discourse and engagement, political beliefs, perceptions of democratic institutions, confidence in political systems and trust in the media. It may also amplify mistrust among communities, discrimination, stigma and marginalization, and social divisions. The DCI has been able to respond quickly to emerging situations such as the COVID-19 pandemic but has limited financial and human resources relative to the size and scale of the overall problem.

It is widely accepted that the issue of disinformation and its related harms are increasing significantly on a global scale, and Canadians are not immune to these harms. Equity groups including racialized populations and women tend to face disproportionate levels of harm because of disinformation; many DCI funded projects have been focused specifically on the needs and priorities of these groups. Participation of and research on issues impacting targeted groups (youth, Indigenous communities, Official Language Minority Communities (OLMCs), ethnocultural communities) were prioritized and considered in many DCI documents. DCI tracks results with partners and considers how it fulfils Official Language (OL) and Gender-Based Analysis Plus (GBA Plus) requirements.

The scale and scope of the problem being addressed by the DCI is expanding along with potential for associated harms. The evaluation concludes that current DCI human and financial resources limit the achievement of additional outcomes and that more impact could be achieved through larger projects and a higher number of projects. Of the 171 applications received in 2019-20 and 2020-21, only 31% were approved, with 69% rejected. The high rate of rejection is, in part, a result of the limited available resources compared to demand.

The DCI appears to complement efforts being made by other government departments. Document review and key informant interviews indicate no duplication and in fact there are synergies between DCI and other federal departments. There is good sharing of information about priorities and potential projects to fund among partners. However, as additional actors enter the space, the potential for overlap and duplication could increase.

Effectiveness

The DCI has made progress in its expected short- and medium-term outcomes. The program funded research and awareness activities, contributing to an increased awareness of online digital issues among funding recipients and participants in its funded activities. Research products being funded by DCI are being made publicly available. DoCO has achieved stated expected outcomes of creating of a multi-stakeholder working group and delivering the Guiding Principles on Diversity of Content Online.

DCI enhances connections between researchers and their access to information. DCI’s funding design and support for dissemination of project outputs help bring together new and established networks of researchers and organizations. Conferences and meetings organized by the DCI to help disseminate research findings and project outputs have been effective, despite having to move online due to the COVID-19 pandemic.

DCI-funded projects appear to have increased the digital media literacy of some Canadians. National Digital Media Literacy Week (NDMLW) events in 2020 and 2021 were funded by DCI through the DCCP. Other citizen-focused projects have also been successful in achieving expected results, including some related to COVID-19 pandemic misinformation.

The evaluation identified some issues with the DCI performance indicators. The evaluation largely relied on available data related to short- and medium-term performance indicators. There are opportunities to confirm and improve the expected outcomes and their associated indicators to better explain the program’s objectives and measure performance for reporting and for decision-making.

Efficiency

Overall, it appears that the DCI is delivered in an efficient manner. During 2019-20 and 2020-21, grants and contributions accounted for 87% of total DCI direct costs; with salaries and Operations and Maintenance (O&M) comprising 13% of total direct costs. Service standards were almost entirely achieved, and project recipients reported that the program is administratively efficient.

Partnerships facilitated the efficient delivery of programming. PCH leads the DCI Interdepartmental Consultative Body, the DCI Steering Committee and the Multi-Stakeholder Working Group on DoCO, which includes members from other governments, funded recipients, civil society organizations, private sector. DCI funds projects in partnership with the Social Sciences and Humanities Research Council (SSHRC) through the SSHRC-DCI Joint Initiative.

Recommendations

Based on the findings and conclusions, the evaluation has made the following three recommendations to ensure continued and strengthened relevance and performance of DCI moving forward. The Senior Assistant Deputy Minister (ADM), Cultural Affairs Sector, should:

Recommendation 1

Clarify existing roles and responsibilities for DCI within PCH and with the growing number of partners to promote strong collaboration and communication while preventing any duplication or overlap of efforts or resources.

Recommendation 2

Review, update and communicate funding priorities for DCI to ensure achievement of key results with the limited available funding.

Recommendation 3

Review and update the DCI Program logic model and performance measurement indicators to ensure information is available for strategic planning, decision-making and reporting.

1. Introduction

This report presents the findings, conclusions and recommendations resulting from the evaluation of the DCI. The evaluation was conducted to address evaluation requirements outlined in the Treasury Board Secretariat Policy on Results (2016) and the Financial Administration Act (FAA).

The evaluation covers the three-year period from April 1, 2018, to March 31, 2021, and examines targeted issues related to relevance, effectiveness, and efficiency.

2. Program profile

DCI was established and initially funded through Budget 2019 with a focus on projects related to election misinformation, it has evolved over the evaluation period to include a broader focus, including COVID-19 specific project funding. It is a multi-component strategy that aims to support democracy and social cohesion in Canada by building citizen resilience against online disinformation and building partnerships to support a healthy information ecosystem. DCI supports a community of Canadian researchers that promote a healthy information ecosystem, helps Canadians and the Government understand online disinformation and its impact on Canadian society, and in turn builds an evidence base to identify potential action and develop future policy-making.^{Footnote 1}

2.1 Program activities, objectives and expected outcomes

The DCI is composed of two primary activities: the DCRP and the DoCO Initiative. In summary:

1. The DCRP provides digital media literacy programming to Canadians, and conducts research that helps understand disinformation, with a focus on supporting the development of a policy and research agenda to guide Canadian action. The DCRP is implemented through two transfer payment activities: the DCCP and a joint initiative with the SSHRC, the Joint Initiative for Digital Citizen Research. As part of the research program, the DCCP supports the priorities of the DCI by providing time-limited financial assistance for research and citizen-focused activities.^{Footnote 2}
2. DoCO is a PCH-led initiative mandated to develop Guiding Principles on Diversity of Content Online through international multi-stakeholder engagement with government partners, industry, civil society organizations. The Guiding Principles are a strategic framework to guide actions and measures that foster greater exposure to diverse cultural content, information and news online.

The program’s logic model shown in Table 1 outlines the DCI’s short, medium, and long-term expected outcomes.

Long-Term Outcome	Canadians’ awareness of digital media literacy is increased through events and products. Researchers are better connected and have access to information on how online disinformation affects Canadians.
Medium-Term Outcomes	Participants are satisfied with citizen-focused activities. Research on online disinformation is made publicly accessible to Canadians and is relevant to government research needs. Participants are satisfied with National Digital Media Literacy Week activities. Stakeholders are engaged in developing the Guiding Principles on DoCO and adopt them.
Short-Term Outcome	Canadians and the Government are better equipped to counter the effects of online disinformation.

Source: PIP – 02 Cultural Marketplace Framework – 2020-21, p. 23-24, 28-30; DCCP T&Cs, p. 2

2.2 Program management and governance

The DCRP and the DoCO Initiative are organizationally under two different branches within PCH’s Cultural Affairs Sector. The DCRP is administered by a core team within the Digital Citizen Initiative Directorate which falls under the Broadcasting, Copyright and Creative Marketplace Branch. The DoCO Initiative is under the responsibility of the International Trade Directorate within the International Trade Branch.^{Footnote 3}

The DCI uses two governance committees – the Consultative Body and the Steering Committee. The former is made up of representatives from other Government of Canada departments. The latter consists of representatives from academia and civil society.

DoCO governance is centered on a Multi-Stakeholder Working Group established by PCH and comprised of representatives from governments, the private sector, civil society, and a para-public institute. The Working Group’s mandate is to develop Guiding Principles that help foster greater exposure to diverse cultural content, information, and news online.

2.3 Program resources

As shown in Table 2, the planned total program spending was $5.7 million and in Table 3 the actual spending was $9.7 million from 2019-20 to 2020-21. The increased actual spending is mostly due to additional funding provided to DCI to administer COVID-19 specific calls for proposals over this period. It should be noted that no funding was received in 2018-19.

Fiscal Year	Vote 1: Salary & EBPTable 2 note 1	Vote 1: O&MTable 2 note 2	Vote 5: G&CsTable 2 note 3	Total
2019-20	121,046	4,000	1,886,715Table 2 note 4	2,011,761
2020-21	823,914	129,250	2,682,263	3,635,427
Total	944,960	133,250	4,568,978	5,647,188

Source: Financial Planning and Resource Management, PCH

Table 2 notes

Table 2 note 1

Employee Benefit Plan

Return to table 2 note 1 referrer

Table 2 note 2

Operation and Maintenance

Return to table 2 note 2 referrer

Table 2 note 3

Grants and Contributions

Return to table 2 note 3 referrer

Table 2 note 4

Planned amount in 2019-20 for the commitment item 4582 (Digital Citizen (c)) under fund 2245

Return to table 2 note 4 referrer

Fiscal Year	Vote 1: Salary & EBPTable 3 note 1	Vote 1: O&MTable 3 note 2	Vote 5: G&CsTable 3 note 3	Total
2019-20	608,067	53,026Table 3 note 4	1,881,335	2,542,428
2020-21	550,085	25,608	6,612,403	7,188,096
Total	1,158,152	78,634	8,493,738	9,730,524

Source: Financial Planning and Resource Management, PCH

Table 3 notes

Table 3 note 1

Employee Benefit Plan

Return to table 3 note 1 referrer

Table 3 note 2

Operation and Maintenance

Return to table 3 note 2 referrer

Table 3 note 3

Grants and Contributions

Return to table 3 note 3 referrer

Table 3 note 4

O&M includes 28,989$ in “Goods and Services” and 24,037$ in “Travel, Conference & Hospitality”

Return to table 3 note 4 referrer

According to PCH’s Financial Planning and Resource Management, the actual full-time equivalents for the DCI was 10.5 in 2019-20 and 18.4 in 2020-21.

3. Approach and methodology

The evaluation was undertaken by the Evaluation Services Directorate (ESD) with support from an evaluation consulting firm. It was conducted as prescribed in the Canadian Heritage (PCH) Departmental Evaluation Plan 2021-22 to 2025-26. This section outlines the evaluation approach and methodology including scope, timelines, calibration, evaluation questions, data collection methods, limitations and mitigation strategies.

3.1 Scope, timeline and quality control

The scope of the evaluation includes DCI activities undertaken between 2018-19 and 2020-21. This evaluation was not an FAA requirement and was undertaken at the request of the Broadcasting, Copyright and Creative Marketplace Branch to document the continued relevance and effectiveness of the grants and contributions funding within DCI. The evaluation therefore focused primarily on the DCRP component of DCI.

Initial scoping meetings were held with the program officials, including senior management, to determine specific information needs and to refine the evaluation scope. Key information needs related to notably, resources, pivoting rapidly to meet governmental priorities, oversubscription and measuring results.

The evaluation was conducted in a manner consistent with the Treasury Board Secretariat Policy on Results (2016) and the Directive on Results (2016). The following quality assurance measures were in place for the evaluation:

• The evaluation was led by professional, experienced evaluators internal to PCH with support from a consultant firm.
• Multiple sources of primary and secondary data were used to ensure reliable findings.
• Findings were validated through appropriate analysis and triangulation.
• Preliminary findings were reviewed with program representatives and evaluation management to ensure clarity of communication and analysis.
• A systematic approach to the synthesis of evidence on each evaluation question was used.
• A phased approach was taken for the lines of evidence, to permit some early data results in Phase 1 (prior to March 31, 2022) with confirmed findings in Phase 2 (after March 31, 2022).

3.2 Calibration

Calibration is the process of adjusting an item, such as an evaluation approach or tool, to the sensitivity required to suit a particular function. It is based on several different factors to produce quality evaluations cost effectively and within timelines. A phased approach was taken for the lines of evidence, to permit some early data results in Phase 1 (prior to March 31, 2022) with confirmed findings in Phase 2 (after March 31, 2022).

This evaluation was also calibrated to focus on key questions by: reducing the number of evaluation questions; focusing on existing data sources where possible; undertaking targeted data collection; and, streamlining the report.

3.3 Evaluation questions

The following questions in Table 4 were used to guide the evaluation. The Evaluation Matrix included as Annex A provides additional detail in the form of data sources and indicators linked to each of the following questions.

3.4 Data collection methods

This evaluation used a mixed-method approach including: administrative data and file review, document review, literature review, and key informant interviews. Each method is briefly described in the subsections below.

The evaluation uses the following legend throughout the report to indicate the proportion of individuals interviewed or surveyed that responded in the same manner:

• Few: findings reflect less than 25% of the observations.
• Some/several: findings reflect at least 25% but less than 50% of the observations.
• Half: findings reflect 50% of the observations.
• Majority: findings reflect more than 50% and less than 75% of the observations.
• Most: finding reflect 75% but less than 90% of the observations.
• All/almost all: findings reflect 90% or more of the observations.

3.4.1 Administrative data and file review

The evaluation team reviewed and analyzed data from PCH’s internal system, Grants and Contributions Information Management System, from the program, as well as other administrative data including financial data received by the Financial Management Branch. All information was validated with Initiative representatives.

3.4.2 Document review

The evaluation team reviewed key DCI and Government of Canada documents. The review included, but was not limited to: submissions for funding, departmental and program policies, directives, guidelines, application forms, Terms & Conditions, program audits, meeting minutes, communications and outreach products, Speeches from the Throne, Federal Budgets, Reports to Parliament, and Statistics Canada reports.

3.4.3 Literature review

A targeted review was undertaken of research from recently published literature, reports, websites, public opinion research and analyses related to disinformation and online harms.

3.4.4 Key informant interviews

A total of 14 key informant interviews were conducted with 5 funding recipients, 5 internal PCH managers and analysts, and 4 members of the Consultative Body, Steering Committee, or other federal government partners.

3.5. Evaluation limits and mitigation

Table 5 outlines and key constraints and limitations of the evaluation process.

4. Findings

4.1 Relevance

4.1.1 Relevance: ongoing need for the program

Evaluation question: To what extent does the DCI address the continued, emerging and changing needs of Canadians on disinformation and online harm?

Key findings:

• DCI addresses real and continuing needs of Canadians with respect to online disinformation and related harms. The demand for DCI funding appears to outweigh available resources. The scope and scale of the issue are rapidly expanding along with the potential for harm.
• The DCI is a flexible program that has shown the ability to respond to emerging issues of online disinformation within a constantly shifting landscape. DCI pivoted in response to disinformation related to the COVID-19 pandemic. The short response time with projects launching within months of the outset of the pandemic was highlighted as evidence of the flexibility and responsiveness of the DCI.

DCI is a flexible program that can respond quickly to emerging issues of online disinformation

Research literature shows that disinformation impacts Canadians’ health and safety, civic discourse and engagement, political beliefs, perceptions of democratic institutions, confidence in political systems and trust in the media. Disinformation may also amplify mistrust among communities, discrimination, stigma and marginalization, as well as exacerbate social divisions.^{Footnote 4},^{Footnote 5} Some groups, such as people with low digital literacy skills, may be more susceptible to misinformation campaigns^{Footnote 6}; others, such as racialized populations^{Footnote 7} and women^{Footnote 8}, may be disproportionately harmed because of the spread of disinformation.

In response to the potential for harm, the Government of Canada provided $7.5 million over 2018-19 and 2019-2020 to support digital, news, and civic literacy programming and tools by funding citizen-focused activities, using existing PCH funding programs. This funding went to 20 projects that reached 12 million Canadians. In 2019-20, a further $19.4 million over four years was budgeted to create the DCI and the DCCP to fund applied research activities in line with the government’s need to better understand the origins, impacts and potential responses to online disinformation in Canada. During the two years of funding covered by this evaluation, 2019-20 and 2020-21, 53 additional projects totalling $9.9 million were funded, as was the SSHRC-PCH Joint Initiative, the Public Policy Forum (PPF) and MediaSmarts’ Digital Media Literacy Week (DMLW).

Key informants, including funding recipients, members of the consultative body and steering committee, and PCH management and staff indicated that the DCI addresses real and continuing needs of Canadians with respect to online disinformation and related harms. They reported that the scope and scale of the issue are rapidly expanding along with the potential for harm, and the DCI is a flexible program that has shown the ability to respond to emerging issues within a constantly shifting landscape.

The demand for DCI funding appears to outweigh available resources

The scale and scope of the problem addressed by the DCI are expanding along with potential for associated harms. Key informants indicated that their current human and financial resources limit the achievement of additional outcomes and that more impact could be achieved through larger projects and a higher number of projects. Of the 171 applications received in 2019-20 and 2020-21, 53 (or 31%) were approved, and 118 (69%) were rejected. Administrative data did not suggest a rationale for why applications were denied, with most rejection decisions categorized simply as ‘unsuccessful’. However, internal key informants indicated that DCI has limited resources relative to the scale of the problem and the demand from interested proponents.

DCI pivoted in response to disinformation related to the COVID-19 pandemic

Canadians have been subject to vast amounts of misinformation and disinformation related to the COVID-19 pandemic. In the first few months of the pandemic, 90% of Canadians used online sources to find information about COVID-19. Of these Canadians, 96% saw COVID-19 information that they suspected was misleading, false or inaccurate; and nearly two in five Canadians (40%) reported believing the information they saw, then later realizing that it was not accurate.^{Footnote 9}

The DCI allocated over $4 million in funding for activities combatting COVID-19 disinformation, including $2.7 million to 8 approved applications in 2019-20 and $1.7 million to an additional 26 approved applications in 2020-21. The speed of response, with projects launching within months of the outset of the pandemic, was highlighted by most key informants as evidence of the flexibility and responsiveness of the DCI.

As shown in Table 6, the number of projects and amounts approved per capita varied between provinces and territories during the evaluation period. For example, Ontario-based applicants had 1.63 projects funded for every million residents of Ontario with approved amounts equal to $0.44 per capita whereas BC-based applicants had 0.77 projects approved for every million population with approved amounts equal to $0.05 per capita, or just 11% of the per capita amount approved to Ontario-based applicants.

Primary province of funded applicants	# of funded applications	Total application amount approved ($)	Average amounts of funding per project ($)	Population (Q1, 2021)Footnote 10	Projects per million population	Funding approved per capita ($)
Ontario	24	6,426,529	267,772	14,740,102	1.63	0.44
Quebec	11	1,448,710	131,701	8,579,010	1.28	0.17
Atlantic Region	4	614,125	153,531	2,447,663	1.63	0.25
Northwest Territories, Nunavut and Yukon	2	248,087	124,044	127,161	15.7	1.95
Alberta	5	793,381	158,676	4,429,077	1.13	0.18
British Columbia	4	271,760	67,940	5,157,293	0.78	0.05
Manitoba	2	80,000	40,000	1,384,025	1.45	0.06
Saskatchewan	1	39,908	39,908	1,179,119	0.85	0.03
Canada	53	9,922,500	187,217	38,043,450	1.39	0.26

Source: GCIMS administrative data, Statistics Canada data

The data in Table 7 and Table 8 shows that most approved applications (46/53 or 87%) to DCI were made by recurrent applicants to DCI (n=26) or other PCH programs (n=20). Of the 53 approved applications, just 7 (13%) were made by applicants who were not recurrent to any PCH program. Stated differently, among the 171 applications, there were 142 made by applicants who were recurrent to PCH and/or DCI, and 37% of those applications to DCI were approved; among the remaining 47 applications made by applicants that were not recurrent, 15% were approved.

Recurrence Status	Applications Approved	Applications Not Approved	Total
Recurrent to DCI	26	57	83
Recurrent to PCH but not to DCI	20	21	41
Not recurrent to PCH	7	40	47
Total	53	118	171

Source: GCIMS administrative data

Recurrence Status	Applications Approved	Applications Not Approved	Total
Recurrent to DCI	49	48	49
Recurrent to PCH but not to DCI	38	18	24
Not recurrent to PCH	13	34	27
Total	100	100	100

Source: GCIMS administrative data

4.1.2 Relevance: harmonization with government priorities, roles, and responsibilities

Evaluation question: To what extent does the DCI support government priorities, including those related to equity-deserving groups?

Key findings:

• DCI supports government priorities, roles and responsibilities and considers equity-deserving groups in its design and implementation.
• DCI funds citizen-focused activities and research to support democracy and social cohesion in Canada by building resilience against online discrimination. It is also designed to support a variety of different activities throughout PCH to foster diversity and inclusion, as well as Canadian identity and values. Equity-deserving groups often face disproportionate harm from disinformation, and many DCI funded projects have been focused specifically on the needs and priorities of these groups.

The DCI supports government priorities, roles and responsibilities

The DCI contributes to PCH’s core responsibilities related to creativity, arts, and culture. The DCI funds citizen-focused activities and research to support democracy and social cohesion in Canada by building resilience against online discrimination. It is also designed to support a variety of different activities throughout PCH to foster diversity and inclusion, as well as Canadian identity and values. For example, DCI supports activities through Youth Take Charge and the Canada History Fund.

Equity-deserving groups often face disproportionate harms due to disinformation

Many DCI funded projects have been focused on the needs and priorities of equity-deserving groups. For example, projects have been directed at countering anti-Asian racism during the COVID-19 pandemic as well as understanding the problem of misogynistic hate speech being directed at women online.

An initial GBA Plus analysis done for the DCI found that there was a gap in current information about how disinformation affects different groups in Canada. To address that gap, the DCI has used its research component to gain a greater understanding of the issues and solutions. As well, participation of and research on issues impacting targeted groups such as youth, Indigenous communities, OLMCs, ethnocultural communities, were prioritized and considered in many DCI documents including in its Terms & Conditions and project proposal assessment forms.

DCI tracks results with partners and considers how it fulfills OL and GBA Plus requirements. In 2019-20, 29%, of DCCP/DCI funded projects directly and specifically addressed ethnic, racial or religious communities, minority communities, in both official languages and in Indigenous communities. In 2020-21, $4.3 million was dedicated specifically to counter COVID-19 disinformation, and the racism and stigmatization that are often the result. Finally, in the context of the SSHRC-PCH Joint Initiative, an effort was made in 2021-22 to continue funding research on vulnerable population and online disinformation.

Funded recipients were asked if they knew of any DCI operating or funding barriers specific to equity-deserving groups. None of the funded recipients indicated that they knew of any such barriers: two funded recipients indicated that their organizations served equity-deserving groups and had no barriers in the program.

4.1.3 Relevance: extent of complementarity or duplication

Evaluation question: To what extent does the DCI duplicate or complement other programs?

Key findings:

• DCI appears to complement efforts being made by research and non-governmental organizations as well as other government departments.
• There is evidence of synergies between DCI and other federal departments who share information about priorities and potential projects to fund.
• As additional actors enter the space, the potential for overlap and duplication could increase.

DCI is complementary and not duplicative of other programs and activities

According to key informants and document review, DCI appears complementary despite a number of other existing initiatives. Many key informants indicated that DCI was the first funding source of its kind. Many were able to mention other government departments and agencies as well as foundations and private sector entities that also provided funding in various ways to help address disinformation-related harms. For example, it was noted that a Global Affairs Canada funding program focused on the international aspects of online harms, and that the Public Health Agency of Canada had funding available for information dissemination and that could include correcting health-based disinformation. Also, an internal key informant mentioned that the PCO funds disinformation-related behavioural science projects through its Impact and Innovation unit; they also reported working closely with this unit to ensure there is no duplication of funded projects and look for opportunities for proponents that have applied to one program but are better suited to another program.

Some efforts have been made by digital industry companies to counter disinformation. These efforts do not appear to be involved in building research communities, digital literacy or resiliency among social media users and so do not overlap with the purpose of the DCI.

Evidence indicates that unique features to DCI include: national scope, funding levels and funding priorities. The primary way DCI maintains complementarity is through good communication and collaboration mechanisms.

4.2 Effectiveness

4.2.1 Effectiveness: achievement of short-term expected outcomes

Evaluation question: To what extent did the DCI achieve its short-term expected outcomes?
- Researchers are better connected and have access to information on how online disinformation affects Canadians.
- Canadians’ awareness of digital media literacy is increased through events and products.

Key findings:

• The DCI has made progress in its short-term expected outcomes.
• DCI funding design supports the development and maintenance of stakeholder and research communities and the dissemination of results. The program funded research and awareness activities over the evaluation period, and these have contributed to an increased awareness of online digital issues among funding recipients and participants in its funded activities. Citizen-focused projects supported by the DCI have found that they are able to impact education and behaviour change by engaging with Canadians on the topic of disinformation and online harm.
• There are some opportunities to improve the expected outcomes and associated indicators.

DCI enhances connections between researchers and their access to information

According to key informants, the DCI’s funding design and support for dissemination of project outputs help bring together new and established networks of researchers and organizations. All five of the project recipients interviewed indicated that the process of carrying out their funded projects relied on stakeholders and networks of researchers or organizations coming together to work on shared issues. Of note, all funding recipients indicated that conferences and meetings organized by the DCI to help disseminate research findings and project outputs have been effective, despite having to move online due to the COVID-19 pandemic. The program also supports access to research and project results by obtaining research abstracts from each funded project and making them available to DCI stakeholders on an internal SharePoint site.

DCI funding also contributes to the holding of different types of events, notably:

• During 2019-20 and 2020-21, 312 research and data collection products produced by DCI funding recipients related to online disinformation were accessible to researchers via a Government of Canada (GoC) website.
• During 2019-20 and 2020-21, the PPF Institute held 63 networking and/or information sharing events; data indicates there were 22 contributors and/or presenters and 72 attendees at the 2019-20 annual PPF Institute Conference for Online Disinformation. In 2021-22 there were 74 attendees, but the number of presenters is unknown.
• The Multi-stakeholder Group on DoCO which is comprised of approximately 14 different stakeholder organizations from the federal government and the private sector held 6 meetings in 2020 and 2021.
• Two NDMLW events funded by DCCP were held.

As well, the SSHRC-PCH Joint Initiative funded applications during the evaluation period. The number of applications submitted to SSHRC-DCI per fiscal year and per type of award or grant during the evaluation period appears in Table 9. Of the 84 applications submitted, administrative data shows that 38 applications were funded a total of $1.5 million. Most (n=32) funded recipients were individuals/sole proprietorships and six recipients were in academia. Most recipients were from Ontario, Québec, and British Columbia. Most (93%) funds went to individual/sole proprietorship recipients. While accounting for just 24% of agreements, recipients from Quebec accounted for 59% of total agreement funds.

Fiscal Year	ConnectionFootnote 11	TalentFootnote 12	InsightFootnote 13	Total
2019-20	7	10	31	48
2020-21	2	18	16	36
Total	9	28	47	84

Source: Program data

DCI-funded projects appear to have increased the digital medial literacy of some Canadians

NDMLW events in 2020 and 2021 were funded by DCI through the DCCP. The inception of the event does predate DCI’s involvement but was previously carried out on a much smaller scale because of a lack of resources, and, therefore, reached far fewer Canadians than it did with events supported by the DCI. According to administrative data, in 2020-21, approximately 59% of participants indicated having increased their awareness because of the event.^{Footnote 14}

Other citizen-focused projects have also had an impact. For example, according to a funding recipient, an evaluation carried out on the impact of a web-app related to COVID-19 misinformation, developed with the support of DCI, found that users of the app had both learned and applied those learnings to behaviour changes and that those changes were maintained over time.

Some issues exist with the short-term outcome performance indicators and data

The analysis has relied on available data from this outcome’s broad, short-term indicators. Issues were identified by assessing the quality and relevance of available performance data, and in considering the relationship between expected outcomes, indicators, and the program logic model. In particular:

• Citizen-focused activities, including NDMLW have the current short-term measures of ‘an increase in awareness of digital media literacy’ and ‘satisfaction with citizen-focused activities is a medium-term expected result’. In general, satisfaction would lead to a participant with increased awareness, rather than the reverse.
• The number of meetings and the number of stakeholder organizations involved in the multi-stakeholder working group on DoCO are included as two performance measures for the expected short-term outcome “Researchers are better connected and have access to information on how online disinformation affects Canadians” but that is not consistent with the international focus of the guiding principles and the work undertaken by DoCO.
• The “number of grant applications to the SSHRC-PCH Joint Initiative” is a short-term performance measure, but the number of funded applications is not. It is not clear how applications on their own, and not specifically funded applications, is reflective of “researchers being better connected or having access to information on how online disinformation affects Canadians”.

4.2.2 Effectiveness: achievement of medium-term expected outcomes

Evaluation question: To what extent did the DCI achieve its medium-term expected outcomes?
- Participants are satisfied with citizen-focused activities and NDMLW activities.
- Research on online disinformation is published, made publicly accessible to Canadian and is relevant to policy-making.
- Various stakeholders are engaged towards adoption of guiding principles on DoCO.

Key findings:

• There is some evidence that that DCI has partially achieved its the medium-term expected outcomes.
• Participants appear to be satisfied with citizen-focused activities and that research products being funded by DCI are being made publicly available and, in many cases, have been of interest to policy-makers in Canada and internationally.
• DoCO has successfully fulfilled its expected outcomes by establishing an international working group and developing the Guiding Principles on DoCO.
• As with the short-term outcomes, there are some challenges with the medium-term outcomes and indicators.

Through the DCCP, DCI has provided funding to support MediaSmarts’ NDMLW since 2020 to raise awareness of digital and media literacy across Canada and to provide funding to research- and citizen-focused activities that support efforts to counter online disinformation. Based on available performance data, 100% of participants in 2020-21 were satisfied with citizen-focused activities, and 89% of participants rated their satisfaction of the 2020 NDMLW event a “4” or “5” on scale of 1 to 5.

According to key informants, including DCI management and funding recipients, several federal government departments have shown an interest in research funded by the DCI and funded recipients have indicated that they have delivered presentations to federal departments and international stakeholders about their findings. In terms of determining the influence of the research on policy-making, there was no clear data available.

DCI performance indicator data supports that the program supported the publication and accessibility of research on online disinformation. In 2019-20 and 2020-21:

• 5 new research products on the scope and impact of online disinformation in the Canadian context were produced by recipients with DCI funding and were made publicly available.
• 17 new research projects produced by funded applicants were published, available and relevant to governmental policy-making.
• 8 awards under the SSHRC-PCH Joint initiative were granted in 2019-20, increasing by 250% in 2020-21 to 20 awards. Partial year’s data for 2021-22 shows 9 awards by October 2021.
• The Multi-stakeholder Group on DoCO held 6 meetings in 2020-21 and 2021-22.

DoCO has developed Guiding Principles on Diversity of Content Online

The Multi-stakeholder Group on DoCO was mandated to develop guiding principles on diversity of content online to help build citizen resilience to disinformation and contribute to greater social cohesion. The Multi-stakeholder Group is comprised of countries such as Australia, Finland, France, and Germany, and representatives from online platforms, and civil society. The evaluation found that the principles were developed. According to key informants, the group has fulfilled its initial mandate. However, it is not clear the extent to which the medium-term outcome performance outcome “various stakeholders are engaged towards adoption of guiding principles on diversity of content online” has been achieved otherwise.

Some issues exist with the medium-term outcome performance indicators and data

There are some challenges with the medium-term outcomes and indicators when considering the relationship between expected outcomes, indicators, and the program logic model. For example:

• The medium-term outcome indicator of ‘satisfaction with citizen-focused activities’ should precede an awareness indicator in order to be meaningful. The short-term outcome indicator ‘an increase in awareness of digital media literacy’ would be better placed as a medium-term outcome.
• It is not clear how the “percent change in the number of grant applications to the SSHRC-PCH Joint Initiative” reflects progress made against the medium-term expected outcome, “research on online disinformation is made publicly accessible to Canadians and is relevant to government research needs”, as opposed to a metric based on the output or uptake of research products supported by the Joint Initiative.
• It appears that two medium-term expected results have the same data to assess performance. The two results are “participants are satisfied with NDMLW activities” and “participants are satisfied with citizen-focused activities”. Departmental reports provide the same value for both performance indicators and so it appears that the value for % satisfied with NDMLW activities is being cross applied to the second medium-term expected outcome.

4.3 Efficiency

Evaluation questions: To what extent is DCI delivered in an efficient manner?

Key finding:

• Overall, the DCI appears to be delivered efficiently.
• Service standards have been achieved nearly 100% of the time and project recipients appear satisfied with program administration. During 2019-20 and 2020-21, grants and contributions accounted for 87% of total DCI direct costs versus 13% for salaries, EBP and O&M.

Service standards have been achieved

In 2019-20 and 2020-21, the program met service standards for the acknowledgement of reception of applications and for funding decisions. As shown in Table 10 and Table 11, in 2019-20, 100% of the 43 applications received were acknowledged within the 2-week standard, and in 2020-21, 99% of the 121 applications were acknowledged within the standard; in both years, all (100%) applications were provided a funding decision within the 26-week standard.

While there is no data available for the issuance of payments service standards, none of the funding recipients interviewed indicated issues with payment or other service standards.

Fiscal Year	Number of applications	Standard (Weeks)	% Met
2019-20	43	2	100
2020-21	121	2	99

Source: Service Standards Results for PCH published on Canada.ca

Fiscal Year	Number of Decisions	Standard (Weeks)	% Met
2019-20	46	26	100
2020-21	125	26	100

Source: Service Standards Results for PCH published on Canada.ca

Funding recipients were asked of their impressions regarding the extent to which DCI operates in an efficient manner. Three of the five interviewees indicated that the DCI staff ensured an efficient process by responding quickly and accurately to requests for information, and all described the application process as simple and straightforward.

During the evaluation period, the administrative ratio of efficiency for the two years DCI was evaluated were respectively 2.8 and 11.5. For every federal dollar (Vote 1) spent on administration, an average of $7.50 were given as funding to recipients. DCI received more emergency funding in 2020-21, increasing the amount of dollars of Vote 5 during the same year. It was beyond the scope of this analysis to develop a comparison with other programs.

Fiscal Year	Vote 1: Salary & EBP	Vote 1: O&M	Vote 5: G&Cs	Total
2019-20	121,046	4,000	1,886,715	2,011,761
2020-21Table 13 note *	823,914	129,250	2,682,263	3,635,427
Total	944,960	133,250	4,568,978	5,647,188

Source: Finance Management Branch Data

Table 13 notes

Table 13 note *

to October 2021

Return to table 13 note * referrer

Fiscal Year	Vote 1: Salary & EBP	Vote 1: O&M	Vote 5: G&Cs	Total	ADM RatioTable 14 note **
2019-20	608,067	53,026	1,881,335	2,542,428	2.8
2020-21Table 14 note *	550,085	25,608	6,612,403	7,188,096	11.5
Total	1,158,152	78,634	8,493,738	9,730,524	7.2

Source: Finance Management Branch Data

Table 14 notes

Table 14 note *

to October 2021

Return to table 14 note * referrer

Table 14 note **

The administrative ratio was determined by dividing the G&Cs by Vote 1 of each year. The total is the average of the 2 years.

Return to table 14 note ** referrer

Partnerships facilitated efficiency

The evaluation found examples of how partnerships support efficiency through strong communication and coordination mechanisms while also avoiding duplication of efforts. For example:

• PCH/DCI leads the DCI Interdepartmental Consultative Body, the DCI Steering Committee and the Multi-stakeholder Working Group on DoCO, which includes members from governments, civil society, and industry.
• DCI funds projects in partnership with SSHRC through the SSHRC-DCI Joint Initiative. Key informants indicated that the level of efficiency of this partnership varies based on the type of SSHRC award.
• There are an increasing number of organizations fund projects related to combatting online disinformation and there is a risk that DCI could inadvertently direct their resources in a manner that could result in some duplication and therefore inefficiencies in the overall ecosystem.

5. Conclusion

Overall, the evaluation confirms the relevance and performance of the DCI during the two-year period of 2019-20 to 2020-21. As the programming is relatively new, there are key opportunities for improvements to further ensure continued relevance and performance moving forward.

The evaluation concludes that the DCI is relevant, in part, to needs of Canadians on disinformation and online harm by supporting policy development and other government departments. The DCI has been able to respond quickly to emerging issues. However, it has limited financial and human resources relative to the size and scale of the overall problem and demands for funding. DCI filled a need by funding research to help understand disinformation, by connecting researchers and others and supporting citizen-focused activities.

The DCI supports government priorities, roles and responsibilities. Participation of and research on issues impacting equity-deserving groups were considered in DCI programming and funding. The DCI appears to complement efforts being made by other government departments. However, as additional actors enter the space, the potential for overlap and duplication could increase.

The DCI has made progress in its expected short- and medium-term outcomes. DCI funding design supports the development and maintenance of stakeholder and research communities and the dissemination of results. The program funded research and awareness activities, conferences as well as events, and these have contributed to an increased awareness of online digital issues among funding recipients and participants in its funded activities.

The DCI is a newly established initiative focused on a rapidly evolving issue in a rapidly evolving landscape. There are synergies between DCI and other federal departments who share information about priorities and potential projects to fund. Considering the funding availability and the expanding mandate of combatting online disinformation, funding priorities for DCI should be reviewed according to desired key results and outcomes. As well, it is prudent to ensure that the structure, agreements, roles and responsibilities that were established and implemented at the outset are still relevant and effective, and to update them if needed.

The evaluation identified some issues with the DCI performance indicators, for e.g., issues with indicator overlap and consistency of outcomes and indicators with program theory as well as activities. This was done by assessing the quality and relevance of available performance data, and in considering the relationship between expected outcomes, indicators, and the program logic model. In general, improved outcomes and indicators would enable the program to better measure and report on its performance.

DCI appears to have been delivered in an efficient manner. The program was able to deliver additional funding to meet new needs. Partnerships like the ones that DCI leads with the DCI Interdepartmental Consultative Body, the DCI Steering Committee and the Multi-stakeholder Working Group on DoCO, have facilitated efficient programming.

6. Recommendations, management response and action plan

Based on the findings and conclusions, the evaluation makes three recommendations to ensure continued and strengthened relevance and performance moving forward. The recommendations relate to clarifying roles and responsibilities, confirming program priorities, and improving the performance measurement strategy.

Recommendation 1

The evaluation recommends that the Senior Assistant Deputy Minister, Cultural Affairs Sector, should clarify existing roles and responsibilities for DCI within PCH and with the growing number of partners to promote strong collaboration and communication while preventing any duplication or overlap of efforts or resources.

Management response

Recommendation accepted.

The Cultural Affairs Sector acknowledges a need for greater clarity of roles and responsibilities relating to disinformation and the information ecosystem both within PCH and in the government at large. A multi-pronged approach to tackling these issues requires close inter-departmental collaboration.

The DCI’s Consultative Body is the primary tool used by PCH to promote collaboration and prevent duplication of efforts between PCH and other Government departments. The Consultative Body is made up of members representing relevant areas of policy cover across the Government of Canada. Members include representatives from PCH’s International Trade Branch and Anti-Racism Secretariat, as well as other Government Departments including Global Affairs Canada, the Privy Council Office, Public Safety Canada, Women and Gender Equality Canada, Environment and Climate Change Canada, and Statistics Canada.

In an effort to promote stronger collaboration and communication, the Broadcasting, Copyright, and Creative Marketplace Branch (BCCM) will review the membership, meeting frequency, and purpose of the Consultative Body. This review will culminate in a new Terms of Reference for the Consultative Body.

In addition, BCCM will conduct a yearly review of government-wide efforts relating to online disinformation. This yearly review will aid efforts to keep track of the growing number of partners and activities in this policy space while preventing duplication of efforts. The first two yearly reports will serve as a demonstration that this new process is in place.

The Sector acknowledges the importance of clarifying existing roles and responsibilities for DCI within PCH specifically, especially with regards to the relationship between the DCI (housed BCCM) and the DOCO initiative (housed within the International Trade Branch). Both teams were created under the umbrella of Canada’s Protecting Democracy Plan in 2019 and both serve PCH’s objectives to build a healthier and more diverse online information ecosystem. Given their shared vision and objectives, these teams regularly collaborate as part of their day-to-day activities.

Since the evaluation report, the Sector has engaged in an organizational restructuring and the separation of the DCI and DOCO has been addressed. The new Digital and Creative Marketplace Frameworks branch will work to advance the sector’s legislative priorities related to broadcasting, news media remuneration, copyright and online safety. Under this branch, the DCI and the DOCO initiative will now be regrouped under the same directorate, who will also lead on the online safety file, policy and research on harmful content and disinformation as well as international engagement. These changes took effect on May 23, 2023.

Action Plan Item	Deliverable	Timeline	Responsible
1.1. Review the membership and format of consultative body.	1.1.1 A revised DG-approved Terms of Reference for the Consultative Body which indicates membership, frequency and roles.	October 2023	Director General, Broadcasting, Copyright, and Creative Marketplace
1.2. Monitor government-wide efforts relating to online disinformation.	1.2.1 A yearly, DG-approved review of the interdepartmental environment with regard to roles and activities pertaining to online disinformation	First-year report: June 2023 Second-year report: June 2024	Director General, Broadcasting, Copyright, and Creative Marketplace
1.3 Clarify, harmonize and communicate the roles and responsibilities of the DCI and DOCO teams.	1.3.1 An ADM approved description of roles, responsibilities and organizational linkages of the DCI and DOCO teams.	October 2023	Director General, International Trade Branch Director-General, Broadcasting, Copyright, and Creative Marketplace
	1.3.2 Updated description of roles, responsibilities and organizational linkages of the DCI and DOCO teams on PCH intranet.	October 2023	Director General, International Trade Branch Director-General, Broadcasting, Copyright, and Creative Marketplace

Full implementation date: June 2024

Recommendation 2

The evaluation recommends that the Senior Assistant Deputy Minister, Cultural Affairs Sector, should review, update and communicate funding priorities for DCI to ensure achievement of key results with the limited available funding.

Management response

Recommendation accepted.

The Digital Citizen Initiative has an established process for every intake period. As the mandate of the funding program is to fund research and citizen-focused activities to combat disinformation, the field of study tends to be constantly shifting and evolving.

For this reason, the program has established a Steering Committee made up of academic and civil society leaders that are consulted on new funding priorities for the DCI each time a new intake is planned. The funding priorities for each intake are determined after consulting with the Steering Committee about the emerging research needs and trends in the field of study. Based on feedback received in the evaluation, we will maintain this practice that has proven effective.

To further ensure the achievement of key results through efficient and informed identification of funding priorities, BCCM will undergo a review of the membership, meeting frequency, and mandate of the DCI’s Steering Committee.

With an eye towards better communicating funding priorities and activities to partners, BCCM will also update the Communications Plan for the Digital Citizen Initiative.

Action Plan Item	Deliverable	Timeline	Responsible
2.1 Review the membership and format of the steering committee	2.1.1 A revised DG-approved Terms of Reference for the Steering Committee which indicates membership, frequency of meetings and roles. The ToR will also establish the frequency of steering committee membership reviews	October 2023	Director General, Broadcasting, Copyright, and Creative Marketplace
2.2 Update the Communications plan for the Digital Citizen Initiative	2.2.1 An updated, ADM-approved communications plan with updated program materials to clarify funding priorities	August 2023	Senior Assistant Deputy Minister – Cultural Affairs

Full implementation date: October 2023

Recommendation 3

The evaluation recommends that the Senior Assistant Deputy Minister, Cultural Affairs Sector, should review and update the DCI Program logic model and performance measurement indicators to ensure information is available for strategic planning, decision-making and reporting.

Management response

Recommendation accepted.

The DCI logic model and performance measurement framework (which includes DOCO) were created during the early stages of developing this program. Some of the indicators rely too much on outputs rather than outcomes and have proven difficult to measure.

The DCI was renewed for an additional two years in 2022. Through the Treasury Board renewal process, the expected results for the two initiatives were updated. As part of this process, PCH committed to updating the Performance Indicator Profile for the DCI. BCCM will work on a new updated performance measurement framework for the DCI with an eye towards identifying a clear and logical theory of change and meaningful indicators that are measurable.

Action Plan Item	Deliverable	Timeline	Responsible
3.1 Revise the DCI performance measurement framework	3.1.1 A revised, ADM-approved Performance Information Profile and Logic Model	December 2023	Director General, International Trade Branch Director General, Broadcasting, Copyright, and Creative Marketplace

Full implementation date: December 2023

Annex A: evaluation framework

Annex B: DCI performance indicators according to expected results

Expected Results	Performance Indicator(s)
Short-term
Canadians’ awareness of digital media literacy is increased through events and products.	NDMLW events held. Percentage of participants at funded events that indicate having increased awareness as a result of the events.
Researchers are better connected and have access to information on how online disinformation affects Canadians.	Number of research and/or data collection products produced by funding recipients related to online disinformation accessible to researchers available via a GoC website. Number of networking and/or information sharing events held by the PPF Institute. Percentage of attendees who indicate that their knowledge of the issue of online disinformation increased as a result of the annual PPF Institute Conference on Online Disinformation. Number of contributors and/or presenters at the annual PPF Institute Conference on Online Disinformation. Number of attendees at the annual PPF Institute Conference on Online Disinformation. Number of grant applications to the SSHRC-PCH Joint Initiative. Number of meetings of the working group held. Number of stakeholders organizations involved in the working group.
Intermediate
Participants are satisfied with citizen-focused activities.	Percentage of participants satisfied with citizen-focused activities.
Research on online disinformation is made publicly accessible to Canadians and is relevant to government research needs.	Number of new research products on the scope and impact of online disinformation in the Canadian context produced by recipients with DCI funding made publicly available. Percentage of PPF Institute conference on online disinformation content that is directly relevant to government of Canada research interests. Percentage change in the number of grant applications to the SSHRC-PCH Joint Initiative.
Participants are satisfied with National Digital Media Literacy Week activities.	Percentage of participants satisfied with NDMLW’s citizen-focused activities.
Various stakeholders are engaged towards adoption of guiding principles on DoCO.	Number of stakeholders outside of the working group engaged towards adoption of guiding principles on DoCO.
Long-term
Canadians and the Government are better equipped to counter the effects of online disinformation.	Number of new research projects produced by recipients showing the reach and impact of online disinformation on Canadians and available to inform government policy.

Source: PIP – 02 Cultural Marketplace Framework – 2020-21, p. 23-24, 28-30; DCCP T&Cs, p. 2

Annex C: bibliography

• Canadian Internet Registration Authority (CIRA). 2018. The gap between us: Perspectives on building a better online Canada.
• Jaiswal, J., LoSchiavo, C., Perlman, D.C., 2020. Disinformation, Misinformation and Inequality-Driven Mistrust in the Time of COVID-19: Lessons Unlearned from AIDS Denialism. AIDS Behav. s10461-020-02925-y.
• Kong, J., Ip, J., Huang, C., Lin, K., 2021. One year of racist attacks: Anti-Asian racism across Canada one year into the COVID-19 pandemic. Chinese Canadian National Council Toronto Chapter.
• McKay, S., Tenove, C., 2020. Disinformation as a Threat to Deliberative Democracy. Polit. Res. Q. 1065912920938143.
• PCH, Digital Citizen Contribution Program
• PCH, Online disinformation – Digital Citizen Initiative
• Statistics Canada. (February 2, 2021). Misinformation During the COVID-19 Pandemic.
• Statistics Canada. Table 17-10-0009-01 Population estimates, quarterly DOI
• Tenove, Chris, Heidi J.S. Tworek and Fenwick McKelvey. 2018. “Poisoning Democracy: How Canada Can Address Harmful Speech Online.” Public Policy Forum

©His Majesty the King in Right of Canada, represented by the Minister of Canadian Heritage, 2023
Catalogue No.: CH7-70/1-2023E-PDF
ISBN: 978-0-660-67773-6