Evaluation of the Digital Citizen Initiative 2018-19 to 2020-21

Evaluation Services Directorate
June 9, 2023

On this page

List of tables

List of acronyms

ADM
Assistant Deputy Minister
DCCP
Digital Citizenship Contribution Program
DCI
Digital Citizen Initiative
DCRP
Digital Citizen Research Program
DoCO
Diversity of Content Online
EBP
Employee Benefit Plan
ESD
Evaluation Services Directorate
FAA
Financial Administration Act
G&Cs
Grants and Contributions
GBA Plus
Gender-Based Analysis Plus
GoC
Government of Canada
NDMLW
National Digital Media Literacy Week
O&M
Operations and Maintenance
OL
Official Language
OLMC
Official Language Minority Community
PCH
Canadian Heritage
PCO
Privy Council Office
PIP
Performance Information Profile
PPF
Public Policy Forum
SSHRC
Social Sciences and Humanities Research Council
T&Cs
Terms and Conditions

Alternate format

Evaluation of the Digital Citizen Initiative 2018-19 to 2020-21 [PDF version - 552 KB]

Executive summary

The Digital Citizen Initiative (DCI) is a multi-component strategy that aims to support democracy and social cohesion in Canada by building citizen resilience against online disinformation and building partnerships to support a healthy information ecosystem. The DCI has two components:

  1. the Digital Citizen Research Program (DCRP) which includes the Digital Citizenship Contribution Program (DCCP); and
  2. the Diversity of Content Online (DoCO Initiative).

The DCI was established and initially funded through Budget 2019 with a focus on projects related to election misinformation but has evolved over the evaluation period to include a broader focus, including COVID-19 specific project funding.

This evaluation is the first for DCI and its scope includes activities undertaken between 2018-19 and 2020-21. This evaluation used a mixed-method approach including administrative data and file review, document review, literature review, and key informant interviews. The evaluation focused primarily on the DCRP.

Relevance

Overall, the evaluation finds that both components of the DCI is relevant to many key needs, to government priorities, and to the Department of Canadian Heritage (PCH)’s mandate. DCI fills an important need by funding research to help define and understand the problem of disinformation; connecting researchers and others in a community of practice; and supporting citizen-focused activities designed to help Canadians become more resilient to disinformation and associated harms.

Disinformation impacts Canadians’ health and safety, civic discourse and engagement, political beliefs, perceptions of democratic institutions, confidence in political systems and trust in the media. It may also amplify mistrust among communities, discrimination, stigma and marginalization, and social divisions. The DCI has been able to respond quickly to emerging situations such as the COVID-19 pandemic but has limited financial and human resources relative to the size and scale of the overall problem.

It is widely accepted that the issue of disinformation and its related harms are increasing significantly on a global scale, and Canadians are not immune to these harms. Equity groups including racialized populations and women tend to face disproportionate levels of harm because of disinformation; many DCI funded projects have been focused specifically on the needs and priorities of these groups. Participation of and research on issues impacting targeted groups (youth, Indigenous communities, Official Language Minority Communities (OLMCs), ethnocultural communities) were prioritized and considered in many DCI documents. DCI tracks results with partners and considers how it fulfils Official Language (OL) and Gender-Based Analysis Plus (GBA Plus) requirements.

The scale and scope of the problem being addressed by the DCI is expanding along with potential for associated harms. The evaluation concludes that current DCI human and financial resources limit the achievement of additional outcomes and that more impact could be achieved through larger projects and a higher number of projects. Of the 171 applications received in 2019-20 and 2020-21, only 31% were approved, with 69% rejected. The high rate of rejection is, in part, a result of the limited available resources compared to demand.

The DCI appears to complement efforts being made by other government departments. Document review and key informant interviews indicate no duplication and in fact there are synergies between DCI and other federal departments. There is good sharing of information about priorities and potential projects to fund among partners. However, as additional actors enter the space, the potential for overlap and duplication could increase.

Effectiveness

The DCI has made progress in its expected short- and medium-term outcomes. The program funded research and awareness activities, contributing to an increased awareness of online digital issues among funding recipients and participants in its funded activities. Research products being funded by DCI are being made publicly available. DoCO has achieved stated expected outcomes of creating of a multi-stakeholder working group and delivering the Guiding Principles on Diversity of Content Online.

DCI enhances connections between researchers and their access to information. DCI’s funding design and support for dissemination of project outputs help bring together new and established networks of researchers and organizations. Conferences and meetings organized by the DCI to help disseminate research findings and project outputs have been effective, despite having to move online due to the COVID-19 pandemic.

DCI-funded projects appear to have increased the digital media literacy of some Canadians. National Digital Media Literacy Week (NDMLW) events in 2020 and 2021 were funded by DCI through the DCCP. Other citizen-focused projects have also been successful in achieving expected results, including some related to COVID-19 pandemic misinformation.

The evaluation identified some issues with the DCI performance indicators. The evaluation largely relied on available data related to short- and medium-term performance indicators. There are opportunities to confirm and improve the expected outcomes and their associated indicators to better explain the program’s objectives and measure performance for reporting and for decision-making.

Efficiency

Overall, it appears that the DCI is delivered in an efficient manner. During 2019-20 and 2020-21, grants and contributions accounted for 87% of total DCI direct costs; with salaries and Operations and Maintenance (O&M) comprising 13% of total direct costs. Service standards were almost entirely achieved, and project recipients reported that the program is administratively efficient.

Partnerships facilitated the efficient delivery of programming. PCH leads the DCI Interdepartmental Consultative Body, the DCI Steering Committee and the Multi-Stakeholder Working Group on DoCO, which includes members from other governments, funded recipients, civil society organizations, private sector. DCI funds projects in partnership with the Social Sciences and Humanities Research Council (SSHRC) through the SSHRC-DCI Joint Initiative.

Recommendations

Based on the findings and conclusions, the evaluation has made the following three recommendations to ensure continued and strengthened relevance and performance of DCI moving forward. The Senior Assistant Deputy Minister (ADM), Cultural Affairs Sector, should:

Recommendation 1

Clarify existing roles and responsibilities for DCI within PCH and with the growing number of partners to promote strong collaboration and communication while preventing any duplication or overlap of efforts or resources.

Recommendation 2

Review, update and communicate funding priorities for DCI to ensure achievement of key results with the limited available funding.

Recommendation 3

Review and update the DCI Program logic model and performance measurement indicators to ensure information is available for strategic planning, decision-making and reporting.

1. Introduction

This report presents the findings, conclusions and recommendations resulting from the evaluation of the DCI. The evaluation was conducted to address evaluation requirements outlined in the Treasury Board Secretariat Policy on Results (2016) and the Financial Administration Act (FAA).

The evaluation covers the three-year period from April 1, 2018, to March 31, 2021, and examines targeted issues related to relevance, effectiveness, and efficiency.

2. Program profile

DCI was established and initially funded through Budget 2019 with a focus on projects related to election misinformation, it has evolved over the evaluation period to include a broader focus, including COVID-19 specific project funding. It is a multi-component strategy that aims to support democracy and social cohesion in Canada by building citizen resilience against online disinformation and building partnerships to support a healthy information ecosystem. DCI supports a community of Canadian researchers that promote a healthy information ecosystem, helps Canadians and the Government understand online disinformation and its impact on Canadian society, and in turn builds an evidence base to identify potential action and develop future policy-making.Footnote 1

2.1 Program activities, objectives and expected outcomes

The DCI is composed of two primary activities: the DCRP and the DoCO Initiative. In summary:

  1. The DCRP provides digital media literacy programming to Canadians, and conducts research that helps understand disinformation, with a focus on supporting the development of a policy and research agenda to guide Canadian action. The DCRP is implemented through two transfer payment activities: the DCCP and a joint initiative with the SSHRC, the Joint Initiative for Digital Citizen Research. As part of the research program, the DCCP supports the priorities of the DCI by providing time-limited financial assistance for research and citizen-focused activities.Footnote 2
  2. DoCO is a PCH-led initiative mandated to develop Guiding Principles on Diversity of Content Online through international multi-stakeholder engagement with government partners, industry, civil society organizations. The Guiding Principles are a strategic framework to guide actions and measures that foster greater exposure to diverse cultural content, information and news online.

The program’s logic model shown in Table 1 outlines the DCI’s short, medium, and long-term expected outcomes.

Table 1: DCI logic model
Long-Term Outcome
  • Canadians’ awareness of digital media literacy is increased through events and products.
  • Researchers are better connected and have access to information on how online disinformation affects Canadians.
Medium-Term Outcomes
  • Participants are satisfied with citizen-focused activities.
  • Research on online disinformation is made publicly accessible to Canadians and is relevant to government research needs.
  • Participants are satisfied with National Digital Media Literacy Week activities.
  • Stakeholders are engaged in developing the Guiding Principles on DoCO and adopt them.
Short-Term Outcome
  • Canadians and the Government are better equipped to counter the effects of online disinformation.

Source: PIP – 02 Cultural Marketplace Framework – 2020-21, p. 23-24, 28-30; DCCP T&Cs, p. 2

2.2 Program management and governance

The DCRP and the DoCO Initiative are organizationally under two different branches within PCH’s Cultural Affairs Sector. The DCRP is administered by a core team within the Digital Citizen Initiative Directorate which falls under the Broadcasting, Copyright and Creative Marketplace Branch. The DoCO Initiative is under the responsibility of the International Trade Directorate within the International Trade Branch.Footnote 3

The DCI uses two governance committees – the Consultative Body and the Steering Committee. The former is made up of representatives from other Government of Canada departments. The latter consists of representatives from academia and civil society.

DoCO governance is centered on a Multi-Stakeholder Working Group established by PCH and comprised of representatives from governments, the private sector, civil society, and a para-public institute. The Working Group’s mandate is to develop Guiding Principles that help foster greater exposure to diverse cultural content, information, and news online.

2.3 Program resources

As shown in Table 2, the planned total program spending was $5.7 million and in Table 3 the actual spending was $9.7 million from 2019-20 to 2020-21. The increased actual spending is mostly due to additional funding provided to DCI to administer COVID-19 specific calls for proposals over this period. It should be noted that no funding was received in 2018-19.

Table 2: DCI total planned spending ($) per fiscal year, 2019-20 and 2020-21
Fiscal Year Vote 1: Salary & EBPTable 2 note 1 Vote 1: O&MTable 2 note 2 Vote 5: G&CsTable 2 note 3 Total
2019-20 121,046 4,000 1,886,715Table 2 note 4 2,011,761
2020-21 823,914 129,250 2,682,263 3,635,427
Total 944,960 133,250 4,568,978 5,647,188

Source: Financial Planning and Resource Management, PCH

Table 2 notes

Table 2 note 1

Employee Benefit Plan

Return to table 2 note 1 referrer

Table 2 note 2

Operation and Maintenance

Return to table 2 note 2 referrer

Table 2 note 3

Grants and Contributions

Return to table 2 note 3 referrer

Table 2 note 4

Planned amount in 2019-20 for the commitment item 4582 (Digital Citizen (c)) under fund 2245

Return to table 2 note 4 referrer

Table 3: DCI total actual spending ($) per fiscal year, 2019-20 and 2020-21
Fiscal Year Vote 1: Salary & EBPTable 3 note 1 Vote 1: O&MTable 3 note 2 Vote 5: G&CsTable 3 note 3 Total
2019-20 608,067 53,026Table 3 note 4 1,881,335 2,542,428
2020-21 550,085 25,608 6,612,403 7,188,096
Total 1,158,152 78,634 8,493,738 9,730,524

Source: Financial Planning and Resource Management, PCH

Table 3 notes

Table 3 note 1

Employee Benefit Plan

Return to table 3 note 1 referrer

Table 3 note 2

Operation and Maintenance

Return to table 3 note 2 referrer

Table 3 note 3

Grants and Contributions

Return to table 3 note 3 referrer

Table 3 note 4

O&M includes 28,989$ in “Goods and Services” and 24,037$ in “Travel, Conference & Hospitality”

Return to table 3 note 4 referrer

According to PCH’s Financial Planning and Resource Management, the actual full-time equivalents for the DCI was 10.5 in 2019-20 and 18.4 in 2020-21.

3. Approach and methodology

The evaluation was undertaken by the Evaluation Services Directorate (ESD) with support from an evaluation consulting firm. It was conducted as prescribed in the Canadian Heritage (PCH) Departmental Evaluation Plan 2021-22 to 2025-26. This section outlines the evaluation approach and methodology including scope, timelines, calibration, evaluation questions, data collection methods, limitations and mitigation strategies.

3.1 Scope, timeline and quality control

The scope of the evaluation includes DCI activities undertaken between 2018-19 and 2020-21. This evaluation was not an FAA requirement and was undertaken at the request of the Broadcasting, Copyright and Creative Marketplace Branch to document the continued relevance and effectiveness of the grants and contributions funding within DCI. The evaluation therefore focused primarily on the DCRP component of DCI.

Initial scoping meetings were held with the program officials, including senior management, to determine specific information needs and to refine the evaluation scope. Key information needs related to notably, resources, pivoting rapidly to meet governmental priorities, oversubscription and measuring results.

The evaluation was conducted in a manner consistent with the Treasury Board Secretariat Policy on Results (2016) and the Directive on Results (2016). The following quality assurance measures were in place for the evaluation:

3.2 Calibration

Calibration is the process of adjusting an item, such as an evaluation approach or tool, to the sensitivity required to suit a particular function. It is based on several different factors to produce quality evaluations cost effectively and within timelines. A phased approach was taken for the lines of evidence, to permit some early data results in Phase 1 (prior to March 31, 2022) with confirmed findings in Phase 2 (after March 31, 2022).

This evaluation was also calibrated to focus on key questions by: reducing the number of evaluation questions; focusing on existing data sources where possible; undertaking targeted data collection; and, streamlining the report.

3.3 Evaluation questions

The following questions in Table 4 were used to guide the evaluation. The Evaluation Matrix included as Annex A provides additional detail in the form of data sources and indicators linked to each of the following questions.

Table 4: evaluation Issues and Questions
Core Issue Evaluation questions
Relevance

1.1 To what extent does the DCI address the continued, emerging and changing needs of Canadians on disinformation and online harm?

1.2 To what extent does DCI supports government priorities, including those related to equity groups?

1.3 To what extent does the DCI duplicate or complement other programs?

Effectiveness 2.1 To what extent did the DCI achieve its short and medium-term expected outcomes?
Efficiency 3.1 To what extent is the programming delivered in an efficient manner?

3.4 Data collection methods

This evaluation used a mixed-method approach including: administrative data and file review, document review, literature review, and key informant interviews. Each method is briefly described in the subsections below.

The evaluation uses the following legend throughout the report to indicate the proportion of individuals interviewed or surveyed that responded in the same manner:

3.4.1 Administrative data and file review

The evaluation team reviewed and analyzed data from PCH’s internal system, Grants and Contributions Information Management System, from the program, as well as other administrative data including financial data received by the Financial Management Branch. All information was validated with Initiative representatives.

3.4.2 Document review

The evaluation team reviewed key DCI and Government of Canada documents. The review included, but was not limited to: submissions for funding, departmental and program policies, directives, guidelines, application forms, Terms & Conditions, program audits, meeting minutes, communications and outreach products, Speeches from the Throne, Federal Budgets, Reports to Parliament, and Statistics Canada reports.

3.4.3 Literature review

A targeted review was undertaken of research from recently published literature, reports, websites, public opinion research and analyses related to disinformation and online harms.

3.4.4 Key informant interviews

A total of 14 key informant interviews were conducted with 5 funding recipients, 5 internal PCH managers and analysts, and 4 members of the Consultative Body, Steering Committee, or other federal government partners.

3.5. Evaluation limits and mitigation

Table 5 outlines and key constraints and limitations of the evaluation process.

Table 5: limitations and mitigation strategies
Limitation Mitigation Strategy
There were changes to the programming context and delivery caused by the COVID-19 pandemic starting in 2019-20. The evaluation examined and considered any changes in context, including those caused by the pandemic, in data collection, analysis and reporting.
The ability of the ESD and programming to respect timelines due to the COVID-19 pandemic and related resourcing issues.

The evaluation project was calibrated to focus on most important information needs identified in planning. It leveraged as much existing data as possible while supplementing with new data collection when needed. An external consultant was engaged to support the project.

To meet the program’s request for early preliminary results, the evaluation was divided into Phase 1 (analysis using only certain lines of evidence by March 31, 2022) and Phase 2 (triangulation of all lines of evidence, after March 31, 2022).

Variations in data entry methodologies could impact the validity of the data in GCIMS. Data pulled from GCIMS was cross-referenced with program data, and validated by program, and limitations in GCIMS data were described, as appropriate.
The evaluation focused primarily on the DCRP component of DCI. When possible, the evaluation report includes finding relating to the DoCO component in the Effectiveness and Efficiency sections. None are included in the Relevance section.

4. Findings

4.1 Relevance

4.1.1 Relevance: ongoing need for the program

Evaluation question: To what extent does the DCI address the continued, emerging and changing needs of Canadians on disinformation and online harm?

Key findings:

  • DCI addresses real and continuing needs of Canadians with respect to online disinformation and related harms. The demand for DCI funding appears to outweigh available resources. The scope and scale of the issue are rapidly expanding along with the potential for harm.
  • The DCI is a flexible program that has shown the ability to respond to emerging issues of online disinformation within a constantly shifting landscape. DCI pivoted in response to disinformation related to the COVID-19 pandemic. The short response time with projects launching within months of the outset of the pandemic was highlighted as evidence of the flexibility and responsiveness of the DCI.
DCI is a flexible program that can respond quickly to emerging issues of online disinformation

Research literature shows that disinformation impacts Canadians’ health and safety, civic discourse and engagement, political beliefs, perceptions of democratic institutions, confidence in political systems and trust in the media. Disinformation may also amplify mistrust among communities, discrimination, stigma and marginalization, as well as exacerbate social divisions.Footnote 4,Footnote 5 Some groups, such as people with low digital literacy skills, may be more susceptible to misinformation campaignsFootnote 6; others, such as racialized populationsFootnote 7 and womenFootnote 8, may be disproportionately harmed because of the spread of disinformation.

In response to the potential for harm, the Government of Canada provided $7.5 million over 2018-19 and 2019-2020 to support digital, news, and civic literacy programming and tools by funding citizen-focused activities, using existing PCH funding programs. This funding went to 20 projects that reached 12 million Canadians. In 2019-20, a further $19.4 million over four years was budgeted to create the DCI and the DCCP to fund applied research activities in line with the government’s need to better understand the origins, impacts and potential responses to online disinformation in Canada. During the two years of funding covered by this evaluation, 2019-20 and 2020-21, 53 additional projects totalling $9.9 million were funded, as was the SSHRC-PCH Joint Initiative, the Public Policy Forum (PPF) and MediaSmarts’ Digital Media Literacy Week (DMLW).

Key informants, including funding recipients, members of the consultative body and steering committee, and PCH management and staff indicated that the DCI addresses real and continuing needs of Canadians with respect to online disinformation and related harms. They reported that the scope and scale of the issue are rapidly expanding along with the potential for harm, and the DCI is a flexible program that has shown the ability to respond to emerging issues within a constantly shifting landscape.

The demand for DCI funding appears to outweigh available resources

The scale and scope of the problem addressed by the DCI are expanding along with potential for associated harms. Key informants indicated that their current human and financial resources limit the achievement of additional outcomes and that more impact could be achieved through larger projects and a higher number of projects. Of the 171 applications received in 2019-20 and 2020-21, 53 (or 31%) were approved, and 118 (69%) were rejected. Administrative data did not suggest a rationale for why applications were denied, with most rejection decisions categorized simply as ‘unsuccessful’. However, internal key informants indicated that DCI has limited resources relative to the scale of the problem and the demand from interested proponents.

DCI pivoted in response to disinformation related to the COVID-19 pandemic

Canadians have been subject to vast amounts of misinformation and disinformation related to the COVID-19 pandemic. In the first few months of the pandemic, 90% of Canadians used online sources to find information about COVID-19. Of these Canadians, 96% saw COVID-19 information that they suspected was misleading, false or inaccurate; and nearly two in five Canadians (40%) reported believing the information they saw, then later realizing that it was not accurate.Footnote 9

The DCI allocated over $4 million in funding for activities combatting COVID-19 disinformation, including $2.7 million to 8 approved applications in 2019-20 and $1.7 million to an additional 26 approved applications in 2020-21. The speed of response, with projects launching within months of the outset of the pandemic, was highlighted by most key informants as evidence of the flexibility and responsiveness of the DCI.

As shown in Table 6, the number of projects and amounts approved per capita varied between provinces and territories during the evaluation period. For example, Ontario-based applicants had 1.63 projects funded for every million residents of Ontario with approved amounts equal to $0.44 per capita whereas BC-based applicants had 0.77 projects approved for every million population with approved amounts equal to $0.05 per capita, or just 11% of the per capita amount approved to Ontario-based applicants.

Table 6: number of funded applications per province and total funding approved, from 2019-20 to 2020-21
Primary province of funded applicants # of funded applications Total application amount approved ($) Average amounts of funding per project ($) Population (Q1, 2021)Footnote 10 Projects per million population Funding approved per capita ($)
Ontario 24 6,426,529 267,772 14,740,102 1.63 0.44
Quebec 11 1,448,710 131,701 8,579,010 1.28 0.17
Atlantic Region 4 614,125 153,531 2,447,663 1.63 0.25
Northwest Territories, Nunavut and Yukon 2 248,087 124,044 127,161 15.7 1.95
Alberta 5 793,381 158,676 4,429,077 1.13 0.18
British Columbia 4 271,760 67,940 5,157,293 0.78 0.05
Manitoba 2 80,000 40,000 1,384,025 1.45 0.06
Saskatchewan 1 39,908 39,908 1,179,119 0.85 0.03
Canada 53 9,922,500 187,217 38,043,450 1.39 0.26

Source: GCIMS administrative data, Statistics Canada data

The data in Table 7 and Table 8 shows that most approved applications (46/53 or 87%) to DCI were made by recurrent applicants to DCI (n=26) or other PCH programs (n=20). Of the 53 approved applications, just 7 (13%) were made by applicants who were not recurrent to any PCH program. Stated differently, among the 171 applications, there were 142 made by applicants who were recurrent to PCH and/or DCI, and 37% of those applications to DCI were approved; among the remaining 47 applications made by applicants that were not recurrent, 15% were approved.

Table 7: recurrent applicants to DCI and PCH by DCI approval status (#)
Recurrence Status Applications Approved Applications Not Approved Total
Recurrent to DCI 26 57 83
Recurrent to PCH but not to DCI 20 21 41
Not recurrent to PCH 7 40 47
Total 53 118 171

Source: GCIMS administrative data

Table 8: recurrent applicants to DCI and PCH by DCI approval status (%)
Recurrence Status Applications Approved Applications Not Approved Total
Recurrent to DCI 49 48 49
Recurrent to PCH but not to DCI 38 18 24
Not recurrent to PCH 13 34 27
Total 100 100 100

Source: GCIMS administrative data

4.1.2 Relevance: harmonization with government priorities, roles, and responsibilities

Evaluation question: To what extent does the DCI support government priorities, including those related to equity-deserving groups?

Key findings:

  • DCI supports government priorities, roles and responsibilities and considers equity-deserving groups in its design and implementation.
  • DCI funds citizen-focused activities and research to support democracy and social cohesion in Canada by building resilience against online discrimination. It is also designed to support a variety of different activities throughout PCH to foster diversity and inclusion, as well as Canadian identity and values. Equity-deserving groups often face disproportionate harm from disinformation, and many DCI funded projects have been focused specifically on the needs and priorities of these groups.
The DCI supports government priorities, roles and responsibilities

The DCI contributes to PCH’s core responsibilities related to creativity, arts, and culture. The DCI funds citizen-focused activities and research to support democracy and social cohesion in Canada by building resilience against online discrimination. It is also designed to support a variety of different activities throughout PCH to foster diversity and inclusion, as well as Canadian identity and values. For example, DCI supports activities through Youth Take Charge and the Canada History Fund.

Equity-deserving groups often face disproportionate harms due to disinformation

Many DCI funded projects have been focused on the needs and priorities of equity-deserving groups. For example, projects have been directed at countering anti-Asian racism during the COVID-19 pandemic as well as understanding the problem of misogynistic hate speech being directed at women online.

An initial GBA Plus analysis done for the DCI found that there was a gap in current information about how disinformation affects different groups in Canada. To address that gap, the DCI has used its research component to gain a greater understanding of the issues and solutions. As well, participation of and research on issues impacting targeted groups such as youth, Indigenous communities, OLMCs, ethnocultural communities, were prioritized and considered in many DCI documents including in its Terms & Conditions and project proposal assessment forms.

DCI tracks results with partners and considers how it fulfills OL and GBA Plus requirements. In 2019-20, 29%, of DCCP/DCI funded projects directly and specifically addressed ethnic, racial or religious communities, minority communities, in both official languages and in Indigenous communities. In 2020-21, $4.3 million was dedicated specifically to counter COVID-19 disinformation, and the racism and stigmatization that are often the result. Finally, in the context of the SSHRC-PCH Joint Initiative, an effort was made in 2021-22 to continue funding research on vulnerable population and online disinformation.

Funded recipients were asked if they knew of any DCI operating or funding barriers specific to equity-deserving groups. None of the funded recipients indicated that they knew of any such barriers: two funded recipients indicated that their organizations served equity-deserving groups and had no barriers in the program.

4.1.3 Relevance: extent of complementarity or duplication

Evaluation question: To what extent does the DCI duplicate or complement other programs?

Key findings:

  • DCI appears to complement efforts being made by research and non-governmental organizations as well as other government departments.
  • There is evidence of synergies between DCI and other federal departments who share information about priorities and potential projects to fund.
  • As additional actors enter the space, the potential for overlap and duplication could increase.
DCI is complementary and not duplicative of other programs and activities

According to key informants and document review, DCI appears complementary despite a number of other existing initiatives. Many key informants indicated that DCI was the first funding source of its kind. Many were able to mention other government departments and agencies as well as foundations and private sector entities that also provided funding in various ways to help address disinformation-related harms. For example, it was noted that a Global Affairs Canada funding program focused on the international aspects of online harms, and that the Public Health Agency of Canada had funding available for information dissemination and that could include correcting health-based disinformation. Also, an internal key informant mentioned that the PCO funds disinformation-related behavioural science projects through its Impact and Innovation unit; they also reported working closely with this unit to ensure there is no duplication of funded projects and look for opportunities for proponents that have applied to one program but are better suited to another program.

Some efforts have been made by digital industry companies to counter disinformation. These efforts do not appear to be involved in building research communities, digital literacy or resiliency among social media users and so do not overlap with the purpose of the DCI.

Evidence indicates that unique features to DCI include: national scope, funding levels and funding priorities. The primary way DCI maintains complementarity is through good communication and collaboration mechanisms.

4.2 Effectiveness

4.2.1 Effectiveness: achievement of short-term expected outcomes

Evaluation question: To what extent did the DCI achieve its short-term expected outcomes?
- Researchers are better connected and have access to information on how online disinformation affects Canadians.
- Canadians’ awareness of digital media literacy is increased through events and products.

Key findings:

  • The DCI has made progress in its short-term expected outcomes.
  • DCI funding design supports the development and maintenance of stakeholder and research communities and the dissemination of results. The program funded research and awareness activities over the evaluation period, and these have contributed to an increased awareness of online digital issues among funding recipients and participants in its funded activities. Citizen-focused projects supported by the DCI have found that they are able to impact education and behaviour change by engaging with Canadians on the topic of disinformation and online harm.
  • There are some opportunities to improve the expected outcomes and associated indicators.
DCI enhances connections between researchers and their access to information

According to key informants, the DCI’s funding design and support for dissemination of project outputs help bring together new and established networks of researchers and organizations. All five of the project recipients interviewed indicated that the process of carrying out their funded projects relied on stakeholders and networks of researchers or organizations coming together to work on shared issues. Of note, all funding recipients indicated that conferences and meetings organized by the DCI to help disseminate research findings and project outputs have been effective, despite having to move online due to the COVID-19 pandemic. The program also supports access to research and project results by obtaining research abstracts from each funded project and making them available to DCI stakeholders on an internal SharePoint site.

DCI funding also contributes to the holding of different types of events, notably:

As well, the SSHRC-PCH Joint Initiative funded applications during the evaluation period. The number of applications submitted to SSHRC-DCI per fiscal year and per type of award or grant during the evaluation period appears in Table 9. Of the 84 applications submitted, administrative data shows that 38 applications were funded a total of $1.5 million. Most (n=32) funded recipients were individuals/sole proprietorships and six recipients were in academia. Most recipients were from Ontario, Québec, and British Columbia. Most (93%) funds went to individual/sole proprietorship recipients. While accounting for just 24% of agreements, recipients from Quebec accounted for 59% of total agreement funds.

Table 10: number of applications submitted by award or grant name to SSHRC-DCI Joint Initiative
Fiscal Year ConnectionFootnote 11 TalentFootnote 12 InsightFootnote 13 Total
2019-20 7 10 31 48
2020-21 2 18 16 36
Total 9 28 47 84

Source: Program data

DCI-funded projects appear to have increased the digital medial literacy of some Canadians

NDMLW events in 2020 and 2021 were funded by DCI through the DCCP. The inception of the event does predate DCI’s involvement but was previously carried out on a much smaller scale because of a lack of resources, and, therefore, reached far fewer Canadians than it did with events supported by the DCI. According to administrative data, in 2020-21, approximately 59% of participants indicated having increased their awareness because of the event.Footnote 14

Other citizen-focused projects have also had an impact. For example, according to a funding recipient, an evaluation carried out on the impact of a web-app related to COVID-19 misinformation, developed with the support of DCI, found that users of the app had both learned and applied those learnings to behaviour changes and that those changes were maintained over time.

Some issues exist with the short-term outcome performance indicators and data

The analysis has relied on available data from this outcome’s broad, short-term indicators. Issues were identified by assessing the quality and relevance of available performance data, and in considering the relationship between expected outcomes, indicators, and the program logic model. In particular:

4.2.2 Effectiveness: achievement of medium-term expected outcomes

Evaluation question: To what extent did the DCI achieve its medium-term expected outcomes?
- Participants are satisfied with citizen-focused activities and NDMLW activities.
- Research on online disinformation is published, made publicly accessible to Canadian and is relevant to policy-making.
- Various stakeholders are engaged towards adoption of guiding principles on DoCO.

Key findings:

  • There is some evidence that that DCI has partially achieved its the medium-term expected outcomes.
  • Participants appear to be satisfied with citizen-focused activities and that research products being funded by DCI are being made publicly available and, in many cases, have been of interest to policy-makers in Canada and internationally.
  • DoCO has successfully fulfilled its expected outcomes by establishing an international working group and developing the Guiding Principles on DoCO.
  • As with the short-term outcomes, there are some challenges with the medium-term outcomes and indicators.

Through the DCCP, DCI has provided funding to support MediaSmarts’ NDMLW since 2020 to raise awareness of digital and media literacy across Canada and to provide funding to research- and citizen-focused activities that support efforts to counter online disinformation. Based on available performance data, 100% of participants in 2020-21 were satisfied with citizen-focused activities, and 89% of participants rated their satisfaction of the 2020 NDMLW event a “4” or “5” on scale of 1 to 5.

According to key informants, including DCI management and funding recipients, several federal government departments have shown an interest in research funded by the DCI and funded recipients have indicated that they have delivered presentations to federal departments and international stakeholders about their findings. In terms of determining the influence of the research on policy-making, there was no clear data available.

DCI performance indicator data supports that the program supported the publication and accessibility of research on online disinformation. In 2019-20 and 2020-21:

DoCO has developed Guiding Principles on Diversity of Content Online

The Multi-stakeholder Group on DoCO was mandated to develop guiding principles on diversity of content online to help build citizen resilience to disinformation and contribute to greater social cohesion. The Multi-stakeholder Group is comprised of countries such as Australia, Finland, France, and Germany, and representatives from online platforms, and civil society. The evaluation found that the principles were developed. According to key informants, the group has fulfilled its initial mandate. However, it is not clear the extent to which the medium-term outcome performance outcome “various stakeholders are engaged towards adoption of guiding principles on diversity of content online” has been achieved otherwise.

Some issues exist with the medium-term outcome performance indicators and data

There are some challenges with the medium-term outcomes and indicators when considering the relationship between expected outcomes, indicators, and the program logic model. For example:

4.3 Efficiency

Evaluation questions: To what extent is DCI delivered in an efficient manner?

Key finding:

  • Overall, the DCI appears to be delivered efficiently.
  • Service standards have been achieved nearly 100% of the time and project recipients appear satisfied with program administration. During 2019-20 and 2020-21, grants and contributions accounted for 87% of total DCI direct costs versus 13% for salaries, EBP and O&M.
Service standards have been achieved

In 2019-20 and 2020-21, the program met service standards for the acknowledgement of reception of applications and for funding decisions. As shown in Table 10 and Table 11, in 2019-20, 100% of the 43 applications received were acknowledged within the 2-week standard, and in 2020-21, 99% of the 121 applications were acknowledged within the standard; in both years, all (100%) applications were provided a funding decision within the 26-week standard.

While there is no data available for the issuance of payments service standards, none of the funding recipients interviewed indicated issues with payment or other service standards.

Table 11: adherence to DCI Service Standards – Acknowledgment of Receipt of Application
Fiscal Year Number of applications Standard (Weeks) % Met
2019-20 43 2 100
2020-21 121 2 99

Source: Service Standards Results for PCH published on Canada.ca

Table 12: adherence to DCI Service Standards – Funding Decision
Fiscal Year Number of Decisions Standard (Weeks) % Met
2019-20 46 26 100
2020-21 125 26 100

Source: Service Standards Results for PCH published on Canada.ca

Funding recipients were asked of their impressions regarding the extent to which DCI operates in an efficient manner. Three of the five interviewees indicated that the DCI staff ensured an efficient process by responding quickly and accurately to requests for information, and all described the application process as simple and straightforward.

During the evaluation period, the administrative ratio of efficiency for the two years DCI was evaluated were respectively 2.8 and 11.5. For every federal dollar (Vote 1) spent on administration, an average of $7.50 were given as funding to recipients. DCI received more emergency funding in 2020-21, increasing the amount of dollars of Vote 5 during the same year. It was beyond the scope of this analysis to develop a comparison with other programs.

Table 13: DCI Planned Spending ($), 2019-20 to 2020-21
Fiscal Year Vote 1: Salary & EBP Vote 1: O&M Vote 5: G&Cs Total
2019-20 121,046 4,000 1,886,715 2,011,761
2020-21Table 13 note * 823,914 129,250 2,682,263 3,635,427
Total 944,960 133,250 4,568,978 5,647,188

Source: Finance Management Branch Data

Table 13 notes
Table 13 note *

to October 2021

Return to table 13 note * referrer

Table 14: DCI Actual Spending and Administrative Ratio of Efficiency ($), 2019-20 to 2020-21
Fiscal Year Vote 1: Salary & EBP Vote 1: O&M Vote 5: G&Cs Total ADM RatioTable 14 note **
2019-20 608,067 53,026 1,881,335 2,542,428 2.8
2020-21Table 14 note * 550,085 25,608 6,612,403 7,188,096 11.5
Total 1,158,152 78,634 8,493,738 9,730,524 7.2

Source: Finance Management Branch Data

Table 14 notes
Table 14 note *

to October 2021

Return to table 14 note * referrer

Table 14 note **

The administrative ratio was determined by dividing the G&Cs by Vote 1 of each year. The total is the average of the 2 years.

Return to table 14 note ** referrer

Partnerships facilitated efficiency

The evaluation found examples of how partnerships support efficiency through strong communication and coordination mechanisms while also avoiding duplication of efforts. For example:

5. Conclusion

Overall, the evaluation confirms the relevance and performance of the DCI during the two-year period of 2019-20 to 2020-21. As the programming is relatively new, there are key opportunities for improvements to further ensure continued relevance and performance moving forward.

The evaluation concludes that the DCI is relevant, in part, to needs of Canadians on disinformation and online harm by supporting policy development and other government departments. The DCI has been able to respond quickly to emerging issues. However, it has limited financial and human resources relative to the size and scale of the overall problem and demands for funding. DCI filled a need by funding research to help understand disinformation, by connecting researchers and others and supporting citizen-focused activities.

The DCI supports government priorities, roles and responsibilities. Participation of and research on issues impacting equity-deserving groups were considered in DCI programming and funding. The DCI appears to complement efforts being made by other government departments. However, as additional actors enter the space, the potential for overlap and duplication could increase.

The DCI has made progress in its expected short- and medium-term outcomes. DCI funding design supports the development and maintenance of stakeholder and research communities and the dissemination of results. The program funded research and awareness activities, conferences as well as events, and these have contributed to an increased awareness of online digital issues among funding recipients and participants in its funded activities.

The DCI is a newly established initiative focused on a rapidly evolving issue in a rapidly evolving landscape. There are synergies between DCI and other federal departments who share information about priorities and potential projects to fund. Considering the funding availability and the expanding mandate of combatting online disinformation, funding priorities for DCI should be reviewed according to desired key results and outcomes. As well, it is prudent to ensure that the structure, agreements, roles and responsibilities that were established and implemented at the outset are still relevant and effective, and to update them if needed.

The evaluation identified some issues with the DCI performance indicators, for e.g., issues with indicator overlap and consistency of outcomes and indicators with program theory as well as activities. This was done by assessing the quality and relevance of available performance data, and in considering the relationship between expected outcomes, indicators, and the program logic model. In general, improved outcomes and indicators would enable the program to better measure and report on its performance.

DCI appears to have been delivered in an efficient manner. The program was able to deliver additional funding to meet new needs. Partnerships like the ones that DCI leads with the DCI Interdepartmental Consultative Body, the DCI Steering Committee and the Multi-stakeholder Working Group on DoCO, have facilitated efficient programming.

6. Recommendations, management response and action plan

Based on the findings and conclusions, the evaluation makes three recommendations to ensure continued and strengthened relevance and performance moving forward. The recommendations relate to clarifying roles and responsibilities, confirming program priorities, and improving the performance measurement strategy.

Recommendation 1

The evaluation recommends that the Senior Assistant Deputy Minister, Cultural Affairs Sector, should clarify existing roles and responsibilities for DCI within PCH and with the growing number of partners to promote strong collaboration and communication while preventing any duplication or overlap of efforts or resources.

Management response

Recommendation accepted.

The Cultural Affairs Sector acknowledges a need for greater clarity of roles and responsibilities relating to disinformation and the information ecosystem both within PCH and in the government at large. A multi-pronged approach to tackling these issues requires close inter-departmental collaboration.

The DCI’s Consultative Body is the primary tool used by PCH to promote collaboration and prevent duplication of efforts between PCH and other Government departments. The Consultative Body is made up of members representing relevant areas of policy cover across the Government of Canada. Members include representatives from PCH’s International Trade Branch and Anti-Racism Secretariat, as well as other Government Departments including Global Affairs Canada, the Privy Council Office, Public Safety Canada, Women and Gender Equality Canada, Environment and Climate Change Canada, and Statistics Canada.

In an effort to promote stronger collaboration and communication, the Broadcasting, Copyright, and Creative Marketplace Branch (BCCM) will review the membership, meeting frequency, and purpose of the Consultative Body. This review will culminate in a new Terms of Reference for the Consultative Body.

In addition, BCCM will conduct a yearly review of government-wide efforts relating to online disinformation. This yearly review will aid efforts to keep track of the growing number of partners and activities in this policy space while preventing duplication of efforts. The first two yearly reports will serve as a demonstration that this new process is in place.

The Sector acknowledges the importance of clarifying existing roles and responsibilities for DCI within PCH specifically, especially with regards to the relationship between the DCI (housed BCCM) and the DOCO initiative (housed within the International Trade Branch). Both teams were created under the umbrella of Canada’s Protecting Democracy Plan in 2019 and both serve PCH’s objectives to build a healthier and more diverse online information ecosystem. Given their shared vision and objectives, these teams regularly collaborate as part of their day-to-day activities.

Since the evaluation report, the Sector has engaged in an organizational restructuring and the separation of the DCI and DOCO has been addressed. The new Digital and Creative Marketplace Frameworks branch will work to advance the sector’s legislative priorities related to broadcasting, news media remuneration, copyright and online safety. Under this branch, the DCI and the DOCO initiative will now be regrouped under the same directorate, who will also lead on the online safety file, policy and research on harmful content and disinformation as well as international engagement. These changes took effect on May 23, 2023.

Table 15: recommendation 1 – action plan
Action Plan Item Deliverable Timeline Responsible
1.1. Review the membership and format of consultative body. 1.1.1 A revised DG-approved Terms of Reference for the Consultative Body which indicates membership, frequency and roles. October 2023 Director General, Broadcasting, Copyright, and Creative Marketplace
1.2. Monitor government-wide efforts relating to online disinformation. 1.2.1 A yearly, DG-approved review of the interdepartmental environment with regard to roles and activities pertaining to online disinformation
  • First-year report: June 2023
  • Second-year report: June 2024
Director General, Broadcasting, Copyright, and Creative Marketplace
1.3 Clarify, harmonize and communicate the roles and responsibilities of the DCI and DOCO teams. 1.3.1 An ADM approved description of roles, responsibilities and organizational linkages of the DCI and DOCO teams. October 2023

Director General, International Trade Branch

Director-General, Broadcasting, Copyright, and Creative Marketplace

1.3.2 Updated description of roles, responsibilities and organizational linkages of the DCI and DOCO teams on PCH intranet. October 2023

Director General, International Trade Branch

Director-General, Broadcasting, Copyright, and Creative Marketplace

Full implementation date: June 2024

Recommendation 2

The evaluation recommends that the Senior Assistant Deputy Minister, Cultural Affairs Sector, should review, update and communicate funding priorities for DCI to ensure achievement of key results with the limited available funding.

Management response

Recommendation accepted.

The Digital Citizen Initiative has an established process for every intake period. As the mandate of the funding program is to fund research and citizen-focused activities to combat disinformation, the field of study tends to be constantly shifting and evolving.

For this reason, the program has established a Steering Committee made up of academic and civil society leaders that are consulted on new funding priorities for the DCI each time a new intake is planned. The funding priorities for each intake are determined after consulting with the Steering Committee about the emerging research needs and trends in the field of study. Based on feedback received in the evaluation, we will maintain this practice that has proven effective.

To further ensure the achievement of key results through efficient and informed identification of funding priorities, BCCM will undergo a review of the membership, meeting frequency, and mandate of the DCI’s Steering Committee.

With an eye towards better communicating funding priorities and activities to partners, BCCM will also update the Communications Plan for the Digital Citizen Initiative.

Table 16: recommendation 2 – action plan
Action Plan Item Deliverable Timeline Responsible
2.1 Review the membership and format of the steering committee 2.1.1 A revised DG-approved Terms of Reference for the Steering Committee which indicates membership, frequency of meetings and roles. The ToR will also establish the frequency of steering committee membership reviews October 2023 Director General, Broadcasting, Copyright, and Creative Marketplace
2.2 Update the Communications plan for the Digital Citizen Initiative 2.2.1 An updated, ADM-approved communications plan with updated program materials to clarify funding priorities August 2023 Senior Assistant Deputy Minister – Cultural Affairs

Full implementation date: October 2023

Recommendation 3

The evaluation recommends that the Senior Assistant Deputy Minister, Cultural Affairs Sector, should review and update the DCI Program logic model and performance measurement indicators to ensure information is available for strategic planning, decision-making and reporting.

Management response

Recommendation accepted.

The DCI logic model and performance measurement framework (which includes DOCO) were created during the early stages of developing this program. Some of the indicators rely too much on outputs rather than outcomes and have proven difficult to measure.

The DCI was renewed for an additional two years in 2022. Through the Treasury Board renewal process, the expected results for the two initiatives were updated. As part of this process, PCH committed to updating the Performance Indicator Profile for the DCI. BCCM will work on a new updated performance measurement framework for the DCI with an eye towards identifying a clear and logical theory of change and meaningful indicators that are measurable.

Table 17: recommendation 3 – action plan
Action Plan Item Deliverable Timeline Responsible
3.1 Revise the DCI performance measurement framework 3.1.1 A revised, ADM-approved Performance Information Profile and Logic Model December 2023

Director General, International Trade Branch

Director General, Broadcasting, Copyright, and Creative Marketplace

Full implementation date: December 2023

Annex A: evaluation framework

Relevance: The extent to which DCI addresses needs and its alignment with federal government priorities, roles and responsibilities as well as PCH priorities, outcomes and objectives.

Question 1.1: To what extent did the DCI address the continued, emerging and changing needs of Canadians on disinformation and online harm?
Indicator Data Collection

(Phase 1 before March 31, 2021)

  • Evidence on the extent to which the DCI addresses continued, emerging and changing needs of Canadians on disinformation and online harm.

(Phase 1 before March 31, 2021)

  • Document review
  • Administrative review
  • Literature review

(Phase 2 after March 31, 2021)

  • Views of Interviews on the extent to with the DCI at PCH addresses continued, emerging and changing needs of Canadians on disinformation and online harm.

(Phase 2 after March 31, 2021)

  • Interviews
Question 1.2: To what extent did DCI support government priorities, roles and responsibilities, including those related to equity, diversity, accessibility, and reconciliation?
Indicator Data Collection

(Phase 1 before March 31, 2021)

  • Evidence on DCI’s alignment with federal government priorities, roles and responsibilities.
  • Evidence that the program considers specific priorities for equity-deserving groups.

(Phase 1 before March 31, 2021)

  • Document review
  • Administrative review

(Phase 2 after March 31, 2021)

  • Evidence and views of Interviews on the DCI alignment with federal government priorities, roles and responsibilities.
  • Evidence and views of Interviews on barriers for equity-deserving groups.

(Phase 2 after March 31, 2021)

  • Interviews
Question 1.3: To what extent does the DCI duplicate or complement other programs?
Indicator Data Collection

(Phase 1 before March 31, 2021)

  • Evidence on the extent the DCI duplicates or complements other similar programs.

(Phase 1 before March 31, 2021)

  • Document review
  • Administrative review
  • Literature review

(Phase 2 after March 31, 2021)

  • Evidence and views of Interviews on the extent the DCI duplicates or complements other similar programs.

(Phase 2 after March 31, 2021)

  • Interviews
  • Document review

Effectiveness: The extent to which DCI has made progress toward achieving its intended outcomes.

Question 2.1: To what extent did the DCI achieve its short-term outcomes?
- Researchers are better connected and have access to information on how online disinformation affects Canadians.
- Canadians’ awareness of digital media literacy is increased through events and products.
Indicator Data Collection

(Phase 1 after March 31, 2021)

  • Evidence that the DCI made progress towards achieving its short-term outcomes:
  • Number of National Digital Media Literacy Week events held.
  • % of participants at funded events that indicate having increased awareness as a result of the events.
  • Number of research and/or data collection products produced by funding recipients related to online disinformation accessible to researchers available via a GoC website.
  • Number of networking and/or information sharing events held by the PPF Institute.
  • % of attendees who indicate that their knowledge of the issue of online disinformation increased as a result of the annual PPF Institute Conference on Online Disinformation.
  • Number of contributors and/or presenters at the annual PPF Institute Conference on Online Disinformation.
  • Number of attendees at the annual PPF Institute Conference on Online Disinformation.
  • Number of grant applications to the SSHRC-PCH Joint Initiative.
  • Number of meetings of the working group held.
  • Number of stakeholders organizations involved in the working group.

(Phase 1 after March 31, 2021)

  • Document review
  • Administrative review

(Phase 2 after March 31, 2021)

Evidence and views of Interviews that the DCI made progress towards achieving its intended short-term outcomes.

  • Researchers are better connected and have access to information on how online disinformation affects Canadians.
  • Participants are satisfied with National Digital Media Literacy Week activities and other funded activities.

(Phase 2 after March 31, 2021)

  • Interviews
  • Document review
Question 2.2: To what extent did the DCI achieve its medium-term outcomes?
- Participants are satisfied with citizen-focused activities.
- Research on online disinformation is published, made publicly accessible to Canadians and is relevant to policy-making.
- Participants are satisfied with NDMLW activities.
Various stakeholders are engaged towards adoption of guiding principles on DoCO.
Indicator Data Collection

(Phase 1 after March 31, 2021)

Evidence that DCI made progress towards achieving its medium-term outcomes:

  • % of applicants who are satisfied with citizen-focused activities.
  • Number of new research products on the scope and impact of online disinformation in the Canadian context produced by recipients with DCI funding made publicly available.
  • % of PPF Institute conference on online disinformation content that is directly relevant to GoC research interests.
  • % change in the number of grant applications to the SSHRC-PCH Joint Initiative.
  • % of participants satisfied with the NDMLW’s citizen-focused activities.
  • Number of stakeholders outside of the working group engaged towards adoption of guiding principles on DoCO.

(Phase 1 after March 31, 2021)

  • Document review
  • Administrative review
  • Literature review

(Phase 2 after March 31, 2021)

  • Evidence and views of Interviews that the DCI made progress towards achieving its intended medium-term outcomes.

(Phase 2 after March 31, 2021)

  • Interviews
  • Document review

Efficiency: The extent to which DCI has delivered its programming in an efficient manner?

Question 3.1: To what extent is the programming delivered in an efficient manner?
Indicator Data Collection

(Phase 1 after March 31, 2021)

Evidence that DCI utilizes its resources efficiently:

  • Service standards have been achieved.
  • Specific issues that impacted the delivery of the DCI.
  • Extent of PCH’s effectiveness as a lead for DCI activities with its delivery partners, committees and other governmental partners.
  • Impacts of the COVID-19 pandemic on DCI’s delivery.
  • Analysis of examples (MediaSmarts, Ryerson University, Digital Public Square).

(Phase 1 after March 31, 2021)

  • Document review
  • Administrative review
  • Financial data

(Phase 2 after March 31, 2021)

  • Views on DCI’s efficiency in utilizing its resources.

(Phase 2 after March 31, 2021)

  • Interviews

Annex B: DCI performance indicators according to expected results

Annex B: DCI performance indicators according to expected results
Expected Results Performance Indicator(s)
Short-term
Canadians’ awareness of digital media literacy is increased through events and products.
  • NDMLW events held.
  • Percentage of participants at funded events that indicate having increased awareness as a result of the events.
Researchers are better connected and have access to information on how online disinformation affects Canadians.
  • Number of research and/or data collection products produced by funding recipients related to online disinformation accessible to researchers available via a GoC website.
  • Number of networking and/or information sharing events held by the PPF Institute.
  • Percentage of attendees who indicate that their knowledge of the issue of online disinformation increased as a result of the annual PPF Institute Conference on Online Disinformation.
  • Number of contributors and/or presenters at the annual PPF Institute Conference on Online Disinformation.
  • Number of attendees at the annual PPF Institute Conference on Online Disinformation.
  • Number of grant applications to the SSHRC-PCH Joint Initiative.
  • Number of meetings of the working group held.
  • Number of stakeholders organizations involved in the working group.
Intermediate
Participants are satisfied with citizen-focused activities.
  • Percentage of participants satisfied with citizen-focused activities.
Research on online disinformation is made publicly accessible to Canadians and is relevant to government research needs.
  • Number of new research products on the scope and impact of online disinformation in the Canadian context produced by recipients with DCI funding made publicly available.
  • Percentage of PPF Institute conference on online disinformation content that is directly relevant to government of Canada research interests.
  • Percentage change in the number of grant applications to the SSHRC-PCH Joint Initiative.
Participants are satisfied with National Digital Media Literacy Week activities.
  • Percentage of participants satisfied with NDMLW’s citizen-focused activities.
Various stakeholders are engaged towards adoption of guiding principles on DoCO.
  • Number of stakeholders outside of the working group engaged towards adoption of guiding principles on DoCO.
Long-term
Canadians and the Government are better equipped to counter the effects of online disinformation.
  • Number of new research projects produced by recipients showing the reach and impact of online disinformation on Canadians and available to inform government policy.

Source: PIP – 02 Cultural Marketplace Framework – 2020-21, p. 23-24, 28-30; DCCP T&Cs, p. 2

Annex C: bibliography

©His Majesty the King in Right of Canada, represented by the Minister of Canadian Heritage, 2023
Catalogue No.: CH7-70/1-2023E-PDF
ISBN: 978-0-660-67773-6

Page details

Date modified: