Evaluation of the Evidence for Health Promotion, Chronic Disease and Injury Prevention Program: 2022

Download the alternative format
(PDF format, 898 KB, 29 pages)

Organization: Health Canada and the Public Health Agency of Canada

Published: January 2022

Table of Contents

List of Acronyms

CCHS
Canadian Community Health Survey
CIHI
Canadian Institute for Health Information
CIHR
Canadian Institutes of Health Research
CPSP
Canadian Paediatric Surveillance Program
CYP-C
Cancer in Young People in Canada
CSAR
Centre for Surveillance and Applied Research
CPHO
Chief Public Health Officer
HPOC
Health Portfolio Operations Centre
HPCDP
Health Promotion and Chronic Disease Prevention
HPCDPB
Health Promotion and Chronic Disease Prevention Branch
IMS
Incident Management System
PIP
Performance Information Profile
PHAC
Public Health Agency of Canada

Executive Summary

Program Profile

With the declaration of COVID-19 as a global pandemic by the World Health Organization in March 2020, all levels of Canadian government faced unprecedented challenges slowing the spread of disease and protecting the health and well-being of Canadians. With a leading role in the response to COVID-19, the Public Health Agency of Canada (PHAC) was tasked with improving the understanding of the wider impacts of COVID-19, incorporating a health equity approach to pandemic preparedness, response, and recovery.

The Evidence for Health Promotion, Chronic Disease and Injury Prevention Program (Evidence Program), situated within PHAC's Health Promotion and Chronic Disease Prevention Branch (HPCDPB), is an important player in understanding the wider impact of the COVID-19 pandemic on various groups.

The Evidence Program, led by the Centre for Surveillance and Applied Research (CSAR), collects and analyzes national surveillance data on maternal, child/youth, and seniors' health; chronic diseases and conditions; injuries; health inequalities; and the associated social, behavioural, and environmental determinants of health. It also carries out research to build a strong evidence base for action on public health. The Evidence Program is responsible for providing information to a broad range of stakeholders, including government senior leadership, Special Advisory Committees, national working groups, and the Canadian public.

What we found

What worked well

  • CSAR pivoted its activities to support the ongoing COVID-19 pandemic response by conducting an early and ongoing prioritization exercise adjusting work activities in the following ways:
    • Identifying projects to be continued, postponed, or discontinued based on COVID-19 evidence needs, legislative requirements, mandate commitments and Agency priorities;
    • Adapting existing surveillance tools to collect relevant data on the wider impacts of COVID-19 by broadening data sources and indicators, as well as gathering new evidence to inform how COVID-19 affected the concurrent opioid crisis
    • Working with long-standing data providers and partners to increase data reporting frequency, launch new surveys, and support new research on COVID-19 and comorbidities to address priority evidence gaps.
  • CSAR implemented a "fast track" model to improve timeliness of publication of COVID-19 related peer-reviewed articles in its HPCDP Journal. It also provided its Infobase platform in support of data dissemination related to COVID-19 cases and vaccine coverage data (with daily or weekly updates).
  • Performance measurement indicators collected as part of the Evidence Performance Information Profile (PIP) on stakeholders' access and use of COVID-19-related products were used to provide continuous insight on products improvements.
  • CSAR also supported the pandemic response through the deployment of over 28 staff members (approximately 30% of its staff), including epidemiologists and large-data analysts, for long-term assignments to support various functions of the COVID-19 response, particularly in relation to the vaccine roll-out efforts.

Challenges and considerations for future

  • There were some challenges related to rapid funding of key research. This affected the timeliness and relevance of the information provided.
  • CSAR was also affected by long-standing challenges inherent to national surveillance data production:
    • Data gaps, particularly in terms of vulnerable populations and longitudinal data; and
    • Timeliness of information, as surveillance data often becomes available only months after the events themselves, thus limiting the information available for decision makers.
  • As the program continues with its research and surveillance agenda on the wider impacts of the pandemic, it will be important to seize opportunities to increase availability and timeliness of evidence information, while linking it to policy actions that inform recovery activities, especially as it relates to vulnerable populations. Ensuring multidisciplinary exchanges with the right players to inform evidence production will be critical moving forward.
  • The program will also need to explore alternative ways in order to fill data gaps, such as models and event-based surveillance. This will help capture information that is more comprehensive and assist in providing clearer linkages between research and policy implications.

Recommendations

The findings from this evaluation have resulted in the three recommendations discussed below. However, in addition to the recommendations, the evaluation also highlighted two challenges which had already been identified in previous reviews and for which Agency-wide work is underway.Endnote 1 These include collaboration with provinces and territories and other data partners to enhance surveillance information across jurisdictions, and mechanisms to facilitate the rapid funding of critical research to fill data gaps during an emergency response. As work on addressing these Agency-wide challenges continues, integrating the findings and lessons learned from this evaluation and other pandemic-related reviews will be essential as a way to fully inform PHAC's vision for the future.

Recommendation 1: Implement consistent practices to link research with policy implications.

Using research evidence and data to guide public health decisions is an integral response in public health emergencies. While the Evidence Program has already taken steps to ensure evidence informs policy decision making, like with the PHAC-hosted Best Brains Exchange on post-COVID-19 conditions, articulating a more comprehensive public health evidence action plan linking research questions and gaps, policy implications and surveillance indicators will further assist in bridging the gap between research, policy, and practice. Continued efforts in leveraging the right expertise and collaborative opportunities, within the Agency as well as with external partners and collaborators, will guide policy responses and support policy programming priorities in managing a post-COVID-19 response. Similarly, it would be helpful to clarify further how the information produced is useful for the public, as well as for policy makers.

Recommendation 2: Explore ways to integrate alternative sources of data for evidence products to improve timeliness of information and compensate for traditional national surveillance data gaps.

Surveillance data often becomes available months after events themselves take place. To fill information gaps, circumvent these data time lags, and better support public health actions, it would be helpful for the Agency to seize opportunities to collect and integrate information by exploring alternative sources of evidence or information for decision-making. Using informal networks through event-based surveillance by scanning reports, as well as using anecdotal information and other sources of information for events allow public health officials to respond to health risks promptly rather than waiting for formal reports. In addition, moving forward, developing more coordinated systems that apply algorithms and methodologies to data as a way to anticipate threatening health events, similar to those developed for opioids modeling, could be explored.

Recommendation 3: Considering the success of Infobase during COVID-19 and its continued need, establish a plan for the future of the platform.

During the pandemic, the Evidence Program's Infobase platform quickly became the system of choice for the whole Health Portfolio to disseminate public health information to Canadians. With the rapid pace of COVID-19 related evidence and data generation, there is a need to consider how this platform will best be used in the future. While the Evidence Program has already moved forward by having some internal discussions, it would be helpful to develop a more detailed business case that outlines the role, expectations, future needs, and necessary resources moving forward.

Program Content and Description

COVID-19 Context

The emergence of COVID-19 in late December 2019, and the declaration of a global pandemic by the World Health Organization in March 2020, brought unprecedented challenges to all levels of the Canadian government in slowing the spread of disease and protecting the health and well-being of Canadians. The Public Health Agency of Canada (PHAC) provided a leading role in the management of COVID-19 preparedness, response, and recovery in ways not experienced with previous public health threats.

As identified by the Chief Public Health Officer (CPHO) in her 2020 report, while the COVID-19 pandemic has affected us all, the health impacts have been worse for some individuals in vulnerable populations, groups, and settings.Endnote 2 Within this context, understanding the wider impacts of COVID-19, and incorporating a health equity approach to pandemic preparedness, response, and recovery will continue to be an essential step.

Program Description

The Evidence for Health Promotion, Chronic Disease and Injury Prevention Program (Evidence Program), situated within PHAC's Health Promotion and Chronic Disease Prevention Branch (HPCDPB), is an important player in understanding the wider impact of the COVID-19 pandemic on various groups. The Evidence Program collects and analyzes national surveillance data on maternal, child/youth and seniors' health; chronic diseases and conditions; injuries; health inequalities; and the associated social, behavioural, and environmental determinants of health. It also carries out research to build a strong evidence base for action on public health. The Evidence Program is responsible for providing information to a broad range of stakeholders, including government senior leadership, Special Advisory Committees, national working groups, and the Canadian public.

The Evidence program is led by the Centre for Surveillance and Applied Research (CSAR), which is comprised of six divisions, including Adult Chronic Disease and Conditions; Maternal, Child and Youth Health; Behaviours, Environment, and Lifespan; Substance-related Harms; Applied Research; and Surveillance Systems and Data Management. In leading the overall Evidence Program, CSAR also collaborates with two other Centres within HPCDPB: the Centre for Health Promotion and the Centre for Chronic Disease Prevention and Health Equity.

Producing evidence in support of public health action

  • The Evidence Program collects and analyzes national surveillance data on maternal, child and youth health; seniors' health; chronic disease; injury; health inequalities; and associated social, behavioural, and environmental determinants of health. It also conducts science and intervention research to build a strong evidence base for public health action.
  • The Evidence Program provides ongoing leadership, engagement, and expert advice by collaborating with national and international partners, and contributing to the development and sharing of evidence-based knowledge products, guidelines, declarations, joint statements, and frameworks.

Disseminating evidence

  • Dissemination of evidence is done through various means including the Health Promotion and Chronic Disease Prevention in Canada: Research, Policy and Practice publication (the HPCDP Journal, a peer-reviewed scientific journal) as well as through infographics, information campaigns, data tools and data visualizations published within the Public Health Infobase.
  • Both the HPCDP Journal and Infobase are fully managed and maintained by CSAR. Infobase is a public-facing website that contains data products and visualizations on a variety of topics relevant to Health Portfolio activities and research. Infobase publications meet all government of Canada accessibility, common look and feel, and technology standards.

Adapting to the COVID-19 context

Since February 2020, the Evidence Program pivoted its activities to address the ongoing response to the COVID-19 pandemic. This included adapting existing surveillance tools to collect relevant data, as well as gathering new evidence to inform how COVID-19 affected the concurrent opioid crisis. In addition, frequent publication of COVID-19 public-facing data and data tools were required to keep the public informed of real-time trends such as COVID-19 cases, epidemiological data, tests performed, and vaccine coverage (with daily or weekly updates). Given the Infobase team's strength in developing, hosting and publishing dynamic, useable, and accessible public-facing data products, the team was engaged to build and disseminate all dynamic COVID-19 products found on Canada.ca.

Evaluation Description

Evaluation Scope

Initially, as indicated in the Departmental Evaluation Plan, the purpose of the evaluation was to review the performance of the Evidence for Health Promotion, Chronic Disease and Injury Prevention Program (Evidence Program). After some initial internal conversations with senior management, it became clear that the scope of the evaluation should focus on the surveillance activities related to COVID-19 from January 2020 to March 2021, led by the Centre for Surveillance and Applied Research (CSAR). By focusing on COVID-19 surveillance activities, the evaluation provides a unique opportunity to conduct a case study on chronic disease surveillance activities, and the wider impacts of COVID-19, within a fast-moving pandemic context. Specifically, it provides an opportunity to identify strengths and weaknesses related to this function, which in turn could offer insights to strengthen future response efforts.

Considering the limited scope period of this evaluation and its focus on COVID-19 activities, only research and surveillance activities of the Evidence Program that fell under CSAR's responsibility were reviewed. Contributions to the overall Evidence Program from the two other implicated Centres (the Centre for Health Promotion and the Centre for Chronic Disease Prevention and Health Equity), were not included in this evaluation, as they are part of PHAC's funded interventions results, and were not focused on COVID-19 at that point in time.

While not normally associated with emergency responses, CSAR and its Evidence Program had to adjust its activities to support the Agency's overall response to the pandemic and its information needs. As such, CSAR adapted its research activities and existing surveillance tools to collect and share data and evidence on the wider impacts of COVID-19 (for example, on substance use and domestic violence).

Evaluation Questions

The evaluation was designed to answer the two questions below. It used multiple lines of evidence, such as a literature and document review, performance data, key informant interviews, to ensure triangulation of findings (see Annex A for a detailed methodology, limitations and mitigation strategies).

  1. How effective has CSAR been in supporting the Agency's evidence needs during the COVID-19 pandemic?
    • How has CSAR adjusted its chronic disease work plan to address COVID-19 evidence needs? What are the impacts of the pandemic and public health response on activities related to the concurrent opioid public health crisis?
    • Are partners, such as the Canadian Institute for Health Information and Statistics Canada, providing the information needed to support development of evidence products?
    • Are processes across the Agency supporting CSAR's capacity to identify evidence needs proactively on the wider health impacts of COVID-19?
    • Does CSAR have the appropriate mechanisms to disseminate COVID-19 evidence in a timely manner, such as the Public Health Infobase?
  2. Are the performance measures included in the current Evidence Program Performance Information Profile (PIP) sufficient to address the current pandemic?
    • What is missing? Are there examples of best practices?

Findings

Question 1 – Supporting the Agency's Evidence Needs During the Covid-19 Pandemic

The Evidence Program was able to pivot its work activities quickly and produce evidence on the wider impacts of COVID-19 in support of the Agency's needs, while maintaining its core surveillance and research activities, and producing data on the concurrent opioid public health crisis. Furthermore, through its Infobase platform, CSAR also provided support to the entire Health Portfolio in the dissemination of COVID-19 related data and data tools. Observed challenges were associated with the completeness and timeliness of third-party data that often informs CSAR-produced evidence. At times, this dynamic hindered having the necessary information for key decision-making.

To assess this question, the evaluation looked at the following sub-themes to identify areas of success and challenges:

  1. Adjustment to CSAR evidence work plan and activities;
  2. Collaboration with internal and external partners; and
  3. Dissemination of evidence.

A. Adjustments to the evidence work plan and activities

Early on, and on an ongoing basis, CSAR conducted a strategic planning exercise that led to evidence work plan adjustments through the identification of projects to be continued, postponed, or discontinued, based on COVID-19 evidence needs, legislative requirements, mandate commitments, and Agency priorities.

Strategic planning exercise

At the onset of the pandemic, an early strategic planning exercise was conducted by CSAR and supported by Branch senior management. This exercise allowed the Centre to quickly identify its priority projects. The document review also showed that CSAR strategic planning was revisited on an ongoing basis to adjust to unexpected additional requests.

This early and ongoing strategic planning exercise was key to CSAR being able to focus on gathering and producing evidence on the wider impacts of the COVID-19 pandemic, such as those on mental health. At the same time, many of the Centre's essential core activities were maintained, including surveillance related to the opioid crisis.

This strategic planning exercise also allowed CSAR to free some of its resources to directly support the COVID-19 response. As such, many CSAR staff members were mobilized to long-term assignments within specific COVID-19 priority areas, including the vaccine roll-out effort and data analysis.

However, as noted by internal key informants, CSAR's limited resources in dealing with the response also meant that its core surveillance work had to be allocated to key priority areas. This meant that, based on these circumstances, other surveillance activities had to be put on hold. However, all chronic disease surveillance and research activities are critical for the health of Canadians as data shows that chronic diseases, such as cancer, cardiovascular disease, chronic lower respiratory disease, and diabetes continue to be the greatest causes of mortality in Canada.Endnote 3

CSAR also took the initiative to establish priorities and review its work plan as early as February 2020; however, this exercise was done at a time when PHAC's senior management was fully engaged in the pandemic response and had limited opportunities to provide specific guidance on COVID-19 and non-COVID-19 prioritization. A few internal key informants stated that this resulted in some ambiguities around roles and responsibilities and uncertainty around communication processes, which, in turn, created challenges in the decision-making system.

Mobilization of CSAR's skills sets in the COVID-19 response

The COVID-19 response has been an all-Agency effort, including branches like the Health Promotion and Chronic Disease Prevention Branch, which typically hasn't been directly associated with emergency responses.

Because of their unique expertise, CSAR was able to deploy over 28 employees (approximately 30%) to long-term assignments in support of various functions of the COVID-19 response, particularly in relation to the vaccine roll-out efforts. CSAR mobilized staff at various levels (from senior management to analysts) with sought-after expertise in epidemiology and large data set analysis.

Concurrent opioid crisis

Between April and June 2020, there were 1,628 people who died of apparent opioid toxicity, a 58% increase from the previous quarter, linking the pandemic to the concurrent opioid crisis.Endnote 4 The most common contributing factors noted were related to public health and safety measures, such as border closures leading to a more toxic drug supply, isolation, and reduced access to support services. Experts claimed that physical distancing was likely contributing to increases in substance use, with many using substances alone and avoiding seeking help, including emergency services.

Given the impact of the pandemic on the opioid overdose crisis, the development of CSAR's dynamic model of opioid use, overdose, and death was accelerated in the summer of 2020 to simulate overdose deaths during the COVID-19 pandemic period. This allowed for the publication in December 2020 of a predictive model of substance-related harms trends as the COVID-19 pandemic public health measures were evolving, with the aim of assisting partners and stakeholders in their intervention planning. CSAR activities on opioids and stimulant-related harms also supported the release of three joint statements from the co-chairs of the Special Advisory Committee on the Epidemic of Opioid Overdoses during the scoping of this evaluation, specifically in September 2020, December 2020 and March 2021.Endnote 5

Creation of an Evidence Synthesis Unit

In support of the Agency's COVID-19 related evidence needs, by early February 2021, CSAR was able to put in place an Evidence Synthesis Unit (within a four-month period, from original request identification, to staffing and having researchers begin work) using the "Safe Restart" fund. The goal of this unit was to undertake both rapid and systematic reviews of the large amount of incoming global research studies and data related to the wider impacts of COVID-19. This unit strived to extract key elements from the published data to inform the office of the CPHO and other areas of the Agency.

Capacity within the Evidence Synthesis Unit was limited, particularly considering the unprecedented number of COVID-19 publications within the first year of the pandemic alone. It has been estimated that in 2020, scientists published well over 100,000 articles about the coronavirus pandemic. Endnote 6 While not all the published articles fell under CSAR's purview, such as evidence related to diagnostic techniques, this unprecedented high volume of publications still put significant pressures on the review team. This also meant that, at times, it became difficult to provide the Agency and the CPHO with the latest information and data.

During the conduct of the evaluation, CSAR contracted out some of this work to universities in order to focus on a limited number of COVID-19-related reviews in house (for example, post-COVID-19 condition, family violence, food insecurity). Many of these academic institutions already had rapid and systematic review mechanisms in place, as well as greater access to surge capacity resources, to review a greater number of publications at a faster pace than the Agency. One key informant noted that, in the future, this may be an efficient way for CSAR to maximize on internal expertise and focus on other topics and to frame the emerging issues, interpret the science, and on refine approaches to translate scientific knowledge into policy and strategic decision-making.

Post-COVID-19 Conditions file

Although most people infected with COVID-19 get better within weeks of illness, some people experience post-COVID-19 conditions. The World Health Organization reported that according to data, approximately 10-20% of people experience persistent or new onset symptoms after three months following a COVID-19 infection, which occurs irrespective of initial disease severity.Endnote 7 PHAC's systematic review of the prevalence of post-COVID-19 conditions in individuals diagnosed with COVID-19 estimates that 53% reported long-term symptoms. To learn more about these new conditions, the Agency identified CSAR as the PHAC lead for this file. As such, CSAR has been leading work on systematic reviews looking at risk factors and potential effective measures to prevent these ailments, in collaboration with internal and external partners, such as with the National Microbiology Laboratory, the Infectious Diseases Prevention Branch, subject matter experts in academia, and people with lived experience. Initial findings from the systematic review were published in June 2021.Endnote 8

CSAR worked in collaboration with the Canadian Institutes of Health Research (CIHR) to organize a PHAC-hosted Best Brains Exchange on post-COVID-19 conditions. This event took place in May 2021 and brought together senior policy makers, researchers, implementation experts, and people experiencing post-COVID-19 conditions. The objectives of the event were to understand the post-COVID-19 conditions and lived experiences, identify information gaps, and gather evidence to help inform public health action. This was a valuable and early example of the Agency's recognition that there is a critical need to have a collaborative approach to establish a network of experts on post-COVID-19 condition to link research, surveillance efforts and support policy decisions for public health action within the Canadian context.

B. Working with internal and external partners

To meet PHAC's evidence needs on the wider impacts of COVID-19, CSAR had to identify the information gaps, then generate data and produce the information needed to support decisions. To do so, CSAR was able to use internal PHAC processes and work with its external data partners to proactively identify evidence needs, including potential gaps, and ensure rapid collection of key data. CSAR capitalized on its regular collaborative mechanisms to either adapt surveillance indicators, modify the frequency of data reporting, or broaden the data sources. Still, within a pandemic context, challenges did occur. These included accessing and reporting data in a timely way, and funding delays that affected the timeliness of the information obtained.

Internal collaborations

The evaluation found that from early spring 2020 and onward, CSAR worked closely with the Health Portfolio Operations Centre (HPOC), the Incident Management SystemEndnote 9 (IMS), the CPHO office, and PHAC's various branches to identify information needs related to the wider impact of the pandemic. They participated in regular meetings with contacts at the working and management levels, which allowed them to identify evidence gaps, reduce duplication of work, and continue to advance surveillance efforts. For example, CSAR worked with various branches, including the CPHO's office to identify evidence needs and adjustments to collect data that could help inform outcomes of COVID-19 on dementia, pregnancy, mental health, and diabetes.

Still, the first four to six months of the COVID-19 response included some challenges. For example, some PHAC key informants acknowledged that due to increased workloads and staff movement, some routine interdepartmental engagement processes did not proceed as usual, and this resulted in lost opportunities for CSAR to raise information needs, particularly regarding content development for national population health surveys. In addition, high staff turnover due to short rotational assignments within HPOC and IMS further compounded communication challenges. Specifically, key informants noted that it was initially unclear whom to contact within HPOC or IMS for certain internal briefings on CSAR evidence to senior management, including the CPHO and President. However, as the pandemic progressed, staff stability within the response, as well as re-normalization of processes, such as more stable workloads, were achieved and some of these challenges diminished.

It was recognized by many that CSAR was adept at making the necessary connections to proactively identify evidence needs. However, a few PHAC key informants external to CSAR, suggested that a more systematic and coordinated approach with various implicated PHAC groups (for example, related to the infections, policy, and clinical matters) could further assist in identifying research questions, data gaps and information needs that are in line with policy considerations moving forward. The implementation of such a systematic approach in a consistent manner would support the Evidence Program in its efforts to bridge the gap between science and policy interventions, by supporting evidence-informed policy decisions, particularly for vulnerable populations and the post COVID-19 condition file.

Adapting surveillance to the COVID-19 context

Over the years, CSAR has established long-standing surveillance data relationships and collaborations with many external partners, such as with the Canadian Chronic Disease Surveillance System, the Canadian Perinatal Surveillance System, the Canadian Paediatric Surveillance Program (CPSP), the Cancer in Young People in Canada program (CYP-C), and the Opioid Overdose Surveillance Task Group, just to name a few.

The evaluation found that such long-lasting relationships were key to CSAR's success in rapidly adapting its surveillance activities to collect additional data on the COVID-19 pandemic. Through these solid collaborations, CSAR was able to expand its data sources and collect evidence of the pandemic's ongoing impacts on mental health and well-being, alcohol and drug consumption, and health inequities. Such information was key to understanding the impacts of the pandemic on the Canadian population better, and to inform future recovery decisions.

For example, as of April 2020, CSAR and CPSP implemented two new data collection protocols to gather information on children's hospitalization rates associated with COVID-19 and comorbidities associated with related infections. An online reporting platform was also developed in support of these two new protocols to allow weekly access to data (usually access to data for the CPSP is monthly). This allowed the Evidence Program to provide weekly information to senior management from early on in the pandemic. A third protocol for data collection was also quickly implemented in May 2020, as cases of inflammatory syndromes like Kawasaki disease were reported around the world. In comparison, implementation of such new protocols would typically take over a year.

As early as April 2020, CSAR also modified, in collaboration with CYP-C, the protocol to capture information on COVID-19 as a complication of childhood cancer. As children living with cancer are more susceptible to COVID-19, the question was asked whether they would also be more susceptible to poor outcomes. Furthermore, to explore the potential implications of the various lockdowns in limiting access to care and implications of cancer diagnosis in youths, CSAR and CYP-C adapted reporting protocols to focus more on collecting information on diagnosis and short-term outcomes, rather than on treatment information, in order to allow faster access to the data.

To inform public health actions beyond the initial COVID-19 response phase, CSAR developed, in collaboration with its partners, a COVID-19 unintended consequences evidence roadmap. In developing this roadmap, CSAR leveraged its traditional partnerships with organizations like CIHR, CIHI, and Statistics Canada, but also made new connections with other researchers to gain access to new sets of information. The box on the next page identifies specific examples of work done with old and new partners to gain access to data on the unintended consequences of COVID-19.

As well, to further data collection on COVID-19 related impacts, CSAR, with Statistics Canada, conducted a post-COVID-19 survey between June and December 2021. This survey will provide PHAC and Statistics Canada with data on inequalities in housing, the working poor, and children in low-income families. Results will be available by sex, region, socioeconomic factors, such as income, education, employment, and occupation, as well as sociodemographic factors, such as Indigenous identity, cultural and racial background, immigrant status, rural/remote/urban geography, living arrangement, and official language minority community status. As noted by some key informants, moving forward to the post-COVID-19 recovery period, inclusion of health equity metrics will be critically important to be able to assess the social risks of future pandemics.

CSAR collaborations with data partners in accessing data on the wider impacts of COVID-19

  • Additions of COVID-19 related questions to the COMPASS prospective longitudinal system, which target youths.
  • Additions of COVID-19 related longitudinal study to the Canadian Longitudinal Study on Aging.
  • Launch of the COHESION research (June 2020), to evaluate the ongoing impacts of the pandemic on Canadians' mental health, well-being and health inequities.
  • Launch two new national surveys on mental health in September 2020.
  • Obtaining access to early vital statistics data (e.g., deaths by suicide) from Statistics Canada, since the start of the pandemic.

Still, CSAR faced challenges during the pandemic. Delays in accessing data from some key national health surveys, including the Statistics Canada-led Canadian Community Health Survey (CCHS), the Canadian Health Measures Survey, and the General Social Survey, limited CSAR's ability to identify rapidly changes related to COVID-19 on health behaviours by socioeconomic status and vulnerable populations. Such surveys are an essential component of CSAR's surveillance systems that facilitates linking chronic disease and mental health outcomes to a range of social and economic characteristics that are meaningful in identifying vulnerable populations and are critical for evidence-based decision making. For example, due to work adjustments resulting from COVID-19 restrictions, the Agency did not have access to CCHS 2020 data until November (which, in normal years, it is usually received during the summer). Furthermore, as the data collection period for the CCHS 2020 was temporarily put on hold at the beginning of the pandemic and resumed in September 2020, this translated into reduced sample sizes, which affected the amount of information retrieved and the quality of evidence produced. In addition, a few internal key informants also stated that existing data sharing protocols affected timeliness of survey data access, which in a pandemic situation, limited the Evidence Program's ability to provide timely evidence to decision makers.

As well, CSAR, and the public health community in general, lacked access to sufficient longitudinal data (collected through a series of repeated observations of the same subjects), relying instead on cross-sectional data (carried out at a single point in time). In order to understand the wider impacts of COVID-19 better and incorporate a health equity approach, longitudinal data is essential, particularly among vulnerable populations. Lack of longitudinal data is an important gap, for example, to assess physical and mental health changes from before and after the different waves of the pandemic.

CSAR was also limited in its efforts to access new sources of data quickly through processes associated with the wider impacts of COVID-19, for which funds were available under the Government of Canada "Safe Restart" initiative. There were some challenges associated with the contracting process to support the rapid undertaking of these research projects. For instance, CSAR identified an important research project intended to investigate an alarming increase (200%) of in-patient admissions to tertiary care centres due to eating disorders. Despite best efforts from CSAR and procurement services, contracting the research took time, which in a fast-moving pandemic context affected the timeliness of research implementation, and ultimately the intervention planning for this patient population. In the first ten months of the response, procurement was able to fast track COVID-19 related contracts, by assembling a special team and prioritizing these types of contracts. However, within a short period of time, COVID-19 related contracts became 80% of the total workload, limiting the unit's ability to prioritize these contracts. At the same time, there was also an increased awareness that resources needed to be managed appropriately, particularly in terms of workload, to ensure the well-being of employees. Therefore, the process for the contracting of research projects took longer than it had earlier in the response, which impacted the program's ability to provide timely information for decision making.

Addressing data gaps in surveillance

Effective surveillance relies on the production of data that can then be translated into information that is used for public health action. While surveillance data gaps have been identified in the past as persistent concerns, the pandemic has further underlined the consequences linked to some of these inherent weaknesses and has provided strong rationales to seek solutions.Endnote 10 National surveillance production often faces two key challenges which relate to data gaps that derive from differences in reporting data among provinces and territories (particularly for vulnerable populations), and timeliness of information, as data often lags six months behind from when the events occur, such as for opioids and for suicide rates.

A recent OAG report notes the 'long-standing issues in health surveillance information, including shortcomings that impeded the effective exchange of health data between the agency and the provinces and territories'.Endnote 11 In addition, the level of detail of surveillance data collected by provinces and territories varies from one jurisdiction to another, resulting in data gaps. For example, in relation to opioid surveillance, while British Columbia and Quebec report on all deaths related to illicit drugs, other provinces and territories are currently only reporting on opioid-related deaths. This affects the usefulness of having national surveillance data for stakeholders, including the CPHO, who would benefit from gaining access to consistent information to present a national picture.

According to key informants, the amalgamation of data from various secondary data sources (e.g., studies, police reports, hospital visits, surveys), causes challenges in the interpretation of data, including the validity of the findings and the accurate reporting of results. Similar challenges, were observed on studies assessing the impact of COVID-19 on mental health, where the methodologies used, in some cases, affected the national representativeness of the findings and their accuracy, through inadequate sample sizes and/or the absence of standard definitions for depression and anxiety.

In terms of wider impacts of the pandemic, key informants also noted that there were evidence gaps in the areas of family violence, child maltreatment, and the use of other substances like alcohol. Having limited data on these issues created challenges for the Evidence Program to provide senior management with the appropriate evidence to inform decisions, particularly during the pandemic. Work is underway to fill these data gaps, aiming to link research questions to policy-relevant questions.Endnote 12 The literature underlines that, in an evidence-informed system, there must be continuous links between data providers, collectors and users for surveillance data to have a maximum impact on the broader policy agenda.Endnote 13

In addition, while substantial pieces of research have included SGBA+ considerations, there are still considerable surveillance data gaps related to racialized populations, including persons of colour, and Indigenous groups. The need for robust data on these groups has been further highlighted by the disproportionate impacts of COVID-19 on specific populations.Endnote 14

Still, CSAR worked with its data partners through various means to fill data gaps and improve timeliness of information. Examples of accomplishments are included in the box below.

Examples of work done by CSAR and its data collaborators to address data gaps and improve timeliness of information:

Increased the frequency of data reporting to either weekly, monthly, or quarterly schedules; for example, for opioid-related hospitalization, data went from being reported annually to monthly.

Expanded on data sources, such as on opioids, by accessing cremation data that is available more rapidly than data derived from coroner's reports.

Developed a dynamic model of opioid use, overdose, and death rates.

The opioid modelling tool developed by CSAR was also regarded as an innovative 'prospective approach' for providing timely information to inform decisions, and as a way to link science to policy implications, which could be used to inspire work on other files.Endnote 15 Given the impact of the pandemic on the opioid overdose crisis, the usefulness of this tool was based not only on how to anticipate increases in deaths but also to advance the opportunity to consider possible measures and interventions that could be used to limit these estimated deaths. While the complexity of this issue does not lend itself to easy fixes, the model has clearly shown there is a need to investigate further the changes in patterns of substance use in response to the mental health and social effects of the pandemic. Specifically, there is a need to focus on how changes in social supports, safe access to substances, and access to services affect substance use.

As well, by applying a health equity lens to their surveillance work, the Substance Related Harms Surveillance Unit was able to explore better how the concurrent public health crises of COVID-19 and opioids were disproportionately leading to negative outcomes among racialized communities.Endnote 16 Specifically, evidence gaps were noted on the public health impact of the opioid crisis among certain Indigenous populations.

To improve further on data gaps and timeliness issues, key informants identified opportunities to integrate event-based surveillance into current tools, through a scan of reports, anecdotal information, and other sources of information, as a way to predict events that could present a serious risk to public health.

C. Dissemination Mechanisms

As a key function of the Evidence Program, CSAR shared information with health professionals, policy and program decision makers, and the public through various mechanisms, including the "Health Promotion and Chronic Disease Prevention in Canada: Research, Policy and Practice" publication (HPCDP Journal), and the Infobase platform. The evaluation found evidence of CSAR's success in their ability to disseminate COVID-19 related information in a timely manner using these two main mechanisms.

It should be noted that, while Evidence Program products are also published in international open access journals, this evaluation only focused on dissemination mechanisms that CSAR oversees.

The HPCDP Journal to "fast track" availability and dissemination of information

In line with the March 2020 Chief Science Advisor of Canada's call for open access to COVID-19 publications, the HPCDP Journal team adjusted their internal review and publication process to disseminate evidence at an accelerated pace. A "fast track" model was developed for COVID-19 related peer-reviewed articles, with the first open call for papers occurring in May 2020.

From the initial call out for scientific papers on the COVID-19 pandemic in May 2020, until the end of the period of this evaluation in March 2021, the Journal received approximately 100 submissions related to COVID-19 from all over the world. The first four articles, including two original quantitative research studies, were published in September 2020, as part of a COVID-19 rapid publication series. These articles then appeared in the regular November/December 2020 issues. As of March 2021, nine COVID-19 related articles had been published. Articles were released online first on the Journal's website, which meant that some of the steps typically included in the production process were not required, allowing access to its content sooner.

Overall, CSAR was able to review a higher number of submissions, in less time than in previous periods. While many of the submissions received were assessed as international articles which were out of scope for this Journal, or of poor scientific quality, an established COVID-19 review committee was still able to review a higher number of submissions, improving the Journal's peer review timelines from eleven weeks to five weeks.

Using Infobase to support Health Portfolio dissemination of COVID-19 related information

The Infobase platform has been operational since 2007, and is fully managed and hosted in-house by CSAR. It is unique in that it supports the delivery of timely, usable, and dynamic data products and visualizations that conform to all government accessibility and IT standards. With time-to-release schedules measured in days rather than months, it provides value to the Agency that is unmatched by other commercial or internal data platform systems. In support of the COVID-19 response, the Infobase team was heavily engaged in defining public-facing data requirements, establishing end-to-end data pipelines and data validation processes, coordinating product development across multiple internal and external stakeholders, as well as developing and hosting numerous public-facing COVID-19 data tools on behalf of the Government of Canada.

Some internal key informants identified that, at the onset of the pandemic, there was hesitancy by some groups to use Infobase as a method to disseminate COVID-19 related data and data tools. There were also concerns that its use could lead to potential duplication of work with groups who were also publishing data, such as Health Canada, Statistics Canada, and CIHI. However, in response to media criticisms and gaps in the Agency's approach to manage and disseminate COVID-19 related data in a timely way, the advantages of Infobase were recognized by the Agency and eventually led to use of the platform by the Health Portfolio at large as the system of choice to communicate COVID-19 related data. The Infobase team was also able to embed the platform into the Canada.ca pages, giving the end user a seamless experience while navigating various COVID-19 information products provided on the Canada.ca site. As such, the very first COVID-19 case count interactive map published by the Government of Canada, in March 2020, was built and hosted by the PHAC Infobase team.

Infobase currently hosts all dynamic, public-facing COVID-19 data products, including:

  • Dynamic COVID Case Count Map (showed in the box beside, with various drop-down options)
  • HPOC daily epidemiology report
  • COVIDTrends
  • COVID-19 vaccine coverage, vaccine safety
  • COVID-19 hospitalizations
  • COVID-19 international summaries
Figure 1. Dynamic COVID Case Count Map
Figure 1. Text version below.
Figure 1 - Text description

Figure 1 is an interactive data visualization map of COVID-19 cases count in Canada. A note just above the illustration of the map explains how you can 'Hover over or tap provinces and territories to see total cases, tests performed, or deaths in Canada over time. Click the play button to animate the map'

This interactive map has a drop-down menu where you can select the counts or rates, with a second drop down menu to choose the corresponding number of cases in the last 7 or 14 days, tests performed, deaths within the last 7 or 14 days, as well as total number of cases or deaths.

The total number of cases of COVID-19 in Canada is illustrated within a black circle at the top of the map. Black circles are also placed over each of the provinces and territories and display the number of cases of the specific province or territory. As the mouse hovers on then numbers for each of the provinces or territories within the black circles, these corresponding numbers are also simultaneously illustrated on a linear graph: the number of total cases on the vertical axis, and the reporting date on the horizontal axis. In the example provided in the report, the linear graph shows the number of cases for Canada with the line moving upward.

Source: Canadian Community Health Survey

As of May 2021, Infobase had a monthly user average of over two million sessions. PHAC's Infobase "Data on COVID-19 in Canada" was the thirteenth most downloaded and most visited resource on the entire Open Government Portal, with over 7,000 downloads and over 14,000 visits.

As secondary COVID-19 trends emerged, such as the exacerbation of the opioid crisis, Infobase also accelerated the publication schedule for data on opioid and stimulant-related harms in Canada. The Infobase team also managed to maintain its publication of non-COVID-19 products throughout the pandemic. As such, they oversaw the publication of over 100 non-COVID-19 related products between January 2020 and March 2021. Considering the many steps required to publish on Infobase (for example, ensuring accessibility, use of plain language, revising products, creating automation with different datasets), managing the current volume of publications created challenges for the Infobase team which is composed of seven full-time employees.

It follows that, as noted by internal key informants, the development of these COVID-19 related products also involved a great deal of collaborative work to define products with clear messages for specific audiences, determine the most appropriate design, and combine data from different sources, including verifying the numbers provided and ensuring they were drawn from the appropriate sources. An additional complication, at least in the beginning of the pandemic, was having unclear roles and responsibilities for the management of data across the Agency. This led to some duplication of efforts by multiple teams, some of whom had little or no experience in publishing dynamic public-facing data. To this day, despite Infobase having established a data pipeline process to ensure proper data management, there still seems to be gaps in understanding of this process and the value it brings.

One opportunity that may support the clarification of roles and responsibilities will be the development of a Knowledge Mobilization Strategy and Knowledge Translation Tool collaboratively with PHAC's Corporate Data and Surveillance Branch and five other federal partners.Endnote 17 This initiative will include creating a web portal on Infobase to enhance streamlining evidence dissemination for five organizations and avoid duplication of efforts, increase awareness and access to evidence products, and identify and address gaps in evidence and knowledge mobilization.

Due to the success of Infobase in disseminating clear, concise, and interactive data, CSAR is now receiving requests from departments outside of the Health Portfolio, to use the platform. Since key informants confirmed that Infobase will be used beyond COVID-19 needs, and as a potential platform for the whole government, determining the best path forward will require careful consideration in terms of resources and continuing demands that may become unsustainable in the longer term.

In addition, moving forward, while Infobase data is accessed widely (as noted above), it is unclear how internal Agency staff and other stakeholders are leveraging data products for their own needs, such as in a policy setting. For instance, web analytics taken from a survey conducted between March 4 to March 17, 2021, show that while there were 67,396 visitors to the site, most had gone on the website for 'general interest'. Several PHAC key informants noted that a better understanding of user needs, in terms of the products hosted on Infobase, may allow for more meaningful engagement with the platform. A few key informants also stated that there is a need to enhance the dissemination and presentation of evidence products to further support decision making. For instance, suggestions for improvements included having a clearer understanding of the need for the data and target audiences so that there are stronger linkages to policy decisions, as well as a way of gaining the most impact with the information presented.

Question 2 – Performance Measures in Support of Decision Making Within a Pandemic Context

CSAR's performance measurement activities integrate many best practices, and the web analytics collected by the team are a key feature that provided critical information to improve rapidly evidence products posted online, even during the pandemic.

To assess this question, the evaluation looked at what type of performance data and information was collected for COVID-19 related activities and products published by CSAR, and how this information was used for decision-making. The evaluation also assessed best practices in performance measurement to determine if elements were missing.

Collection and use of performance information

Within the context of a fast-moving pandemic, not all indicators collected though the Evidence Program's Performance Information Profile (PIP) could inform management quickly on the usefulness of their COVID-19 related evidence products. For example, citation rates take time to collect. Still, since access and use of the evidence by stakeholders are short- and medium-term outcomes of the PIP, many of the indicators in support of these outcomes were particularly useful to support rapid improvements on products, such as ongoing analysis of web analytics data.

Even during the pandemic, CSAR fully maintained all its performance measurement data collection practices, meeting performance expectations to assess the reach and use of the documents posted through its dissemination mechanisms. This continued self-assessment also allowed CSAR to provide document "owners" with key information to gage the popularity and usefulness of published information.

The CSAR performance team gathered and presented specific data sets on a weekly basis that included web analytics such as: the number of visits and visitors from various sources like the government of Canada, public computers, newsletter web links, etc. It also included time spent on a page and user behaviour patterns based on 'heat maps' that show users' curser and scroll movements on specific pages, including the clicking of specific links. Such information was also complemented by user satisfaction data.

Such performance information was identified by key informants as highly valuable in helping design or adapt the content of documents (e.g., informing the layout of the reports, tables, and infographics) in order to ensure that users access the most significant data and key messages rapidly. For instance, after the national action plan on opioids was posted online, content was modified to reflect the patterns of page scrolls by users, ensuring the most frequently 'viewed' information was moved to the top of the page. However, the increased use of Infobase by the Health Portfolio and for COVID-19 data publication has caused a knock-on effect on the workload of the performance unit, which now monitors and reports on performance related to a larger number of publications.

CSAR also maintained data collection on the efficiency of their activities, particularly in light of adaptations made in response to the pandemic, allowing them to assess the effectiveness of those adaptations and compare their results to those in the period before COVID-19. A good example of this relates to activities in support of the HPCDP Journal and their "fast track" publication model discussed earlier in this report. The benchmark data, collected before COVID-19, allowed for key comparison of performance data collected during COVID-19, providing management with solid evidence of the benefit of adopting a fast-tracking approach during the pandemic.

Best Practices in Performance Measurement

Documentation shared by the program clearly shows that not only did CSAR fully implement its Evidence Program PIP, but it also participated in an internal exercise to revise some of its indicators in 2020, demonstrating continuous improvement. The evaluation found that the measures included in the PIP were aligned with best practices as they relate to performance measurement, and there were no key elements missing.Endnote 18

Examples of good practices having been integrated to the Evidence Program PIP include a good balance of strategic and operational measures, with performance measures tracking outputs, operations, and outcomes. The adopted methodology also balanced various sources of information, integrating both quantitative and qualitative performance data from various sources. Web analytics measures are combined with the number of 'views' by evidence products. Not only did CSAR collect performance measures, but evidence also showed that they rolled up this information and shared it in a timely manner with management and the owners of documents posted, to inform decision making related to each of the products published.

The evaluation found that many internal PHAC partners had high regard for CSAR's performance measurement activities. They saw them as examples of best practices, with meaningful outcome indicators and measures. For example, PHAC's Immunization Program has worked closely with CSAR to benefit from sharing their lessons learned and ongoing improvements. This led to the integration of indicators and methodologies, like the ones adapted by CSAR, to the Immunization Program performance measurement strategy.

Conclusions

The COVID-19 response has been an all-Agency effort, including branches like the Health Promotion and Chronic Disease Prevention Branch, who typically haven't been directly associated with emergency responses. Between January 2020 and March 2021, CSAR as the lead of the Evidence Program supported PHAC's efforts by collecting data to fill existing gaps on the wider impacts of COVID-19 and providing senior management with timely information.

What worked well

By means of an early and ongoing prioritization exercise, supported by the Branch's senior management, CSAR was able to identify projects to be continued, postponed, or discontinued based on COVID-19 evidence needs, legislative requirements, mandate commitments, and Agency priorities. This exercise supported activities that focused on broadening data sources and indicators to collect data quickly on the wider impacts of COVID-19. Furthermore, long-standing relationships with data providers, health care experts and academics enabled CSAR to increase data reporting frequency, launch new surveys, leverage existing longitudinal survey platforms, and support new research on COVID-19 and comorbidities to continue to address evidence gap priorities.

CSAR was also very successful in disseminating COVID-19 related data and information to the Health Portfolio, as well as in sharing evidence with policy and program decision makers, external health professionals, and the public. Through two main vehicles, the HPCDP Journal and the Infobase platform, they used a "fast track" model to publish COVID-19 related peer-reviewed articles, and provided data products, and visualizations on COVID-19 cases, and vaccine coverage data (with daily or weekly updates), including quarterly releases on Opioid and Stimulant-Related Harms in Canada.

Many of CSAR's Evidence PIP indicators proved particularly useful in supporting dissemination efforts in the pandemic context, and in providing insights for continuous products improvements, especially with the ongoing analysis of web analytics and the rolled-up data gathered from voluntary online surveys.

CSAR also supported the pandemic response through the deployment of over 28 staff members (approximately 30% of the staff), including epidemiologists and large data analysts, to long-term assignments to support various functions with the COVID-19 response, particularly in relation to the vaccine roll-out efforts.

What were the challenges and considerations for the future

Many of the Agency's activities in this area were successful. However, the Evidence Program experienced some challenges, including delays in procuring new research. This affected the timeliness and relevance of the information. There were also long-standing challenges inherent to national surveillance data production including: 1) data gaps, particularly in terms of vulnerable populations and longitudinal data; and, 2) timeliness of information, as surveillance data often becomes available only months after the events themselves, thus limiting the information available for decision makers.

As CSAR continues with its research and surveillance agenda on the wider impacts of COVID-19, it will be important to seize opportunities to increase the availability and timeliness of evidence information, while linking it to policy actions in order to inform recovery activities, especially as they relate to vulnerable populations. Ensuring multidisciplinary exchanges with the right players to inform evidence production, similar to what was started with the Best Brains Exchange and the post-COVID-19 condition file, will continue to be critical moving forward. As well, using alternative approaches, such as models and event-based surveillance to fill data gaps, will also be an important and innovative way for presenting information in a more comprehensive fashion and a method for strengthening important linkages between research and policy implications.

Recommendations

The findings from this evaluation resulted in the three recommendations discussed below. However, in addition to these recommendations, the evaluation also highlighted two challenges which had been identified in previous reviews and for which Agency-wide work is underway.Endnote 19 These include collaboration with provinces and territories and other data partners (such as Statistics Canada) to enhance surveillance information across jurisdictions, and mechanisms to facilitate the rapid funding of critical research and surveillance activities to fill data gaps during an emergency response. As work on addressing these Agency-wide challenges continues, integrating the findings and lessons learned from this evaluation and other pandemic-related reviews will be essential as a way to fully inform PHAC's vision for the future.

Recommendation 1: Implement consistent practices to link research with policy implications.

Using research evidence and data to guide public health decisions is an integral response in public health emergencies. While the Evidence Program has already taken steps to ensure evidence informs policy decision making, like with the PHAC-hosted Best Brains Exchange on post-COVID-19 conditions, articulating a more comprehensive public health evidence action plan linking research questions and gaps, policy implications and surveillance indicators will further assist in bridging the gap between research, policy and practice. Continued efforts in leveraging the right expertise and collaborative opportunities, within the Agency as well as with external partners and collaborators, will guide policy responses and support policy programming priorities in managing a post-COVID-19 response. Similarly, it would be helpful to further clarify how the information produced is useful for the public as well as policy makers.

Recommendation 2: Explore ways to integrate alternative sources of data for evidence products to improve timeliness of information and compensate for traditional national surveillance data gaps.

Surveillance data often becomes available months after events themselves take place. To fill information gaps, circumvent these data time lags, and better support public health actions, it would be helpful for the Agency to seize opportunities to collect and integrate information by exploring alternative sources of evidence or information for decision-making. Using informal networks through event-based surveillance by scanning reports, as well using anecdotal information and other sources of information for events allow public health officials to respond to health risks promptly rather than waiting for formal reports. In addition, moving forward, developing more coordinated systems to apply algorithms and methodologies to data as it is collected as a way to anticipate threatening health events, similar to those developed for opioids modeling, could be adapted for disease modeling.

Recommendation 3: Considering the success of Infobase during COVID-19 and its continued need, establish a plan for the future of the platform.

During the pandemic, the Evidence Program's Infobase platform quickly became the system of choice for the whole Health Portfolio to disseminate public health information to Canadians. With the rapid pace of COVID-19 related evidence and data generation, there is a need to consider how this platform will best be used in the future. While the Evidence Program has already moved forward by having some internal discussions, it would be helpful to develop a more detailed business case that outlines the role, expectations, future needs and necessary resources moving forward.

Management Response and Action Plan

Recommendation 1

Implement consistent practices to link research with policy implications

Management Response

Management agrees.

  • There is a valuable opportunity for CSAR to document existing practices within the Centre that bridge research, surveillance and policy functions on key issues, including post-COVID condition, and share these within the Agency to promote such linkages for other topic areas.
  • The issue of policy integration with evidence goes beyond surveillance in CSAR, and is something of relevance Agency wide (and beyond). Given ongoing efforts led by the Corporate Data and Surveillance Branch (CDSB) and Strategic Policy Branch (SPB) to advance and coordinate whole-of-Agency activities, including through responding to recent audits, CSAR will:
    • share the results of its scan and supporting documents with Branch and Agency stakeholders, and at Tier 2 committees; and
    • continue its engagement in CDSB-led initiatives.
Management response : Management agrees.
Action Plan Deliverables Expected Completion Date Accountability Resources
The Program will conduct a scan of examples where successful linkages have been made between its evidence and policy. It will then engage with PHAC corporate surveillance and strategic policy partners to advance a knowledge mobilization work plan that collaboratively addresses horizontal considerations and common interests linked to evidence-policy integration.
  • Development of an organizational scan of successful and innovative practices in CSAR and other selected programs in the Health Portfolio that have linked evidence and policy implications
  • Development of a knowledge mobilization work plan in collaboration with horizontal partners (e.g. CDSB, SPB), based on items and trends identified in the scan for potential future collaboration.

February 2023

July 2023

Executive Director, CSAR

Executive Director, CSAR

Existing resources

Existing resources

Recommendation 2

Explore ways to integrate alternative sources of data for evidence products to improve timeliness of information and compensate for traditional national surveillance data gaps

Management Response

Management agrees.

  • Throughout the COVID-19 pandemic, CSAR has leveraged alternative processes and data sources to overcome timeliness issues and data gaps in traditional data sources. CSAR will document these experiences and share lessons learned within the Centre to begin to facilitate further use of these data sources, as appropriate.
  • There is continued need for corporate leadership and approaches to establish clear guidance and governance that supports use of alternative data. Recognizing the horizontal nature and relevance of alternative data sources Agency wide, CSAR will support CDSB in efforts by sharing its lessons learned and contributing to a whole-of-Agency approach to these matters.
Management response : Management agrees.
Action Plan Deliverables Expected Completion Date Accountability Resources
CSAR will develop a case study document that identifies how it uses alternative data sources, and then engage with Agency stakeholders to support corporate development of tools and guidance for continued and optimal use of such data sources.
  • Scan of innovative data uses through development of inventory/case study document on alternative data sources in CSAR and select Health Portfolio partners (e.g., AI/webscraping, triangulation across multiple data sources, modelling)
February 2023 Executive Director, CSAR in collaboration with Chief Data Officer, CDSB Existing resources
  • Develop and distribute or present to key stakeholders knowledge-sharing materials on CSAR's experience with use of alternative data sources, in collaboration with CDSB and other PHAC partners
August 2023 Executive Director, CSAR in collaboration with Chief Data Officer, CDSB Existing resources
  • Develop guidance and governance for surveillance program areas to appropriately use alternative data sources
January 2024 Chief Data Officer, CDSB Existing resources

Recommendation 3

Considering the success of Infobase during COVID-19 and its continued need, establish a plan for the future of the platform

Management Response

Management agrees.

Management response : Management agrees.
Action Plan Deliverables Expected Completion Date Accountability Resources
CSAR will develop a set of options to strengthen and enhance Infobase capacity and governance, in collaboration with CDSB and other key stakeholders.
  • An options analysis to strengthen and enhance Infobase capacity and governance
July 2022 Executive Director, CSAR Existing resources
  • Presentation to appropriate PHAC Tier 1 committee for decision
January 2023 Executive Director, CSAR Existing resources

Annex A – Evaluation Methodology

The evaluation reviewed the performance of the program as it relates to its COVID-19 pandemic activities from January 2020 to March 2021. Evaluation data was collected using various sources and methods, including:

Document and File Review

Program staff at PHAC provided documents for evaluators for review, including 374 internal files were reviewed.

Key Informant Interviews

Evaluators conducted interviews with 54 key informants:

  • 40 internal to PHAC
    • Within CSAR: 27 (most of whom were seen in group interviews)
    • Other PHAC areas: 13
  • 14 external collaborators:
    • Data partners: 5
    • Funded researchers: 4
    • Special Advisory Committee on the Epidemic of Opioid Overdoses: 5

Emerging themes from interviews were identified and quantified using NVIVO qualitative analysis software.

Academic and Grey Literature

A focused review of academic and grey literature was conducted to inform evaluation findings.

Performance Measurement Data

PHAC provided performance measurement data, which was analyzed to look for key trends.

Data collected by these various methods was analyzed by triangulation to increase the reliability and credibility of the evaluation findings and conclusions.

Still, most evaluations face constraints that may affect the validity and reliability of evaluation findings and conclusions. The table below outlines the limitations encountered during the implementation of the selected methods for this evaluation, and the mitigation strategies put in place to ensure that evaluation findings are sufficiently robust

Limitations encountered during the implementation of the selected methods for this evaluation
Limitation Impact Mitigation Strategy
Key informant interviews are retrospective in nature, providing only a recent perspective on past events. This can affect the validity of assessments of activities or results that may have changed over time. Triangulation with other lines of evidence substantiated or provided further information on data captured in interviews.
Key informants within CSAR had limited time for interviews during the COVID-19 response. Availability to contribute to the evaluation process were limited. Group interviews were used where possible to increase efficiency and allow access to a wider range of perspectives.
Data was collected on COVID-19 activities, as the response was still advancing and changing. This may affect the completeness of information gathered. Validation of the information collected occurred on several occasions, to ensure that to the most up-to-date information was used.
As the scope for the evaluation was limited to 15 months, data on medium and long-term outcomes associated with COVID-19 related evidence were not assessed. The evaluation could not conclude if the evidence produced resulted in policy or intervention changes. The impacts of the evidence products were only assessed in a descriptive way and for the short-term outcomes. Where possible, the evaluation complemented findings from key informant interviews with relevant performance data to assess impacts.

Endnotes

Footnote 1

Auditor General of Canada (2021). Report 8 – Pandemic Preparedness, Surveillance, and Border Control Measures https://www.oag-bvg.gc.ca/internet/English/parl_oag_202103_03_e_43785.html; and Public Health Agency of Canada, Office of Audit and Evaluation (2020). Audit of Surveillance Activities https://www.canada.ca/en/health-canada/corporate/transparency/corporate-management-reporting/internal-audits/surveillance-activities.html

Return to footnote 1 referrer

Footnote 2

Chief Public Health Officer of Canada (2020). From Risk to Resilience: AN EQUITY APPROACH to COVID-19. Chief Public Health Officer of Canada's Report on the State of Public Health in Canada 2020 Retrieved from: https://www.canada.ca/en/public-health/corporate/publications/chief-public-health-officer-reports-state-public-health-canada/from-risk-resilience-equity-approach-covid-19.html#a2.3

Return to footnote 2 referrer

Footnote 3

Statistic Canada (2021). Leading causes of death, total population, by age group. Retrieved from: https://www150.statcan.gc.ca/t1/tbl1/en/tv.action?pid=1310039401

Return to footnote 3 referrer

Footnote 4

Public Health Agency of Canada (December 2020). Joint Statement from the Co-Chairs of the Special Advisory Committee on the Epidemic of Opioid Overdoses. Retrieved from: https://www.canada.ca/en/public-health/news/2020/12/joint-statement-from-the-co-chairs-of-the-special-advisory-committee-on-the-epidemic-of-opioid-overdoses--latest-national-data-on-the-overdose-crisis.html

Return to footnote 4 referrer

Footnote 5

Public Health Agency of Canada (September 2020). Joint Statement from the Co-Chairs of the Special Advisory Committee on the Epidemic of Opioid Overdoses. Retrieved from: https://www.canada.ca/en/public-health/news/2020/09/joint-statement-from-the-co-chairs-of-the-special-advisory-committee-on-the-epidemic-of-opioid-overdoses--latest-national-opioid-related-harms-data.html
Public Health Agency of Canada (December 2020). Joint Statement from the Co-Chairs of the Special Advisory Committee on the Epidemic of Opioid Overdoses. Retrieved from: https://www.canada.ca/en/public-health/news/2020/12/joint-statement-from-the-co-chairs-of-the-special-advisory-committee-on-the-epidemic-of-opioid-overdoses--latest-national-data-on-the-overdose-crisis.html
Public Health Agency of Canada (March 2021). Joint Statement from the Co-Chairs of the Special Advisory Committee on the Epidemic of Opioid Overdoses. Retrieved from: https://www.canada.ca/en/public-health/news/2021/03/joint-statement-from-the-co-chairs-of-the-special-advisory-committee-on-the-epidemic-of-opioid-overdoses--latest-national-data-on-the-overdose-crisis.html

Return to footnote 5 referrer

Footnote 6

Else, H. (2020). How a torrent of COVID science changed research publishing — in seven charts, Nature, December 2020. Retrieved from: https://www.nature.com/articles/d41586-020-03564-y

Return to footnote 6 referrer

Footnote 7

World Health Organization (2021). A clinical case definition of post COVID-19 condition by a Delphi consensus, 6 October 2021. Retrieved from: https://www.who.int/publications/i/item/WHO-2019-nCoV-Post_COVID-19_condition-Clinical_case_definition-2021.1

Return to footnote 7 referrer

Footnote 8

Prevalence of long-term effects in individuals diagnosed with COVID-19: a living systematic review https://www.medrxiv.org/content/10.1101/2021.06.03.21258317v1

Return to footnote 8 referrer

Footnote 9

During a public health event an Incident Management System is implemented to manage and coordinate the response. The acronym IMS is used by PHAC to refer to both the Incident Management System, and Incident Management Structure supporting the system.

Return to footnote 9 referrer

Footnote 10

Auditor General of Canada (2021). Report 8 – Pandemic Preparedness, Surveillance, and Border Control Measures. Retrieved from: https://www.oag-bvg.gc.ca/internet/English/parl_oag_202103_03_e_43785.html; and Public Health Agency of Canada, Office of Audit and Evaluation (2020). Audit of Surveillance Activities https://www.canada.ca/en/health-canada/corporate/transparency/corporate-management-reporting/internal-audits/surveillance-activities.html

Return to footnote 10 referrer

Footnote 11

Auditor General of Canada (2021). Report 8 – Pandemic Preparedness, Surveillance, and Border Control Measures. Retrieved from: https://www.oag-bvg.gc.ca/internet/English/parl_oag_202103_03_e_43785.html

Return to footnote 11 referrer

Footnote 12

Clancy, C.M., Glied, S.A., Lurie, N,. (2012). From Research to Health Policy Impact, Health Services Research; Retrieved from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3393010/

Return to footnote 12 referrer

Footnote 13

Stachenko, S., (2008). The role of surveillance and data use in the development of public health policies. IUHPE – PROMOTION & EDUCATION VOL. XV, NO. 3 2008. Retrieved from : https://journals.sagepub.com/doi/pdf/10.1177/1025382308095654

Return to footnote 13 referrer

Footnote 14

Chief Public Health Officer of Canada (2020). From Risk to Resilience: AN EQUITY APPROACH to COVID-19. Chief Public Health Officer of Canada's Report on the State of Public Health in Canada 2020 Retrieved from: https://www.canada.ca/en/public-health/corporate/publications/chief-public-health-officer-reports-state-public-health-canada/from-risk-resilience-equity-approach-covid-19.html#a2.3

Return to footnote 14 referrer

Footnote 15

The opioids modeling work was possible as data on this event had been collected over a number of years, allowing for prediction of prospective data patterns and events. Not all events can be modeled using this approach as they may not have clear 'reproduction rates' and patterns. Establishing predictive modeling for suicide rates, for example, is very difficult.

Return to footnote 15 referrer

Footnote 16

Ontario Drug Policy Research Network, Office of the Chief Coroner for Ontario/Ontario Forensic Pathology Service, Public Health Ontario and Centre on Drug Policy Evaluation (2020). Preliminary Patterns in Circumstances Surrounding Opioid-Related Deaths in Ontario during the COVID-19 Pandemic. Retrieved from: https://www.publichealthontario.ca/-/media/documents/o/2020/opioid-mortality-covid-surveillance-report.pdf?la=en

Return to footnote 16 referrer

Footnote 17

The Data Pillar is a group of partner organizations comprising of: Public Health Agency of Canada (PHAC); Health Canada (HC); Canadian Institutes of Health Research (CIHR); Statistics Canada (STC); Canadian Institute for Health Information (CIHI); Indigenous Services Canada (ISC).

Return to footnote 17 referrer

Footnote 18

De Vries. M., S. (2010). Performance measurement and the search for best practices. International Review of Administrative Science, Volume: 76, Issue:2. Retrieved form: https://journals.sagepub.com/doi/abs/10.1177/0020852309365668

Return to footnote 18 referrer

Footnote 19

Auditor General of Canada (2021). Report 8 – Pandemic Preparedness, Surveillance, and Border Control Measures
https://www.oag-bvg.gc.ca/internet/English/parl_oag_202103_03_e_43785.html; and
Public Health Agency of Canada, Office of Audit and Evaluation (2020). Audit of Surveillance Activities https://www.canada.ca/en/health-canada/corporate/transparency/corporate-management-reporting/internal-audits/surveillance-activities.html

Return to footnote 19 referrer

Page details

Date modified: