Horizontal Evaluation of the Drug Safety and Effectiveness Network (DSEN) 2014-15 to 2018-19

Table of contents

• Executive summary
• Program description
  • Why was DSEN created?
  • Program structure
• Evaluation description
  • Evaluation scope and approach
• Evaluation findings
  • Question 1: Uptake of DSEN post-market evidence
  • Question 2: Timeliness
  • Question 3: Alternative models
• Conclusions and recommendations
• Management response and action plan
• Annexes
  • Annex A: DSEN logic model (December 2018)
  • Annex B: Comparative models
  • Annex C: Limitations and mitigation strategies
  • Annex D: Query process map
  • Annex E: DSEN organizational chart

Executive summary

Program context

The Drug Safety and Effectiveness Network (DSEN) was established in 2009 by Health Canada and the Canadian Institutes of Health Research (CIHR). DSEN is a network of funded research teams that conduct post-market research on drug safety and effectiveness in Canada. It also includes a training program to develop drug safety and effectiveness research capacity in Canada. Research produced by DSEN is intended to support Health Canada and other Canadian decision makers in targeted oversight and decision making related to human drug products.

DSEN has a budget of $10 million per year, $7.4 million of which is dedicated to grants and awards for the network research teams and its training component. Grants and awards funding is administered by CIHR.

Key findings

Impact and use

• Evidence points to an important need for post-market research on drug safety and effectiveness in the current, rapidly evolving drug environment. However, most decision makers found that DSEN's post-market evidence, while of high quality, was not timely for their decision-making needs, negatively affecting its relevance and usefulness.

Timeliness and alternative models

• The main roadblocks to the timely production of information were communication issues within the query process, decentralized leadership leading to key responsibilities not being carried out, availability of data, and the evolving nature of decision makers' needs.
• Other producers of post-market evidence have achieved better results in addressing decision makers' needs in a timely and relevant way. An analysis of the delivery models adopted by these producers identifies key lessons learned that support the production of relevant and timely information for decision makers. These include streamlined governance that includes decision makers and researchers at the same table, clearly established priorities and agendas, centralized knowledge translation, ready access to data, and dedicated training and capacity building.

Implementation challenges

• Part of the challenges associated with DSEN can be linked to design limitations associated with the research funding authorities obtained in 2008.
  • The funding authorities only allow CIHR to fund research through grants and not contracts, which limits the nimbleness of the network to access to post-market research outside of DSEN-granted research teams.
  • Grant funding, which is subject to the Policy on Transfer Payments, does not allow DSEN-granted researchers to participate at the governance table where strategic and priority-setting decisions are made, as it could create a real or perceived conflict of interest for the program's administration.

Recommendations

1. Consider alternative models for delivering timely and relevant evidence to decision makers, while continuing to develop the capacity-building strengths of DSEN in post-market drug safety and effectiveness.

   There is an important need for post-market drug safety and effectiveness information, and with the fast pace of innovation in drug development and approvals, this is expected to increase significantly in the next decade. Although DSEN is a valuable asset in terms of expertise, its current context and operational processes do not support the timely production of the evidence needed by decision makers. To fully support the production of timely and relevant evidence and improve the program's value, the delivery model needs to be redesigned to remove the barriers preventing the current host from implementing the flexible funding mechanism originally envisioned for the network.

2. Moving forward, the limitations of the current design should also be addressed as follows:
   1. Streamlined governance that brings researchers and policy makers to the same decision-making table, as this promotes communication and a culture focused on meeting the needs of decision makers;
   2. Firmly established priorities and research agendas, aligned with policy and decision-making needs;
   3. Centralized knowledge translation and a unified approach to project management that includes using templates of analytic plans for frequently used study designs, and standardized approaches to respond to policy maker requests; and
   4. Ready access to data through partnerships with data holders, including private industry (i.e., private insurance companies).

   Regardless of the alternative model resulting from Recommendation #1, the chosen model should be delivered with several key elements at top of mind to support its success. These elements are intended to address barriers to communication, promote centralized and responsive leadership and organizational focus, ensure the network functions to meet the needs of decision makers, and support DSEN's long-term sustainability. Putting these elements into effect is part of building a service-oriented research culture that prioritizes the needs of policy makers, whose decisions have a significant impact on clinicians and patients.

Program description

Why was DSEN created?

In 2007, Health Canada (HC) was granted authority under the Food and Consumer Safety Action Plan^{Footnote a} to develop the Drug Safety and Effectiveness Network (DSEN) program. The intent behind DSEN was to address evidence gaps in the post-market safety and effectiveness of drugs, and to build capacity to answer urgent research requests in a timely manner, in order to assess the risks associated with drugs as early as possible.

Evidence generated by DSEN was intended to enhance HC's ability to conduct ongoing assessments of drug safety risks relative to their therapeutic benefits. This would enable government authorities to determine better safety profiles for medications and implement preventive measures to reduce adverse health effects. More specifically, DSEN research activities were intended to provide direct support for the targeted oversight of drug products and decision making using the product life cycle approach to drug regulation.

DSEN-produced evidence was also intended to support other Canadian decision makers regarding public drug plan coverage and reimbursement, as well as the safe and optimal prescription and use of drugs within the Canadian health care system.

To support the development and implementation of DSEN, HC partnered with the Canadian Institutes of Health Research (CIHR). Through its expertise in funding research and networks, CIHR was seen as an asset to the capacity-building needs of an emerging DSEN. Together, they had to establish processes within the DSEN operational framework that would:

• facilitate collaborative interdisciplinary research efforts;
• identify gaps in the information needed to make evidence-based decisions and set strategic directions;
• develop an integrated and prioritized research agenda;
• provide a streamlined process to rapidly address critical research;
• facilitate the exchange of new research knowledge; and
• establish new, and strengthen existing national and international partnerships in this area of research.

Program structure

DSEN is comprised of four research teams that conduct post-market research on drug safety and effectiveness in Canada. The objectives of DSEN are to increase the amount of evidence on drug safety and effectiveness available to regulators and policy makers, and to develop the capacity within Canada to undertake high-quality post-market research in this area.

DSEN is administered jointly by CIHR and HC. CIHR administers research grant funding and acts as a central hub to coordinate the network's operations. HC, through its Office of Pharmaceuticals Management Strategies and its Marketed Health Products Directorate, provides policy oversight and ensures that DSEN research informs regulatory decision making at the federal level. Three governance committees also support DSEN activities: a Steering Committee, an Executive Working Group, and a Scientific Advisory Committee.

The total annual budget allocation for DSEN is $10 million. Table 1 details this annual planned budget.

Table 1: DSEN Annual Budget Allocation

Grants and awards	CIHR operation and accommodation costs	HC operation and accommodation costs	Total per year
$7,435,516	$1,578,141	$986,343	$10,000,000
Source: CIHR and HC

DSEN governance at a glance

Canadian Institutes of Health Research

DSEN Coordinating Office: Implements, facilitates, and coordinates DSEN operations, establishes research funding mechanisms, and promotes knowledge translation.

Health Canada

Office of Pharmaceuticals Management Strategies: Provides policy oversight and coordinates provincial and territorial (P/T) participation.

Marketed Health Products Directorate: Oversees and manages DSEN queries across the branch, informs DSEN strategic direction through its role with the DSEN Steering Committee.

DSEN steering committee

Comprised of various stakeholders and knowledge users, including from CIHR and HC, the Steering Committee provides advice on the strategic direction of DSEN and the priorities for needed research, as identified through the development of a common research agenda.

DSEN executive working group

Comprised of senior managers from CIHR and HC, as well as key partners, the Executive Working Group is mandated to facilitate horizontal management between DSEN partners via collaboration on strategic initiatives, policy direction, and the evaluation of DSEN.

DSEN Scientific Advisory Committee

Comprised of representatives from the DSEN research teams, query submitter organizations, and other stakeholders, the Scientific Advisory Committee is an informal discussion body where members work to develop and refine queries.

How DSEN works

Eligible decision makers can submit post-market drug safety and effectiveness queries to the network through the DSEN Coordinating Office. Eligible decision makers include the federal regulator, P/T drug plan managers, and organizations mandated to support federal, provincial, and territorial (F/P/T) drug-related decisions, such as the Canadian Agency for Drugs and Technologies in Health (CADTH) and the Canadian Institute for Health Information (CIHI).

Submitted queries fall within the following categories:

1. If marketed prescription drugs are safe for different patient populations.
2. How different population sub-groups respond to an approved drug.
3. Whether new drugs should be publicly insured (i.e., when they are more effective than what is already available).
4. How specific drugs should be safely and optimally prescribed and used.

If a query is determined feasible (i.e., it can be answered by a DSEN research team after analysis of potential methodology, project duration, availability of data, and costs), then one or more research projects are proposed to the submitter by the researchers. For a single query, a submitter can be offered several research projects. The submitter then approves the projects that will best answer their query, and the projects are initiated. Upon completion, results are communicated to the submitter and disseminated further as appropriate.

Overview of DSEN research teams

DSEN is comprised of four research teams that provide decision makers with an array of research methodologies and multiple options for responding to queries.

MAGIC (Methods and Applications Group for Indirect Comparisons)

• Finds evidence through reviews of published and unpublished literature, regulatory documents, etc.
• Summarizes and synthesizes evidence from relevant and related studies to identify combined results and key messages from the data.

CNODES (Canadian Network for Observational Drug Effect Studies)

• Examines the appropriate and inappropriate real-world use of drugs across and between provinces.
• Assesses the safety of prescription drugs.
• Studies the comparative effectiveness of prescription drugs.
• Has ready access to anonymous health care data on more than 100 million patients.

CAN-AIM (Canadian Network for Advanced Interdisciplinary Methods for Comparative Effectiveness Research

• Analyzes historical data on patients.
• Uses observational studies of patients over time to assess real-world safety and effectiveness of drugs.
• Integrates diverse data sources, including clinical cohorts.
• Evaluates new data sources (e.g., use of social media).

SEARCH & PREVENT (Active Surveillance and Evaluation of Adverse Reactions in Canadian Healthcare Team and Pharmacogenomics of Adverse Events National Team)

• Conduct studies of Canadian patients to identify clinical and genetic factors that put certain people at higher risk of adverse drug reactions (e.g., children, pregnant women).
• Access to Canadian hospitals (10 for children and 18 for adults).
• Access to biological samples (e.g., blood, saliva).

Evaluation description

Evaluation scope and approach

This is the second evaluation of DSEN. It was led by HC's Office of Audit and Evaluation (OAE), in collaboration with CIHR's evaluation function. DSEN was previously evaluated in 2014. The first evaluation identified a need to enhance efficiency and effectiveness, including clarifying key aspects of operations and establishing service standards for the query submission process. It also recognized a need to improve performance monitoring against expected outcomes. The Management Response and Action Plan associated with this evaluation was partially implemented as of March 31, 2019.

The first evaluation focused on assessing the design and delivery of DSEN's capacity building, coordination activities, and its achievement of immediate outcomes. This second evaluation focused on assessing the intermediate outcomes of the program, specifically if DSEN-generated evidence was used by decision makers in the pharmaceutical management system to inform their decisions, and if DSEN was seen as a source of timely and relevant information that supports decisions related to the benefits and risks of drugs in the marketplace. Refer to annex a: DSEN logic model (December 2018) for the DSEN logic model.

To help assess these goals, the evaluation was guided by the following three targeted evaluation questions, developed in consultation with the program and the DSEN Steering Committee:

1. To what extent do policy makers, regulators, CADTH, drug plan managers, and other key players in the pharmaceutical management system rely on post-market evidence and advice generated by DSEN for their decisions on key questions of drug safety and effectiveness?
2. Do DSEN's current processes and funding mechanisms meet the needs of these same players for timely and relevant post-market drug safety and effectiveness evidence and advice?
3. What alternative models or approaches could optimize DSEN's investment to best support the provision of timely and comprehensive evidence and advice on drug safety and effectiveness issues that answer the needs of policy makers, federal regulators, CADTH, and drug plan managers?

The scope of the evaluation included all DSEN activities from April 2014 to March 2019. Evaluation data was collected using various sources and methods, including:

Academic and grey literature

A scan of relevant peer-reviewed and grey literature published between 2014-15 and 2018-19 was conducted, with support from HC's Health Library Information Branch. Sixty articles were selected for inclusion.

Key informant interviews

Evaluators conducted interviews with a total of 38 key informants (17 internal to CIHR and HC, and 21 external):

• CIHR program: 6
• Governance: 9
• Decision makers and query submitters: 13
• DSEN researchers: 6
• External experts: 4

Emerging themes from interviews were identified and quantified using NVIVO qualitative analysis software.

Document and files

Program staff at CIHR and HC provided administrative and policy documentation to evaluators for review. A total of 290 internal files were reviewed.

Financial data

HC and CIHR program staff provided financial data on planned and actual program expenditures, which was validated by their respective Office of the Chief Financial Officer. This data was used to assess the extent to which funding was spent as anticipated, and to estimate the cost-per-query for DSEN-funded projects.

Comparative analysis

DSEN was compared to four international and domestic alternative models of post-market drug evidence producers. The networks used for comparison were:

• The European Network of Centres for Pharmacoepidemiology and Pharmacovigilance (ENCePP);
• The US Food and Drug Administration's (FDA) Sentinel Initiative;
• The Ontario Drug Policy Research Network (ODPRN); and
• The Institute of Health Economics.

The analysis was based on a review of public program documents, academic literature, and five key informant interviews (three with researchers and two with decision makers). See annex b: comparative models for a brief description of the comparative models.

Query process review

A qualitative and quantitative review of the DSEN query process was conducted. Using program documents, evaluators calculated the timeliness of each phase of a DSEN project during the evaluation period.

An in-depth qualitative exploration of the barriers and facilitators to the process was conducted on a sample of four DSEN queries, based on a review of documents produced in the query process and six key informant interviews.

Data collected by these various methods was analyzed by triangulation to increase the reliability and credibility of the evaluation findings and conclusions.

Still, most evaluations face constraints that may affect the validity and reliability of evaluation findings and conclusions. Annex c: limitations and mitigation strategies provides an outline of the limitations encountered during the evaluation and the mitigation strategies used to obtain findings that can be used with confidence.

Evaluation findings

What was found about the uptake of DSEN post-market evidence and advice

Regulators, policy makers, and other key players sometimes used DSEN information to inform certain drug-related decisions. Still, DSEN's impact is limited, as the information it produces is often received outside the policy or decision-making window of decision makers.

Question 1: Uptake of DSEN post-market evidence

Supporting decision makers

DSEN was created to provide decision makers in the pharmaceutical regulatory and management system with timely and useful post-market drug safety and effectiveness evidence. This evidence is meant, in turn, to inform regulatory and policy decisions related to drugs in the Canadian market.^{Endnote 1}

Increasing need for post-market evidence in Canada

The evaluation found an increasing need for post-market drug safety and effectiveness evidence in Canada.

Drugs as health care intervention

In Canada, drug expenditure is the second biggest component of health care spending, and is increasing faster than the two other major spending areas of hospitals and doctors. A large proportion of drug spending goes towards high-cost drugs used for a small number of individuals.^{Endnote 2}

In 2018, HC approved 217 drugs, including 135 generics, 4 biosimilars, and 78 new drugs, of which 40 introduced new active substances (i.e., medicinal ingredients that had never been approved for sale in Canada).^{Endnote 3} The need to monitor approved drugs throughout their life cycle for safety and effectiveness to inform the regulator's decisions is only going to increase with each new drug approved.

The rapid pace of innovation in pharmaceuticals, along with an increased focus on expedited approval pathways, such as approvals through a Notice of Compliance with Conditions^{Footnote b}, is seen by several key stakeholders as contributing to the increased need for post-market evidence. In 2018 alone, 35% of new drug submissions for new active substances were approved by HC via an expedited pathway.^{Endnote 3}

Gaps in data for some sub-populations

According to the literature, clinical trials used to approve new drugs typically exclude sub-populations, such as pregnant or nursing women, children, elderly individuals, people with complex medical histories, and people with multiple illnesses. This leaves a gap in safety and effectiveness data for these sub-populations.

Real-world evidence collected through various studies is seen as a key source of information to help fill in these gaps and determine the risks and benefits associated with pharmaceuticals. During interviews, several key informants also identified several existing evidence needs in this area.

With increased attention on the effects of pharmaceuticals on sub-groups, and accelerated spending on drugs, the need for post-market drug safety and effectiveness information by policy makers, especially drug plan managers, can be expected to grow.

Evolving government priorities

A few DSEN stakeholders also predicted that new mechanisms for the management of formularies (i.e., lists of prescription drugs that are approved to be prescribed by a particular health insurance policy, or in a specific health system or hospital) and the negotiation of drug prices, which rely on both pre- and post-market evidence, may be created as part of federal aspirations for a national pharmacare program, as announced in Budget 2019.^{Endnote 4} The same stakeholders see this as increasing the demand for strong, relevant, and timely contributions by DSEN to inform decision makers.

Uptake of DSEN information

In 2018, CIHR conducted a Utility and Impact Analysis on a stratified convenience sample of 28 DSEN research projects (out of 81 queries considered). While this methodology allows for representation of several query submitters, it does not allow for generalization of results. Still, it provides insight on the perspective of a group of query submitters. The analysis showed that 64% (n=18) of respondents reported that information obtained through their query was useful for decision making, see figure 1.

Figure 1: Usefulness of DSEN Information: Text description

The figure depicts the percentage of respondent who expressed that the DSEN information was useful (64%) and the percentage of respondent who expressed that the DSEN information was not useful (36%)

N/A	Percent
Useful	64%
Not Useful	36%

Source: CIHR Utility and Impact Analysis (2018)

Furthermore, a 2019 bibliometric analysis conducted by CIHR found that, of all DSEN-supported publications issued between 2010 and 2016, approximately 31% have had an influence beyond academia, and can be linked to a policy decision. It is worth mentioning that this number could be an underestimation of impact, as policy documents do not always quote their source studies.

Examples of uptake and use

Health Canada

The results of a HC query on the safety of incretin-based therapies, used to manage diabetes, contributed to a safety review.

A study on the epilepsy treatment levetiracetem was used by HC, in addition to evidence from other sources, to update safety information on a potential interaction with another drug.

DSEN evidence was also used to advance policy work in the area of the appropriate prescribing of opioids.

CADTH

DSEN evidence on anti-vascular endothelial growth factor drugs, typically used to treat various types of cancers, was used by CADTH to inform decisions related to using these drugs to also treat various retinal conditions.

DSEN evidence on hepatitis C treatments was used to inform the relative listing of hepatitis C therapies in formularies, including interferon-free hepatitis C therapies.

Other submitters

The Nunavut government used evidence on the treatment of latent tuberculosis to shape treatment policy.

The Public Health Agency of Canada used DSEN evidence on the Herpes Zoster vaccine to inform immunization policy in this area.

Uptake lower than expected

Considering that queries submitted to DSEN originate directly from the information needs of the regulator and policy makers, the uptake of DSEN-produced evidence by regulators and policy makers was expected to be greater than what was found.

Most interviewed decision makers identified that, while DSEN evidence was of high quality, its usefulness was primarily limited because it was delivered too late, negatively affecting its relevance. They added that evidence produced was more often used to support or complement a decision that had already been made, as opposed to providing direct support to the original decision. Consequently, their needs were rarely met.

Question 2: Timeliness of this report will further examine timeliness.

Number of queries

The number of queries received by DSEN over the evaluation period remained stable. For the period of January 2014 to March 2019, 56 queries were received, of which 45 proceeded to research. Figure 2 represents the number of queries per year and by submitting organization. While the number of queries received over the last five years is slightly higher than the number of queries reported in the previous five years, with 47 queries received and 29 proceeding to research from 2009 to 2013, this slight increase appears to be linked mostly to HC's use of the network. As shown in figure 2, queries received from HC have been stable since 2015, while queries from other decision makers have been inconsistent.

Figure 2: Total queries received by calendar year and submitter: Text description

The figure depicts the number of queries received per calendar year from 2014 to 2019. For calendar year 2019, only queries received up to the end of March are included.

N/A	2014	2015	2016	2017	2018	2019
Health Canada	4	0	9	9	6	5
CADTH	2	12	1	0	1	0
Other decision makers	0	1	1	0	4	0

Some queries did not proceed to research

Of the queries represented in figure 2, 11 queries did not proceed to research: three from CADTH (all received in 2015), and eight from HC (three received in 2014, one in 2016, two in 2017, and two in 2018). Of these eight queries, two were withdrawn because the requester found an alternative source of data, two were withdrawn due to a lack of proposals coming from research teams, two were withdrawn because the requester found little decision-making value in the proposal, and one was withdrawn due to a change in the safety signal. The reasons for withdrawal of the four other queries were not documented.

CADTH queries

CADTH, which was intended to be one of the main users of DSEN, submitted the majority of its queries in 2014 and 2015. As can be seen in figure 2, CADTH submitted only two queries to DSEN between 2016 and 2019. Furthermore, document reviews indicated that the 2018 CADTH query was initiated jointly with one of the research teams.

Reviewed documents identified that, between 2013 and 2015, CADTH received a CIHR-DSEN directed grant to provide a portal through which P/T drug plan managers could interact efficiently with DSEN.^{Endnote 5} Documents also identified that the directed grant was created when it became evident to DSEN management that, after three years of existence, there were still limitations in the interface between these managers and DSEN researchers.^{Endnote 5} The same documents further detailed that CADTH, having existing consultative mechanisms with P/T drug plan representatives, was viewed as an essential partner to act as a portal connecting these drug plan representatives to DSEN.

The years covered by this directed grant correspond to the highest usage of DSEN by CADTH. Unfortunately, it is not possible to know how many queries submitted by CADTH during this period originated from P/T drug plan needs, versus CADTH's own needs, as this level of detail was not required at the end of the grant. There was no subsequent grant issued to CADTH by DSEN. According to a few key informants, CADTH declined the renewal of the directed grant, which corresponded to a decline in the number of queries submitted to DSEN by CADTH (see figure 2).

While CADTH queries to DSEN have decreased, interview data from CADTH representatives indicates that their need for post-market drug evidence has remained constant and is projected to increase in the future. The same informants also identified that, instead of using DSEN, CADTH is now taking the path of answering its post-market evidence needs by contracting research outside of DSEN. Such contracting was said to allow CADTH to remain in control of research timelines through their own project management process. Informants also identified that this led to improved timeliness, as well as relevance and usefulness of research products, which now are better aligned with CADTH's needs.

In 2016, CADTH stopped assuming the role of linking DSEN with P/T drug plan representatives, when the directed grant was not renewed. However, some key internal informants with CIHR and HC believe that CADTH remains the portal for P/T drug plans queries to DSEN.

Queries by other decision makers

In figure 2, the six queries submitted by other decision makers since 2014 include one query from Nunavut, one from the Public Health Agency of Canada (PHAC), and four from l'Institut national de l'excellence en santé et en services sociaux (INESSS). A review of documents and interview data identified that the query from Nunavut, and the four from INESSS, were first proposed by a DSEN research team and sponsored by a decision maker, whereas the PHAC query first originated from its own needs.

Interviewed decision makers who were involved in some of these queries were impressed overall with their experience with DSEN, most particularly by the quality of the research produced and access to national data. At the same time, they also commented on how lengthy the research timelines were.

Despite the low number of queries received from other organizations, some representatives of these organizations identified in interviews that they still have a need for post-market evidence. The majority of key informants from interviewed P/T authorities highlighted that drug managers often need post-market information within the context of answering a safety signal caused by adverse reactions to a drug, and that such signals need to be answered within very short timelines (i.e., within 90 to 180 days). The same key informants stated that they did not believe that DSEN could answer them this quickly. Furthermore, one representative of a P/T authority identified that, while well aware of DSEN's work over the last 10 years, they had never considered using them, as the pace of the decisions they have to take is too rapid for DSEN.

It is worth mentioning that, as of November 2019 (following the end of the evaluation period), DSEN had received three queries from other decision makers. One was from a province and two from PHAC. The evaluation was not able to assess if these queries originated from the submitter's needs, or if they were first initiated by a DSEN research team.

Decision makers seeking evidence from other sources

All interviewed decision makers that have used DSEN in the past, other than those from HC, specified that they did not use DSEN when timeliness was key. They instead obtained post-market evidence through other sources, with many of them contracting research from experts in the field. The same decision makers specified that they were satisfied with the relevance and timeliness of the research they obtained from these other sources.

The format of the end product is a barrier to uptake

Most interviewed decision makers, as well as some subject matter experts in the field of post-market drug evidence, identified the format of DSEN knowledge products as a barrier to their uptake. More specifically, DSEN reports were perceived as too technical in nature and not easily transferable into policy decisions.

Several decision makers specified that the format of the end product often appeared to be directed more toward researcher-to-researcher communication than researcher-to-policy-maker communication. Some also reported having to assign staff to adapt DSEN research results into a product more suitable to their needs.

What was found about the timeliness and usefulness of the evidence produced

Most users found that DSEN's post-market drug safety and effectiveness evidence, while of high quality, was not timely for their decision-making needs. They also added that long timelines negatively affected the relevance of the information produced, and thus its usefulness for decision making.

The main roadblocks to the timely production of useful evidence:

• Communication issues within the query process;
• Absence of centralized project management;
• Availability of data; and
• Decentralized leadership leading to some key responsibilities not being carried out.

Question 2: Timeliness

Why timeliness is important

Within its regulatory role, HC has used DSEN primarily to gather information to respond to safety signals. A 'safety signal' is defined here as information on a new or known adverse event that may be associated with a drug.^{Endnote 6} Safety signal assessments may result in actions being taken in response their findings, and may include informing the public and health care professionals of new safety information, recommending labelling changes, and even removing a drug from the market in the most serious situations.^{Endnote 6} HC has between 90 to 180 days to make an initial decision related to a safety signal. In 2018 alone, HC received 1,091,696 post-market reports of adverse reactions and undertook 620 post-market actions related to drugs for human use.

Other intended users of DSEN, such as P/T drug plan managers and CADTH, also operate under similar time constraints when making drug-related decisions.

Reviewed documents noted that when DSEN was first designed, timeliness of evidence was identified as essential to the success of the network. The large majority of decision makers interviewed as part of this evaluation stated that, most of the time, they needed access to post-market safety and effectiveness evidence within one year or less to inform their decision-making process.

Average time for DSEN to produce post-market evidence

Over the last five years, it took, on average, 620 calendar days for DSEN to complete a research project, while 569 days is the median value. By contrast, the Ontario Drug Policy Research Network (ODPRN) can complete most of its research studies, including drug safety and effectiveness studies, in 30 to 60 days.^{Endnote 7} For more complex research projects, such as drug class reviews, they are usually completed in 203 to 238 days.^{Endnote 8} As for the US Food and Drug Administration's (FDA's) Sentinel, they are able to respond to most queries within a few months and to urgent requests in as little as a few days.^{Endnote 9} It should be noted that the length of time expected for each of these networks to conduct studies could vary.

While the evaluation did not compare if products produced by alternative models under shorter timelines were of the same quality as those produced by DSEN, it did find lines of evidence that provided insight on their usefulness and relevance for decision making. As such, a review of publically available performance documents for both ODPRN and Sentinel was conducted, as well as an interview with an ODPRN key user, who also happened to have been involved with DSEN for several years. Data collected through these lines of evidence revealed high level of relevance and usefulness of evidence produced by the alternative models.

Figure 3 presents the results of a 2018 Utility and Impact Analysis conducted by CIHR on a stratified convenience sample of 28 DSEN research projects (out of 81 queries considered). The results showed that 39% (n=11) of the projects were deemed timely by the submitter, and 61% (n=17) were not. When looking solely at HC-submitted queries, (n=16), the proportion of queries considered not timely goes up to 68% (10 out of 16).

Figure 3: Timeliness of DSEN Research: Text description

The figure depicts the percentage of respondent who expressed that the DSEN research was timely (39%) and the percentage of respondent who expressed that the DSEN research was not timely (61%)

N/A	Percent
Timely	39%
Not Timely	61%

Source: CIHR Utility and Impact Analysis (2018)

Based on a sample of DSEN research projects for which key data points collected as part of the query monitoring process were available (i.e., the query submitter's desired delivery date for the research, the agreed upon delivery date, and the date of completion of the request), about 77% of the queries received by DSEN were originally requested within a timeframe of 12 months or less. The majority of these projects (16 out of 20) were delivered one year later than the submitter's desired delivery date, on average.

Furthermore, on a sub-sample of 13 research projects, CIHR analyzed the difference between the query submitter's original desired timeframe, the negotiated research timeframe, and the final delivery, revealing that:

• 10 projects had negotiated timelines that were, on average, 432 days longer that the originally requested timeline, while three projects had negotiated shorter timelines, with an average of 73 days.
• Of the 13 projects, 11 were delivered later than the negotiated timeframe by an average of 56 days, and two were delivered earlier, one by a day and the other by 90 days.

While a portion of DSEN queries are suited for longer research timelines, such as prospective research, the majority of queries received (over 75%) are time sensitive for the decision maker.

Barriers to timeliness

Issues related to timeliness were not linked to a lack of research expertise. On the contrary, it was recognized by all users that DSEN has been successful at attracting highly respected post-market experts as researchers. Instead, barriers to producing timely and relevant post-market drug safety and effectiveness research were linked to the query process, a lack of available data, and a decentralized leadership model that has resulted in the network not carrying out key roles and responsibilities.

Communication within the query process

DSEN's query process (refer to annex d: query process map for the process map), is considered by both query submitters and researchers as hindering their direct communication. This, in turn, has created challenges for completing the query clarification stage in a timely manner, or ensuring that the query question was well understood by all.

Not every query submitted is clearly understood by researchers and not all research methodologies are well understood by the submitter. The document review provided examples of delays that could be attributed to either a misunderstanding of the query or of the proposed methodology. Having both parties in direct communication from the beginning of the process was seen by interviewed decision makers and researchers as being key to supporting clarification of the need for a query, which in turn helps with the identification of the best methodology to respond to this need, according to timelines. The DSEN Coordinating Office noted that pre-query discussions among researchers and query submitters were implemented at the end of fiscal year 2018-19. The evaluation was not in a position to verify or assess any results associated with this process.

Reviewed data appears to support views on the length of time for query clarification, as the average number of days between the initial query submission and the initiation of the research was 177 days, with 101.5 days as the median value, over the last five years. This represents a little over 28% of the overall research timelines (620 days on average). Of these 177 days, half are dedicated to clarifying the query and developing research proposals (89 days on average), and the rest are dedicated to finalizing and approving the research proposal and funding processes.

Through a query process analysis, it was identified that interrupted communications throughout the query process could be linked to all parties (query submitters, researchers, and DSEN Coordinating Office), and could explain, in part, the delays. The analysis also identified that the DSEN query process' communication practices were characterized by having several "middle men" in place between query submitters, users, and researchers. Specifically, this refers to the process of communications being relayed through both the DSEN Coordinating Office, and HC's DSEN coordination function, a function implemented by HC in addition to the epidemiological and biostatistical support they provide to assist query submitters across the Health Products and Food Branch. Some internal and external informants indicated that this often adds to the complexity of communications and creates misunderstandings that result in delays or in products that do not meet the needs of users.

It was noted that the timeliness of the query process was also negatively affected by query submitters' changing needs and staff turnover. Interview data and reviewed documents identify that, with projects lasting over several years, the needs of the query submitter often evolved, as new and unexpected information became available. In turn, these changing needs affected either the timeliness of project, as the research scope had to be modified, or the relevance of the end product, if the scope could not be changed. With longer timelines also came the challenge of staff turnover from the query submitter's side. As new employees often required additional briefing to understand the research project, this also affected its timelines.

Bypassing the query process to improve communication and timeliness

To address communication challenges, research teams and end users have been encouraged to communicate more directly at the outset of the query process. A review of five queries completed in less than 180 days demonstrated the benefit of having direct communication between the query submitter and the researcher early in the process. The review also identified that direct communication between researchers and query submitters skipped several of the early steps of the current DSEN query process, resulting in improved timeliness of the process.

Centralized project management

DSEN does not have a centralized project management system used across research teams nor accessible to the DSEN Coordinating Office and query submitters. This has hindered the network's ability to track projects accurately in order to proactively identify if they are falling behind. While each research project must have a Project Management Action Plan that tracks its milestones and identifies the critical path of the project and its protocols, the benefits of this plan appear limited. It was found that updates made to tools by researchers running a query project were not accessible by all three parties involved. This can leave the DSEN Coordinating Office or query submitter unaware that an element of the plan has changed, and in turn affects the timeliness and utility of deliverables, which have often failed to meet the original expectations of the query submitter.

While the DSEN Coordinating Office, research teams, and query submitters all conduct their own project monitoring, the lack of centralized and standardized project management practices has led to numerous instances of miscommunication and delays attributable to all three parties.

Furthermore, in the absence of centralized project management, it becomes difficult for the program to track the overall research capacity of the network, and get accurate data on research project costs. This prevents the program from being able to decide how to best allocate queries and financial resources among teams. It also prevents the program from assessing whether undertaking a query is worth the investment. This then limits CIHR and HC's ability to manage the network in a way that maximizes its value for decision makers.

Availability of data

The lack of available data to conduct research was already an issue when DSEN was first designed. Therefore, program authorities called on the shared responsibility between CIHR and HC to address this issue, and to leverage partnerships with existing data holders. Partnerships, collaborations, and agreements with data holders, such as P/T authorities and other F/P/T funded stakeholders within the post-market drug safety and effectiveness environment, were seen as essential to the success of DSEN. While recognizing that access to data is a common challenge for programs across the Health Portfolio, the evaluation found that data partnerships within DSEN were limited overall, despite the program having identified the importance of such partnerships even prior to DSEN's inception.

While data partnerships are limited, the evaluation found a promising practice that was developed in recent years. The CNODES research team received funding in 2017, through an HC query, to oversee the development and implementation of a common data model inspired by the US Sentinel initiative.^{Endnote 10} In implementing this approach, CNODES worked closely with an advisory board composed of representatives from the DSEN Coordinating Office, HC, CADTH, CIHI and P/T drug plans. This common data model is expected to facilitate the use of data coming from multiple data holdings, including P/T drug plan authorities, and will lessen the time required to gather and process large amounts of data. Information to assess success to date is not yet available.

While this model is promising, some key informants have still raised questions regarding DSEN's overall capacity to address data challenges, as it relates to access and linkage to data sets beyond the common data model. Key informants have also questioned whether the right partnerships are currently in place to support research on effectiveness.

Data is not an issue for all projects

It is also important to note that data availability is not an issue that affects all DSEN research projects. For instance, projects requiring review and meta-analysis of published literature are not affected by data access issues. Such projects have accounted for 41% of all DSEN's completed and ongoing projects since its inception. Nonetheless, when looking at completion times for these type of projects over the last five years (n=20), they took on average 510 days to complete, with a median of 521 days, and as noted earlier, such timelines are not aligned with the needs of most DSEN users.

The average time to initiate research for review and meta-analysis of published literature projects was 136 days, and accounted for 27% of the overall project timelines, which is in line with the proportion for all research projects (28%), as discussed earlier. Even when access to data is not an issue for the research project, timelines are still not aligned with the needs of DSEN users.

Decentralized leadership

DSEN was originally intended to be led by an Executive Director employed by CIHR. The Executive Director was to be responsible for the overall leadership of network operations, in accordance with the strategic direction provided by the DSEN Steering Committee.

This Executive Director was to be supported by a central office of five full-time equivalents (FTEs), for a total of six FTEs in CIHR. The office was to administer the DSEN grant program to carry out the prioritized research agenda, facilitate the building of partnerships and liaisons between funded DSEN research teams and other collaborating entities, provide a secretariat service to the Steering Committee, support knowledge translation, and facilitate network evaluations.

While the DSEN Steering Committee was established to provide strategic direction and set priorities through the development of a common research agenda, overall responsibility for DSEN operations was to be assigned to CIHR.

With respect to HC, seven FTEs were committed to support the DSEN program. HC was to provide policy oversight, coordinate P/T government participation, identify evidence and research needs to support its regulatory responsibilities and health system interests, participate in the Steering Committee, provide input into the DSEN prioritized research agenda, and incorporate evidence produced by DSEN into decision making within HC's areas of responsibility. As of March 2019, only five FTEs were responsible for DSEN activities at HC.

The DSEN organizational model within CIHR, originally to be led centrally by an Executive Director, was replaced in 2015 by a decentralized model where the Executive Director's responsibilities were redistributed among several players from various operational levels. This change was the result of organizational restructuring within CIHR. Only four of the six original intended FTEs within CIHR were dedicated to DSEN as of March 2019.

According to several key informants, including researchers, and documents reviewed, the decentralized leadership model has resulted in some key roles and responsibilities no longer being carried out by either the DSEN Coordinating Office, HC, or DSEN governance. Examples of these are priority setting, promoting P/T engagement, knowledge translation, collaboration with other research producers, and the establishment of partnerships with data holders.

The negative impact of these roles not being carried out can particularly be seen in relation to the responsibility of engaging external stakeholders to encourage the growth of the network, as queries received from organizations outside of HC have decreased overall since 2015. This also applies to the responsibility of negotiating data-sharing agreements, as data access has been slow to acquire and continues to be identified as a challenge for the network.

Funding model

With the aim of being as responsive as possible to user needs, DSEN's authorities established two streams of funding where applications are peer-reviewed:

• Centre Funding (also called Team Grant Funding): This funding was intended to provide a stable source of funding to at least four research teams, in order to create a dedicated research capacity for the DSEN program. This was to allow research teams to respond more rapidly to decision makers' queries and questions arising from the research agenda.
• Project Funding (also called Rapid Funding): This funding was intended to expand the scope of DSEN research activities beyond the DSEN-funded teams to address additional questions on prioritized research that cannot be addressed by DSEN-funded Centres.

Implementation and challenges of the funding model

Both funding streams were implemented in 2011. The first Team Grant Funding was awarded to CNODES in 2011 for five years. Meanwhile, SEARCH, PREVENT, CAN-AIM, and the three research teams who became MAGIC in 2014 were funded through the project funding stream in order to manage the level of demand for each DSEN teams' methodologies and to allow time for teams to build a track record and confirm the relevance of their work with decision makers. The MAGIC team received Team Grant Funding in September 2014. At the same time, Project funding for SEARCH, PREVENT, and CAN-AIM was renewed for 3 to 5 years. In April 2018, SEARCH and PREVENT merged and received their first Team Grant funding. CAN-AIM team received it in September 2019.

The project funding implemented by DSEN had the DSEN research team apply for research funding when queries were submitted. Interview data from most internal and external key informants revealed that Project Funding was seen as adding to the length of the process, due to the significant administrative burden associated with it. Furthermore, several interviewed researchers also believed it inhibited collaboration between teams, as they had to compete against one another for funding.

In the Project Funding model originally envisioned (i.e., open to researchers outside of DSEN teams), the query process included a step that would allow for an open call to researchers outside the DSEN research teams, in cases where the necessary capacities did not exist to answer a specific query. This step was never undertaken to support decision makers' needs for timely, relevant post-market evidence (i.e., the application process for project funding was not extended to teams not already funded by the DSEN program).

Data points collected as part of the query monitoring process (dates corresponding to the receipt of the query, its clarification, approval of research, initiation of research, etc.) do not allow for detailed analysis of the time required to process Project Funding.

Several key informants within HC have also noted significant challenges with the Team Grant Funding model. More specifically, they identified that there was no mechanism to re-allocate unused funding between research teams to meet emerging priorities, and there was no mechanism to reserve funding for future years in multi-year projects. Informants also added that this was exacerbated by the approval process being based on the chronological order of query submissions, rather than on a scientific prioritization process. They also specified that the implications of these challenges was that it was not possible to allocate funding to either projects or teams in a way that would return the best value to decision makers, especially in an environment of potentially rapidly changing needs.

Grant versus contract funding

The original business plan for DSEN^{Endnote 11} was created in 2007 and was recommended for approval by the Standing Committee on Health (HESA) in 2008.^{Endnote 12} This plan identified the contracting of centres (the word centre is used to refer to various types of research producers) as the funding mechanism for DSEN. It also identified contract management as a responsibility of the office that would be in charge of DSEN.

In the end, of the four organizations identified, and based on various factors, CIHR was determined to be the logical choice to host the network. Contract funding was therefore replaced by grant funding in the original financial authority. Grant funding only allows CIHR to fund research through grants and not contracts.

What can be learned from alternative models

There are other producers of post-market drug safety and effectiveness information in Canada that can provide relevant evidence that meets decision makers' needs in a timelier and more efficient way.

A comparative review identified key lessons from successful alternative models.

Some of these lessons cannot be fully implemented by the program in its current context.

Question 3: Alternative models

How DSEN compares

Table 2 details DSEN actual expenditures over the last five years. Out of the total planned budget of $50 million over 5 years, a total of $45.9 million was actually spent (variance of 8%). Most of this variance can be explained by the actual cost of operations, which were 40% lower than originally planned ($7.8 million instead of $12.8 million).

Table 2: DSEN actual expenditures, per year

Year	Grants and awards (G&A)	CIHR and HC operation and accommodation costs	Total program expenditures
2014-15	$8,702,821	$1,755,837	$10,458,658
2015-16	$7,978,045	$1,811,072	$9,789,117
2016-17	$7,325,569	$1,373,067	$8,698,636
2017-18	$6,693,625	$1,430,684	$8,124,309
2018-19	$7,485,516	$1,439,502	$8,925,018
Total	$38,185,576	$7,810,162	$45,995,738
Source: CIHR and HC

Table 3 presents calculations of the cost per output associated with DSEN research activities from January 2014 to March 2019. During this period, 56 new queries were received, 11 of which did not proceed to research. As of March 2019, research teams had worked on 84 completed or ongoing research projects, 23 of which were carried over from before 2014. It should be noted that a single query could result in more than one research project. Table 3 calculations also only take into account research projects that have started since January 2014, thus excluding projects carried over from previous years. Since costs per output were not being individually tracked by the program, it was not possible to retrieve the costs of carried over projects that existed prior to 2014 (i.e., 23 carried over projects). It should be noted that financial data is recorded by fiscal year, while research project data is tracked by calendar year. Thus, projects resulting from queries received as early as January 2014 were included in the calculation, even if the financial data used only covers April 2014 onward.

To provide a better picture of costs, calculations of table 3 were done on three different categories of outputs, with total program costs incurred over the last five years.

• The first row includes only projects that were completed and made available to a large audience through publication.
• The second row includes projects for which the decision makers have received results allowing a decision to be made.
• The third row takes into account all projects worked on, including those that are not completed. Cost per output of this row does not include any additional costs that might be needed to complete projects in development or underway.

The average cost per DSEN project that was completed, or where preliminary findings were communicated, is estimated at around $1,243,128, with all expenditures taken into account, including funding for the Drug Safety and Effectiveness Cross-disciplinary Training team for DSEN capacity building. Overall program results have led some key DSEN Steering Committee members to express a need to consider alternative delivery models in order to increase the program's overall impact and value.

Table 3: Cost per DSEN output, January 2014 to March 2019

Output	Number	Average cost per output (using only G&A DSEN expenditures)Tablenote a	Average cost per output (using total DSEN expenditures)Tablenote b
Research where results were publishedTablenote c	27	$1,414,281	$1,703,545
Research where results were provided to submitter (includes research published, n=27)	37	$1,032,043	$1,243,128
Research projects in development or underway (24) and for which results were communicated (n=37)	61	$625,993	$754,028
Source: CIHR and HC Tablenotes Tablenote a Total grants and awards expenditures divided by number of outputs Return to tablenote a referrer Tablenote b Total program expenditures divided by number of outputs Return to tablenote b referrer Tablenote c Published research is available to the larger audience Return to tablenote c referrer

It was not possible to find an alternative model that fully compares to DSEN. However, two other producers of drug safety and effectiveness evidence provided helpful information on products produced and the costs associated with this process. While these comparisons are not perfectly aligned, they still provide information that allows for a better understanding of DSEN overall performance and results.

• Over the last five years, the Ontario Drug Policy Research Network (ODPRN) has completed 133 drug policy reports with a total budget of approximately $7 million (about $1.4 million a year, including capacity-building activities). While ODPRN policy reports are not usually peer reviewed (although publications are peer reviewed), they undergo a research ethics board review. ODPRN products include descriptive and utilization studies, as well as reports on drug policy and evaluation, and drug safety and effectiveness.
• Over the last three years, the US Food and Drug Administration's (FDA) Sentinel Initiative, which compares more closely to the DSEN CNODES common data model, has produced over 383 Active Risk Identifications and Analyses, which are drug safety signal analyses.^{Endnote 9} The specific costs for maintaining Sentinel over this period are not available. Still, it has been reported that the costs of maintaining Sentinel have ranged from $US 12M to $US 20M per year.

Key lessons from successful alternatives

A comparative analysis was conducted, examining four alternative models of post-market drug safety and effectiveness evidence producers: ODPRN, the European Network of Centres for Pharmacoepidemiology and Pharmacovigilance (ENCePP), the Sentinel Initiative, and the Institute of Health Economics, based in Alberta. These models were selected because they were recognized by peers in the broader post-market evidence field as meeting the needs of their intended users for timely and quality research, and for improving the availability y of post-market evidence for decision makers. Annex b: comparative models provides additional information on these models, including details on the key best practices adopted by these successful models, which are listed in the following figure. The evaluation also examined whether these practices had been implemented by DSEN.

Streamlined governance that includes decision makers and researchers

Alternative models have adopted a governance model that includes researchers and policy makers sitting at the same priority-setting table. These committees offer the advantage of fostering clear communication and collaboration among key players. It was also noted that these committees have well-established roles and responsibilities that they carried out fully.

DSEN's governance

The evaluation found that the key priority-setting governance body of DSEN, the Steering Committee, has no representative from the DSEN research teams. While researchers are involved at a lower level of governance within DSEN (the Scientific Advisory Committee), they are not members of the main committee intended to set up a research agenda and priorities for the network. The Treasury Board Policy on Transfer Payments limits CIHR's ability to invite DSEN research representatives to assist in making strategic and priority-setting decisions for DSEN. This is to prevent researchers from influencing what rapid funding they can have access to, or the eventual renewal of their Team Grant funding.

DSEN's governance and leadership is currently spread across three committees: the DSEN Steering Committee, the Executive Working Group, and the Scientific Advisory Committee (see annex e: DSEN organizational chart for the DSEN organizational chart). Several key informants from CIHR and HC and from other external organizations involved in DSEN's governance, revealed confusion regarding the roles and responsibilities of these committees, leading to some key responsibilities not being carried out. This finding was also identified in the 2014 evaluation of DSEN.

An example of a key role not being carried out by DSEN governance is the development of a strategic direction and research agenda for the network, which should fall under the responsibility of the Steering Committee. While work plans have been established in recent years by the DSEN Executive Working Group, these do not provide strategic direction to the network as a whole.

Firmly established priorities

Successful alternative models were found to have firmly established strategic priorities that guide the work of the initiatives, as well as a clear research prioritization process.

DSEN's strategic priority setting and research prioritization

According to reviewed documents and interviews with internal key informants, DSEN has not developed an approach to setting strategic priorities, nor a research agenda to guide its activities. However, as stated in the authorities associated with DSEN, the Steering Committee has a key role to play in establishing a research agenda and strategic priorities for the research network.

As identified in several key informant interviews and reviewed documents, the DSEN Steering Committee has not been truly engaged in setting network priorities, nor a research agenda. Moreover, one key informant familiar with DSEN governance was of the understanding that the Steering Committee simply accepted priorities suggested by the researchers, rather than proposing an advanced research agenda in conjunction with decision makers and researchers.

Instead, the DSEN research agenda is solely based on accepting queries as they are submitted. Although a matrix was developed to prioritize queries for research, it was never implemented by the DSEN Steering Committee. Since DSEN never received more queries than they had the capacity to investigate, this prioritization was never needed. Consequently, the matrix has never been reviewed or updated to ensure that it can still meet emerging operational needs. This leaves DSEN underprepared to meet any increases in query submissions.

The evaluation found that the absence of a strategic direction and research agenda generally had a negative impact on DSEN. For example, leadership responsibilities have not been mobilized to effectively address long-standing issues that were identified as priorities in the original program authorities (e.g., data access, P/T engagement, setting a forward research agenda). This in turn has constrained the effectiveness of the network.

Dedicated knowledge translation and project management

It was noted in reviewed documents that successful post-market drug evidence providers had dedicated staff for knowledge translation.^{Endnote 13} Furthermore, most of them have ensured that their research is made publicly available in various formats.

With respect to project management, it was found that successful producers have adopted a unified project management process in order to ensure the timely delivery of quality research to decision makers. Some are also using analytic plan templates of frequently conducted study designs in order to support their researchers.^{Endnote 14}

DSEN's knowledge translation

One of the original roles envisioned for the DSEN Office of the Executive Director was to undertake knowledge translation activities that support DSEN as a whole. While the DSEN Coordinating Office delivered activities aimed at fostering knowledge translation, such as discussions between decision makers and researchers. However, several interviewed query submitters were still of the view that that the format of the research reports they received, and the information contained within them, was too technical and difficult to understand.

Knowledge translation of research findings is primarily the responsibility of each of the research teams, with the DSEN Coordinating Office issuing general guidance. However, DSECN Coordinating Office has not monitored the degree to which research teams have complied with the guidance.

CNODES created its own integrated knowledge translation approach to serve the needs of users, and a system to evaluate it, although this system does not appear to have been informed by input from decision makers.^{Endnote 15} However, there is no evidence to indicate that other DSEN research teams have taken similar approaches.

Although the Knowledge Translation Guidance Document for DSEN researchers provides a framework for promoting good knowledge translation practices, many interviewed query submitters felt that the knowledge translation practices in place were more targeted to research-to-researcher communication than research-to-policy-maker communication.^{Endnote 16},^{Endnote 17} Some decision makers even noted that it was necessary to interpret DSEN research to make it usable for developing policy.

The Guidance Document envisioned a range of summary documents and full reports, as part of a DSEN knowledge translation repository.^{Endnote 17} However, the evaluation found evidence of reported policy restrictions preventing CIHR, who is hosting the DSEN webpages, from publishing DSEN reports in addition to abstracts on its web site. This presents challenges to the broader dissemination of DSEN research, as well as restricting the visibility of DSEN.

DSEN's project management

DSEN's project management practices have been discussed earlier in the report, under question 2: timelines, barriers to timeliness (see page 17).

Ready access to data

The evaluation found that successful alternative models have an extensive number of partnerships or collaboration agreements, giving them ready access to data, including private data.

DSEN's data partnerships

As noted previously, DSEN's access to post-market data was identified as a priority at the outset of the program. The document review revealed that data-sharing partnerships were slow to form. While a DSEN asset mapping exercise conducted in 2017 identified that DSEN research teams have access and experience with a large variety of data sets, quick access to data remained an issue. As reported under question 2: timelines of this report, the majority of interviewed key informants, including researchers, identified data access as a barrier to timeliness.

Dedicated training and capacity building

Integrating a capacity-building element through a formalized training program to develop researchers in the field of post-market drug safety and effectiveness research was also identified as an important element.

DSEN's capacity building

Over the past several years, DSEN has continued to be successful at building a high level of expertise in the field of post-market drug safety and effectiveness research. In addition to its four research teams, DSEN also funds a program dedicated to capacity building: the Drug Safety and Effectiveness Cross-Disciplinary Training (DSECT) program. This program trains scientists across the areas of drug discovery, applied clinical practice, and policy, with a major focus on post-market drug safety and effectiveness. One hundred and fifty-five trainees have completed the program so far. Several interviewed key informants identified the program as an important contributor to the success of DSEN's capacity-building objective.

Considerations for implementing alternative models

Except for dedicated project management, the original DSEN program authorities outlined all of the other elements associated with the models mentioned above. These elements were considered essential for the production of timely and relevant post-market evidence for decision makers. While the program was able to implement a successful capacity-building function through funding of the DSECT program, the other elements have not been implemented as initially intended.

Furthermore, the contracting model initially outlined in the DSEN 2007 business plan^{Endnote 11} was not implemented. Instead, the financial authority sought was only for grant funding. Therefore, the DSEN model cannot be used for contracting research. While some decision makers have the flexibility to contract outside of DSEN from other providers, HC has no other means to access post-market evidence other than through DSEN research teams.

Other considerations include flexible financial administration practices that could support a nimble network that focuses on delivering value. For instance, there is currently no mechanism to re-allocate unused funding between research teams for meeting emerging priorities or funding queries that have the potential for a greater return. There is also no mechanism to reserve funding for future years for projects that may, for example, be de-prioritized in a current fiscal year, but are still of interest. Some key informants have also identified that these limitations are exacerbated by the fact that the chronological order in which queries are received and approved for research represents the order in which research begins, rather than following a prioritization scheme.

Taking into account the challenges in implementing the original vision for DSEN, and the limitations inherent to CIHR's legislative and policy framework (e.g., barriers to contracting, participation of researchers at the DSEN priority-setting table), some interviewed key informants asked if CIHR was still the best host organization for DSEN. There are currently several pan-Canadian health research organizations that are able to enter into research contracts to achieve results for their stakeholders. While not subject to Treasury Board policies, these organizations still provide public accountability for their performance and finances through the publication of annual reports, program evaluations, and the submission of grant funding reports.

It should also be noted that changing the host organization for DSEN was supported by the 2015 Unleashing Innovation report, in which one of the recommendations was to move DSEN to an organization that was better suited to aligning its work with other federal or federally funded players in the area of post-market research.^{Endnote 18}

Conclusions and recommendations

Conclusions

Impact of DSEN

DSEN has had a limited impact, as the timeliness of its research has often been outside the policy or decision-making window, thereby limiting its usefulness. These results, combined with recurrent challenges with the administration of the program, point to a need to consider alternative models of delivery in order to increase the overall impact and value of the program.

Timeliness

For the most part, DSEN has not been able to meet the needs of its users for the timely delivery of research results. In turn, this lack of timeliness has had implications on the degree to which DSEN is positioned to address the anticipated challenges of a rapidly evolving drug environment, where decision makers need to be well informed in order to make optimal decisions.

There are multiple reasons for this situation that reveal the necessity of adopting a more user-focused approach to research. There is a need to streamline the network's governance and improve its priority setting and research planning agenda in order to align the network's strategic direction with user needs. The query process also needs reviewing in order to improve communications between DSEN users and researchers. Although progress has recently been made in this area, and changes were initiated in 2018, full implementation of these changes only occurred after March 2019, and the results are not yet documented.

Design limitations

Part of the challenges associated with DSEN can be linked to design limitations related to the research funding authorities obtained back in 2008.

• The funding authorities only allow CIHR to fund research through grants and not contracts, which limits the nimbleness of the network to access to post-market research outside of the DSEN-granted research teams.
• The grant funding obtained, which is subject to the Policy on Transfer Payments, also does not allow DSEN granted researchers to participate at the governance table where strategic and priority-setting decisions are made, as it could create a real or perceived conflict of interest in the program's administration. This barrier between the two groups inhibits a shared understanding of perspectives, negatively affecting the alignment of the needs of researchers and decision makers and thus, the responsiveness of the network.

Implementation challenges

The availability of data for conducting research was already an issue when DSEN was first designed. A promising practice, 0the Common Data Model, has only been recently implemented. However, overall, data access partnerships have been slow to develop, affecting DSEN's ability to deliver timely research products.

The network's leadership was originally designed to be centralized, and the current decentralized model leaves some key roles and responsibilities unattended. Not having an established DSEN strategic direction and research agenda negatively affected the network's ability to address long-standing issues, such as P/T engagement and data access.

Furthermore, the absence of centralized project management and knowledge translation practices and protocols across the research teams, DSEN Coordinating Office, and HC also limits DSEN's capacity to monitor and adjust its research activities to respond to the needs of decision makers.

Recommendations

This evaluation has proposed two recommendations to address the current limitations of the network, with the intent of better positioning DSEN to answer its users' needs.

Management response and action plan

Second Horizontal Evaluation of the Drug Safety and Effectiveness Network (DSEN) 2014 - 2019

Recommendation 1

Consider alternative models for delivering timely and relevant evidence to decision makers, while continuing to build on the capacity building strengths of DSEN in post-market drug safety and effectiveness.

Management response

Management agrees with the recommendation. Management will identify suitable alternative models and implement the preferred option while continuing to build research capacity.

Action Plan	Deliverables	Expected Completion Date	Accountability	Resources
1.1 Identify alternative models for the DSEN (including host) and implement the preferred option.	1.1.1 and 2.1 A scoping paper of the potential options including a rationale and recommendation on a new model and associated governance structure (see recommendation 2a)	1.1.1 and 2.1 June 2020 - December 2020	1. Lead: Health Canada's (HC) Office of Pharmaceuticals Management Strategies (OPMS) Executive Director (ED) and Strategic Policy Branch (SPB) Assistant Deputy Minister (ADM) In collaboration with Canadian Institutes of Health Research (CIHR) and Health Products and Food Branch (HPFB).	1.1.1 and 2.1 Existing DSEN resources within HC and CIHR program management, with input from decision makers and researchers for 2a
	1.1.2 Development of a critical path and transition plan for implementation of the new DSEN model.	1.1.2 June 2020 - December 2020		N/A
	1.1.3 Implementation of the new model	1.1.3 January 2021 - June 2021		N/A
1.2 Identify DSEN's current investments in building DSE research capacity and determine the optimal mechanisms for supporting and sustaining DSE training and capacity	1.2.1 Undertake an analysis of current research capacity and possible future needs for DSE research in Canada (human resources and methods). Identify potential gaps.	1.2.1 June 2020 - December 2020		1.2 Existing DSEN resources in HC and CIHR with input from decision makers and researchers.
	1.2.2 Develop a DSEN Training Strategy that will monitor and support DSE-related training and capacity building needs.	1.2.2 June 2020 - December 2020

Recommendation 2

Moving forward, the limitations within the current design should also be addressed:

1. Streamlined governance that brings researchers and policy makers to the same decision-making table, as this promotes communication and a culture focused on meeting the needs of decision makers;
2. Firmly established priorities and research agendas, aligned with policy and decision-making needs;
3. Centralized knowledge translation and a unified approach to project management that includes applying templates of analytic plans for frequently used study designs, and standardized approaches to respond to policy maker requests; and
4. Ready access to data through partnerships with data holders, including private industry, such as private insurance data.

Management response

Management agrees with the recommendation. Being cognisant of the fact that the actions in response to recommendation 1 are developing a new model, this new model must also seek to incorporate the key elements outlined under recommendation 2.

Action Plan	Deliverables	Expected Completion Date	Accountability	Resources
2. Address the limitations of the current design as identified in the four streams under recommendation 2 within the context of the development and implementation of the new model under recommendation 1.	2.1 See Deliverable 1.1 above	2.1 See Deliverable 1.1 above	2. Lead: OPMS ED and SPB ADM in collaboration with CIHR and HPFB.	2.1 See Deliverable 1.1 above.
	2.2 Within the new governance structure, develop a strategic planning and prioritization forum that is responsive to the needs and priorities of decision makers.	2.2 January 2021 - June 2021		2.2 Existing DSEN resources in HC and CIHR with input from decision makers and researchers
	2.3.1 Establish a centralized knowledge translation component within the program. Review and update the existing DSEN KT guidance and develop new KT deliverables that meet the needs of decision makers.	2.3.1 January 2021 - June 2021		2.3 Existing DSEN resources in HC and CIHR with input from decision makers and researchers.
	2.3.2 Develop a revised project/query management process that is responsive to the needs of decision makers. Develop an associated performance measurement and evaluation strategy.	2.3.2 January 2021 - June 2021
	2.4 Undertake an analysis to determine existing data access by researchers and decision makers across Canada. Identify gaps and considerations. Establish data access guidelines (to optimize the rights of both researchers and decision makers to DSEN derived findings).	2.4 June 2020 - June 2021		2.4 External contract, supported by existing DSEN resources in HC and CIHR with input from decision makers and researchers

Note: Due to the extraordinary circumstances of COVID-19, which is still ongoing, it is possible that the MRAP action item timelines are further impacted. The DSEN program management (Health Canada and CIHR) will be mindful of this and respond as necessary, in consultation with the Office of Audit and Evaluation (OAE).

Annexes

Annex A: DSEN logic model (December 2018)

DSEN logic model: Text description

This figure depicts the component of the DSEN logic model, which is a visual illustration of the program's inputs (resources), activities, outputs and expected outcomes.

The inputs for the DSEN include: human resources and financial resources; expert knowledge; and existing data sources.

Using these inputs, DSEN conducts activities including the administration of the network; identifying gaps in post-market drug safety and effectiveness knowledge and generating queries; facilitating networking in post-market drug safety and effectiveness research; engaging stakeholders and partners; funding research-related activities that address identified knowledge gaps; and, DSEN promotion and communication.

Together, the outputs of the activities combine to produce:

• Finalized queries that are submitted and managed for investigation;
• Creation and sustaining of a virtual network of drug safety and effectiveness researchers;
• Undertaking and completion of research and related activities; and,
• Awareness by drug safety and effectiveness eligible query submitters of DSEN and its capacity to respond to their evidence needs.

The immediate outcomes resulting from the outputs are:

• The development and sustaining of increased capacity in post-market drug safety and effectiveness research in Canada;
• The generation and dissemination of new post-market drug safety and effectiveness knowledge; and,
• Greater coordination and collaboration to address evidence gaps.

The immediate outcomes are expected to lead to intermediate outcomes of:

• Use of DSEN generated post-market evidence to inform decisions regarding drug safety and effectiveness; and,
• Recognition of DSEN by its stakeholders as a source of trusted information to support the evaluation of benefits and risks of drugs in the marketplace.

The ultimate outcome of DSEN is for Canadians to have timely access to therapeutic products that are safe, effective and of high quality.

Annex B: Comparative models^{Endnote 19}

European Network Centres for Pharmacoepidemiology and Pharmavigilance

The European Network Centres for Pharmacoepidemiology and Pharmacovigilance (ENCePP) is a network coordinated by the European Medecines Agency. The members of this network (ENCePP partners) are public institutions, as well as contract and research organizations (CRO) involved in research in pharmacoepidemiology and pharmacovigilance. Research interests are not restricted to the safety of medicines, but may include the benefits and risks of medicines, disease epidemiology, and drug use. Participation to ENCePP is voluntary. ENCePP aims to strengthen the monitoring of the benefit-risk balance of medicinal products in Europe by:

• Facilitating the conduct of high-quality, multi-centre, and independent post-authorization studies with a focus on observational research;
• Bringing together expertise and resources in pharmacoepidemiology and pharmacovigilance across Europe, and providing a platform for collaborations; and
• Developing and maintaining methodological standards and governance principles for research.

The US Food and Drug Agency's Sentinel Initiative

Sentinel began in 2008 as a multi-year effort to create a national electronic system for monitoring the performance of FDA-regulated medical products. The Initiative is the FDA's response to a legislated requirement that the FDA work with public, academic, and private entities to develop a system to obtain existing electronic health care data from multiple sources in order to assess the safety of approved medical products. Sentinel's accomplishments include:

• The creation of a common data model and distributed data approach that enables the FDA to monitor the performance of medical products while securing and safeguarding patient privacy.
• The development of a distributed database with more than 300 million person-years of high-quality, unduplicated, and curated data.
• The development of processes for turning FDA safety concerns into queries of Sentinel distributed data that can be responded to quickly by data partners, often within weeks, in support of FDA's regulatory needs.

Ontario Drug Policy Research Network

Established in 2008, the Ontario Drug Policy Research Network (ODPRN) is a collaboration of established researchers from across Ontario. It was formed to rapidly respond to policy makers' need for relevant research to guide and inform their decisions. ODPRN has the capacity to generate scientifically sound evidence related to real-world drug use, safety, effectiveness, and costs of drugs in Ontario, and have developed partnerships that engage in cross-provincial comparisons of drug safety and use. ODPRN researchers have expertise in pharmaceutical use, outcomes, economics, and drug policy research. The goals of ODPRN are to:

• Support evidence-informed drug policy decision making through rapid research.
• Be a leader in generating high-quality, relevant, and timely research of interest to clinicians, policy makers, academics, and the general public.
• Effectively communicate findings to stakeholders.
• Provide a challenging and engaging training environment for students interested in pharmacoepidemiology and drug policy research.

Institute of Health Economics

The Institute of Health Economics is an independent, not-for-profit organization with key competencies in health economics and decision analytic modelling, health technology assessment, and knowledge transfer and exchange. Its mission is to inform coordinated, innovative, and evidence-guided health policy and practice.

The Institute has broad linkages to the academic, private, and public sectors, on provincial, national, and international levels that permit significant capacity and access to expertise, and amplify the contribution of the organization via network dissemination of evidence and information.

N/A	ODPRN	SENTINEL	ENCePP	IHE
Streamlined governance that includes decision makers and researchers	The Stakeholder Advisory Panel "serves as the broad oversight committee of the ODPRN to lead independent research activity and provide scientific and policy guidance."Endnote 20 The Panel includes: all of the network's research unit leads; several scientific advisors from outside institutions; two citizen and patient representatives; four formulary representatives, including two from the province of Ontario, and one each from the Pan-Canadian Pharmaceutical Alliance and the Non-Insured Health Benefits program of Indigenous Services Canada; and six research partners, including representatives from CADTH and CIHI.Endnote 20	The Planning Board includes representatives of the FDA, each of the Collaborating Institutions, including the investigators (researchers), as well as data holders and the patient community.Endnote 21 The Board advises Sentinel's Operations Center and the FDA, and it provides a forum for communication between the various entities.	The European model of ENCePP uses a Steering Group composed of members from academia and collaborating research centres, European Medicines Agency representatives, and other stakeholder groups, including patients and an industry observer.Endnote 22Endnote 23 The success of ENCePP is partly attributed to its strong governance model, with its diverse representation, and its use of plenary meetings to encourage the development of good practices and regulatory guidance.Endnote 24	With respect to the Institute of Health Economics, its Board of Directors includes executives from the Institute, members from academia, and representatives from government and public authorities.
Firmly Established Priorities	ODPRN's overall research agenda is aligned to respond to the strategic priorities of its main client at the provincial level (Ontario's Ministry of Health). ODPRN's research prioritization system ensures that the majority of the network's workload (up to 70%) is dedicated to projects originating from the needs of policy makers, while still leaving room for various types of research initiated by researchers.Endnote 25	The Sentinel Governance Board sets the long-term strategic direction. The FDA has also established various strategic plans for Sentinel. The most recent plan was published in January 2019, and covers activities until 2023.Endnote 26 While the main objectives of the plan are to respond to a need for post-market information, they also support innovation and the development of methodologies.	The ENCePP Steering Group has established various work plans to guide the overall development of the network and its expansion. The main goal of its 2017-19 work plan was to optimize the network's input into regulatory decision making; it also included a focus on the development of methods in impact research.Endnote 27	The IHE released a strategic plan in April 2018, covering the years 2018 to 2021.
Dedicated knowledge translation and project management	ODPRN reports are easily available online, and ODPRN has dedicated staff to create research summaries and infographics to support certain research products. Within the main Rapid Response Unit, there is a dedicated Project Management Team, comprised of project managers and research administrators dedicated to overseeing the work of all of ODPRN's various units. Analytic plan templates are used for frequently conducted study designs in order to support their researchers.Endnote 28	Sentinel publishes all reports, surveillance tools, assessments, and communications materials online, and provides a search tool to assist in finding these products.Endnote 29 Sentinel lists some of its analyses that have influenced decision making.Endnote 30 Project management is undertaken centrally by using specialized open-source project management software and wiki pages that encourage transparency and communication between the FDA and Sentinel on project advancement. Sentinel also routinely employs standardized approaches to accelerate responses to research requests.Endnote 31	ENCePP provides European Medicines Agency members with access to the EU PAS Register, which has "changed the landscape of pharmacoepidemiology and pharmacovigilance by giving public access to evaluations carried out on specific drugs and safety concerns and providing visibility on investigators, data availability, methods, and funding sources."Endnote 32	The Institute of Health Economics also has an online search function for facilitating the dissemination of its research, and has a business line dedicated to knowledge translation.Endnote 33,Endnote 34
Ready access to Data	ODPRN has a working relationship with Ontario's ICES, and benefits from agreements and partnerships with Health Quality Ontario, Public Health Ontario, and CIHI, who primarily provides access to the National Prescription Drug Utilization Information System (NPDUIS). It has additional agreements with private data sources, such as Green Shield and IQVIA, an American multinational company serving the combined industries of health information technology and clinical research. It has also used Ontario's Office of the Chief Coroner data for a variety of projects.	The Sentinel model also has access to a variety of data holdings, including large amounts of private data, through a range of partners, such as research institutions and health care providers.	ENCePP has brought together over 168 research centres (private, public, and not-for-profit) as part of its collaboration efforts, and has facilitated the establishment of research consortia that investigate drug safety.Endnote 35	IHE's strategic plan identifies that a priority will be to work with Alberta Health to expand the use of Alberta's unique Real World Evidence data assets.

Annex C: Limitations and mitigation strategies

Most evaluations face constraints that may have implications for the validity and reliability of evaluation findings and conclusions. The following table outlines the limitations encountered during the implementation of the selected methods for this evaluation and mitigation strategies put in place to ensure that the evaluation findings are sufficiently robust.

Limitation	Impact	Mitigation Strategy
There are limited peer-reviewed academic publications describing Canada-specific post-market drug safety and effectiveness evidence needs.	Limited the evaluation's ability fully assess the current and anticipated needs for post-market evidence in Canada.	Grey literature and international sources were used to supplement Canadian peer-reviewed literature.
Policy makers consult multiple data sources when making decisions. As a result, it can be difficult to attribute policy change directly to specific research projects.	Limited the ability to quantify the impacts of DSEN-funded research beyond academia.	Supplemental qualitative data was used to develop an understanding of the facilitators and barriers to uptake of research by policy decision makers.
Due to the current financial data reporting procedures at HC and CIHR, limited information on human resource (FTE) allocation was available.	Limited the ability to assess whether human resources were allocated as described in the original program authorities.	Supplemented available data on FTE allocations with interview data from senior management to determine actual source of human resource allocations.
Funding allocation by research project or team was not indicated in financial reporting.	Limited the ability to directly compare the efficiency of research teams and DSEN's cost effectiveness, relative to other research networks.	An overall calculation of cost-per-query for each team was produced, based on the total grant and contribution allocations.
Key informant interviews are retrospective in nature, providing only a recent perspective on past events.	This can affect the validity of assessments of activities or results that may have changed over time.	Triangulation with other lines of evidence substantiated or provided further information on data captured in interviews. Document review also provided corporate knowledge.
A program performance measurement strategy and logic model were implemented late in the period covered by the evaluation.	Performance measurement data was not consistently collected over the five-year period covered by the evaluation.	An evaluation framework and matrix, validated with program representatives, were developed to ensure that sufficient information was available to address all evaluation questions.
Quality of query tracking data was uneven, including an inconsistent use of terminology, and missing or incomplete data.	Thorough analysis of the various time points associated with the various research projects could not be done for all projects covered by the evaluation period.	A sampling strategy was adopted to conduct a timeliness analysis on the queries with the best data available.

Annex D: Query process map

Query process map: Text description

This figure depicts the various steps of the query process.

Step 1: Decision maker submits a Query

Step 2: Initial evaluation by the DSEN Coordinating Office is undertaken

1. If Query is acceptable, it is forwarded to the next step.
2. If Query is found to not be in scope or require additional information, it is returned to the submitter who decides to refine the Query or cancel it.

Step 3: Scientific Advisory Committee (SAC) feasibility assessment

1. If Query is feasible by DSEN team(s), it is forwarded to the next step.
2. If Query is possibly feasible, but not by DSEN teams, an open call is made to determine if the Query is answerable using CIHR funding tools.
3. If Query is found to not be in scope or require additional information, it is returned to the submitter who decides to refine the Query or cancel it.

Step 4: Query development and refining

Step 5: Formal submission of the Query

Step 6: The Query is prioritized by the Steering Committee which is informed of the results of the MCDA analysis.,

1. If the Query is answerable by one of the research teams with Team Grant funding, the Query enters into the research mode.
2. If the Query is answerable by one of the other teams, a Rapid Funding for DSEN Targeted Research grant is pursued.
   1. Proposal is peer reviewed and is evaluated by the DSEN CO (can be returned to proposal submitting team for further refinement).
      1. The proposal proceeds to research if approved.
      2. If Query is possibly feasible, but not by one of the teams, an open call is made to determine if the Query is answerable using CIHR funding tools.
      3. If funding is not approved or no proposals are received, it is returned to the submitter who decides to refine the Query or cancel it.

Step 7: Research results are communicated back to decision makers prior to other dissemination activities being undertaken

Annex E: DSEN organizational chart

DSEN organizational chart: Text description

This figure describes the organizational chart of the Drug Safety and Effectiveness Network (DSEN).

DSEN is organized under the Canadian Institutes of Health Research with its DSEN Coordinating Office (DSEN CO). The DSEN CO links horizontally to three advisory bodies: to the DSEN Steering Committee and the Science Advisory Committee on the right, and to the DSEN Executive Working Group on the left.

Under the DSEN CO there are three Collaborative Centres based on methodologies. These collaborative centres focus on Observational Studies, Prospective Studies, and Network Meta-Analysis and are formed by four DSEN funded Research Teams.

The Collaborative Centre for Observational Studies is composed of the Canadian Network for Observational Drug Effect Studies (CNODES - 95 researchers). The DSEN Active Surveillance and Evaluation of Adverse reactions in Canadian Healthcare & the Pharmaco-genomics of Adverse Events National Team (SEARCH & PREVENT - 31 researchers), and the Canadian Network for Advanced Interdisciplinary Methods for Comparative Effectiveness Research (CAN-AIM - 61 researchers), cumulatively form the Collaborating Centre for Prospective Studies. The Collaborating Centre for Network Meta-Analysis is comprised of one team: the Methods and Applications Group for Indirect Comparisons (MAGIC) (50 researchers).