Evaluation of the Health Canada Canadian Blood Services Contribution Programs 2013-14 to 2016-17

Final Report
Office of Audit and Evaluation
Health Canada and the Public Health Agency of Canada

February 2018

Table of Contents

• Executive Summary
• Management Response and Action Plan
• 1.0 Evaluation Purpose
• 2.0 Program Description
  • 2.1 Organ and Tissue Donation and Transplantation (OTDT) Program
  • 2.2 Blood Research and Development (R&D) Program
• 3.0 Evaluation Description
  • 3.1 Evaluation Scope, Approach and Design
  • 3.2 Limitations and Mitigation Strategies
• 4.0 Findings
  • 4.1 Relevance: Issue #1 – Continued Need for the Programs
  • 4.2 Relevance: Issue #2 – Alignment with Government Priorities
  • 4.3 Relevance: Issue #3 – Alignment with Federal Roles and Responsibilities
  • 4.4 Performance: Issue #4 – Achievement of Expected Outcomes
  • 4.5 Performance: Issue #5 – Demonstration of Economy and Efficiency
• 5.0 Conclusions
  • 5.1 Relevance
  • 5.2 Performance
• 6.0 Recommendations
• Appendix 1 – Logic Models
• Appendix 2 – Evaluation Description

List of Tables

• Table 1: Limitations and Mitigation Strategies
• Table 2: Proportion of Survey Respondents Rating CBS’ OTDT Activities as Important
• Table 3: Awareness of OTDT Leading Practices
• Table 4: Use of OTDT Leading Practices
• Table 5: Variance between Planned Spending and Expenditures - OTDT Program
• Table 6: Allocation of Expenditures - OTDT Program
• Table 7: Variance between Planned Spending and Expenditures - Blood R&D Program
• Table 8: Allocation of Expenditures - Blood R&D Program

List of Figures

• Figure 1: Transplants vs. People Waiting for Transplants in Canada, 2006-2015
• Figure 2: CBS OTDT Program Logic Model
• Figure 3: CBS Blood R&D Program Logic Model

List of Acronyms

BGTD

Biologics and Genetic Therapies Directorate

CBS

Canadian Blood Services

CBUC

Canadian Blood Utilization Collaborative

CCDT

Canadian Council for Donation and Transplantation

CIHI

Canadian Institute for Health Information

CNTRP

Canadian National Transplant Research Program

CIHR

Canadian Institutes of Health Research

CORR

Canadian Organ Replacement Register

CTR

Canadian Transplant Registry

DPMP

Donors per million population

F/P/T

Federal/provincial/territorial

FTE

Full-time equivalent

G&Cs

Grants and contributions

HIV

Human Immunodeficiency Virus

HQP

Highly qualified personnel

HSP

Highly Sensitized Patient

KE

Knowledge exchange

KPD

Kidney Paired Donation

MOU

Memorandum of Understanding

NOTDAW

National Organ and Tissue Donation Awareness Week

NSERC

Natural Sciences and Engineering Research Council

ODTEAC

Organ Donation and Transplantation Expert Advisory Committee

O&M

Operating and maintenance

OTDT

Organ and tissue donation and transplantation

PMS

Performance measurement strategy

P/T

Provincial/territorial

P/Ts

Provinces/territories

PTBLC

Provincial Territorial Blood Liaison Committee

R&D

Research and development

SPB

Strategic Policy Branch

SRAC

Scientific and Research Advisory Committee

WHO

World Health Organization

Executive Summary

Evaluation Purpose and Scope

The purpose of this evaluation was to assess the relevance and performance (effectiveness, efficiency and economy) of the Health Canada Canadian Blood Services (CBS) Contribution Programs, which include the Organ and Tissue Donation and Transplantation (OTDT) Program, and the Blood Research and Development (R&D) Program. The evaluation focused on the period from 2013–14 to 2016–17, and updated findings from the previous evaluation, which covered the period from 2008–09 to 2012–13. The evaluation was undertaken in accordance with the requirements of the Financial Administration Act and the Treasury Board Policy on Results (2016).

Program Description

Through the OTDT Program, Health Canada funds CBS for activities relating to national governance, development of leading clinical practices, system performance improvement, and public education and awareness. Through the Blood R&D Program, Health Canada funds CBS to undertake R&D activities to improve the safety and supply of the Canadian blood system.

Conclusions - Relevance

Continued Need

There is a continued need to address a persistent imbalance in Canada between the demand for organs/tissues and available supply. The transplant waitlist has exceeded 4,500 persons since 2010. In 2015, even though 2,580 transplants were performed, there were still 4,713 patients on organ waitlists at the end of that year, with an average wait time of 430 days. Similarly, there is a continued need for ongoing blood R&D to respond to emerging threats, while taking advantage of opportunities to further promote the safety of the blood system in Canada.

Alignment with government priorities

The OTDT and Blood R&D Programs both align with the government’s commitment to innovation in the health care system (e.g., funding health innovation research and innovation in health services delivery). The Blood R&D Program aligns with the Government of Canada’s commitment to the safety of the blood system. In addition, both programs align with the October 2017 Minister’s Mandate Letter, which highlighted the Government’s commitment for the Minister of Health to work with the provinces and territories to “develop a long-term vision for blood services that ensures safety and non-discrimination in donation policies, and facilitate collaboration on an organ and tissues donation and transplantation system that gives Canadians timely and effective access to care”. Both programs align with and support Health Canada’s Strategic Outcome #2: Canadians are informed of, and protected from, health risks associated with food, products, substances and environments, and are informed of the benefits of healthy eating.

Alignment with federal roles and responsibilities

The OTDT and Blood R&D Programs both align with federal roles and responsibilities as defined by federal legislation, including the Department of Health Act, the Canada Health Act, the Food and Drugs Act, and related regulations. The federal role in the OTDT Program was originally intended to be a temporary support to the provinces and territories (P/Ts) for the development and implementation of a coordinated OTDT system. However, external stakeholders, including P/Ts, support an ongoing or even an enhanced role for Health Canada in the OTDT Program.

Some of the OTDT Program’s activities, such as public awareness messaging, professional education, and development of leading practices, may overlap with activities undertaken by other stakeholders. However, they may equally be seen as complementary. In particular, activities that appear to duplicate work in larger, well-funded jurisdictions may also be addressing gaps in smaller, less well-resourced jurisdictions that lack the capacity to sustain those activities themselves. Furthermore, by virtue of their national scope, the activities of the OTDT Program are intended to add value beyond the benefits resulting from analogous P/T activities.

Health Canada’s role in the Blood R&D Program is largely uncontroversial from the perspective of stakeholders. Although some research currently funded by the Blood R&D Program would be eligible for funding through the Canadian Institutes for Health Research (CIHR), the Natural Sciences and Engineering Research Council (NSERC), provincial agencies and other organizations, stakeholders believe competition for these funds is such that it could be difficult to sustain ongoing Canadian research in this area without a dedicated program.

A focus on operational and regulatory issues was built into the Blood R&D Program’s current funding agreement, in response to the last evaluation. CBS has used R&D findings to inform submissions to Health Canada for changes to its internal policy or operational practice, which were subsequently approved by the department. In addition, BGTD representatives were open to learning more about the Blood R&D Program’s activities, with a view to identifying potential opportunities for collaboration among CBS and BGTD researchers.

Conclusions – Performance

Achievement of Expected Outcomes (Effectiveness)

Both the OTDT and Blood R&D Programs are making progress toward their immediate, intermediate, and longer-term outcomes. However, the OTDT Program has encountered some challenges in a number of areas.

Immediate outcomes

• Collaboration among OTDT partners. Through its various activities and advisory committee structure, the OTDT Program collaborates with a diverse range of stakeholders, primarily on the front-line, but has encountered challenges in engaging effectively with P/Ts. At present, CBS and the P/Ts have divergent perspectives regarding the need for an integrated, national OTDT system and differing levels of commitment to it. The Program has been unable to deliver on the Health Canada-funded activity aimed at developing a national governance model and accountability framework. The program is working to advance relationships with P/T partners, and national accountability discussions are ongoing between CBS and the P/Ts.
• Stakeholders are knowledgeable about leading practices, evidence, and knowledge generated by the programs. Large majorities of surveyed stakeholders of the OTDT and Blood R&D Programs are aware of the knowledge and evidence generated by these programs. However, there are opportunities to increase  awareness of the Blood R&D Program’s research activities among certain partners within the federal health portfolio, in particular BGTD and the Public Health Agency of Canada (PHAC). Health Canada has begun taking steps, in response to the previous evaluation, to more actively promote the exchange of knowledge generated by the Blood R&D Program within the federal health portfolio, and plans to continue doing so in the future.
• Public and patient awareness and understanding of OTDT. Most measures of public and patient awareness and understanding of OTDT have remained stable between 2012 and 2017. The level of awareness varies depending on the measure. Most importantly from the perspective of addressing the gap in Canada between the demand for organs/tissues and the available supply, the proportion of Canadians who plan to donate their organs/tissues at the time of death has hovered around 50% over this time period.

Intermediate outcomes

• Highly qualified personnel (HQP) participate in transfusion science and medicine. The Blood R&D Program has supported the training of HQP in transfusion science and medicine through graduate fellowship programs, post-doctoral fellowship programs, and provision of training positions in CBS research laboratories. Over the period covered by the evaluation, 82 people completed training for HQP through the Blood R&D Program.
• Stakeholders use R&D-generated knowledge and adopt new or modified OTDT knowledge and practices. Large majorities of surveyed stakeholders who are aware of the activities and knowledge generated by the OTDT and Blood R&D Programs have used this knowledge to inform their work. Knowledge generated by the programs has been used to develop or modify policy and practice, develop training or educational curricula, and design research, among other things. The perceived quality and relevance of the knowledge generated by the two programs is high.
• OTDT activities are integrated across jurisdictions. Some progress has been made in integrating OTDT activities across jurisdictions, most notably through the provincially-funded Canadian Transplant Registry (CTR) and related initiatives, but there is less evidence of integration resulting from federal funding. Respondents to the survey of OTDT Program stakeholders have expressed considerable support for CBS to continue its efforts to achieve a nationally coordinated and integrated OTDT system. Key informants emphasized the importance of respecting existing P/T activities and jurisdiction in OTDT in further efforts at integration.

Longer-term outcomes

• Contribution to the quality of OTDT systems across jurisdictions. Many OTDT stakeholders believe the quality of OTDT systems across jurisdictions has improved because of CBS’ activities. However, objective data are relatively limited at the present time, and disentangling CBS’ contribution from that of other stakeholders is challenging.
• Contribution to the confidence of stakeholders in the OTDT system. Public opinion surveys conducted between 2014 and 2017 show moderately high levels of public trust in the OTDT system. Among OTDT stakeholders surveyed, just over half believe stakeholder confidence in the OTDT system has improved due to the activities of CBS.
• Contribution to the safety, quality and supply of blood and blood products. Canadian surveillance data indicate low prevalence rates of select blood-borne pathogens among first time donors. While the extent to which the Blood R&D Program had a bearing on these long-term trends, relative to other factors, is unknown, it is reasonable to assume that it is contributing to the safety, quality and supply of blood and blood products through its research activities and training of HQP in transfusion medicine.

Demonstration of Economy and Efficiency

The OTDT Program has taken some steps to lower costs, but has experienced challenges in operating efficiently. In some cases, progress in generating outputs and outcomes has been slower than expected. Most notably, the Program has been unable to deliver on the Health Canada-funded activity to develop a national governance model and accountability framework for the OTDT system. The lack of progress with respect to governance, along with P/T questions about the need for a national model, may suggest that Health Canada needs to review this aspect of its funding to the OTDT Program. In other cases, particularly with respect to development and uptake of leading practices, program achievements are difficult to discern due to limited performance data and a lack of clarity and transparency in performance reporting.

Within the narrow context of the funding agreement between Health Canada and CBS, there are opportunities to improve governance and working relationships. CBS believes Health Canada could more actively demonstrate its support for the OTDT Program to other stakeholders, including the P/Ts. Health Canada would like improved transparency and timeliness in CBS’ provision of information to Health Canada. Within the broader context of governance of the F/P/T OTDT Program and the OTDT system in general, there are a number of unresolved differences of perspective among the implicated parties with regard to overall vision, roles, responsibilities, and accountability.

The Blood R&D Program is operating efficiently and economically, as evidenced by the large number of projects undertaken with a modest amount of funding, the relatively small proportion of program funds spent on operating and maintenance (O&M), strong program leadership, and selection of research projects aligned with stakeholder priorities. In addition, although CBS has not identified alternative sources of funding, it has increased its leveraging and collaborative work with other organizations (e.g., CIHR, Mitacs). This has resulted in more in-kind contributions, as well as a limited number of instances of direct financial support from partners. Both Health Canada and CBS believe current program governance and working relationships are effective. The Blood R&D Program is systematic and complete in its tracking of output and outcome information.

Recommendations

Recommendation 1

With respect to the Blood R&D Program, improve knowledge exchange of research projects and results among partners in the federal Health Portfolio to optimize awareness and uptake.

There are opportunities to increase awareness of the Blood R&D Program’s research activities among certain partners within the federal health portfolio, in particular BGTD and the Public Health Agency of Canada, as there is interest among these partners in learning more. Further efforts by Health Canada to actively promote the exchange of R&D knowledge within the federal health portfolio should optimize awareness and uptake of research findings, as well as identify areas for potential collaboration.

Recommendation 2

With respect to the OTDT Program, and in fulfillment of the October 2017 Minister’s Mandate Letter, Health Canada should facilitate collaboration with key stakeholders on the long-term objective of a comprehensive Canadian OTDT system and define Health Canada’s role in developing such a system.

While the federal role in the OTDT Program was intended to be a temporary support to the P/Ts for the development and implementation of a coordinated OTDT system, external stakeholders, including P/Ts, support an ongoing or even an enhanced role for Health Canada in the OTDT Program. Given the challenges the OTDT Program has encountered in delivering on some Health Canada-funded activities, and in light of the recent direction given to the Minister of Health to work with the provinces and territories to facilitate collaboration on an OTDT system that gives Canadians timely and effective access to care, Health Canada should consult with key stakeholders on the objective of a comprehensive Canadian OTDT system and define Health Canada’s role in developing such a system.

Recommendation 3

With respect to the OTDT Program, improve consistency and clarity in the collection and reporting of performance data.

In many cases, the evaluation was challenged in determining what the OTDT Program has achieved, due to limited performance data and/or lack of clarity/transparency in performance reporting. One notable example is the lack of performance measurement data on uptake of leading practices, as well as a lack of clarity with regard to the number of leading practices developed and updated during the evaluation period. The OTDT Program should take steps to improve consistency and clarity in the collection and reporting of performance data in order to better demonstrate progress toward expected outcomes.

Management Response and Action Plan

Evaluation of the Health Canada Canadian Blood Services Contribution Programs
2013-2014 to 2016-2017

Recommendations	Response	Action Plan	Deliverables	Expected Completion Date	Accountability	Resources
Blood R&D
Recommendation 1: With respect to the Blood R&D Program, improve knowledge exchange of research projects and results among partners in the federal Health Portfolio to optimize awareness and uptake.	Health Canada (HC) agrees. The evaluation findings indicated that partners in the federal Health Portfolio, in particular BGTD and PHAC, would like to improve awareness about the program’s research results.	In collaboration with CBS-R&D, HC will actively promote and facilitate the exchange of knowledge generated by the program to optimize awareness and uptake of research findings among Portfolio partners. This includes organizing annual knowledge exchange events for Portfolio staff and working with CBS-R&D to develop a knowledge translation (KT) plan to support better uptake of the research and best practices.	Annual knowledge exchange event, “An Insider’s Update on Blood Safety Research, Including Sex and Gender-Relevant Developments”.	November 14, 2017	Executive Director, Health Programs and Strategic Initiatives (HPSI)	No additional costs, as within existing budget and human resources.
			New funding agreement includes clear expectations for knowledge translation.	March 31, 2018
OTDT
Recommendation 2: With respect to the OTDT Program, and In fulfillment of the October 2017 Minister’s Mandate Letter, Health Canada should facilitate collaboration with key stakeholders on the long-term objective of a comprehensive Canadian OTDT system and define Health Canada’s role in developing such a system.	Health Canada agrees. HC will re-examine the federal role.	HC plans to engage with PTs, CBS and other stakeholders to review the federal role in regards to OTDT. Based on the above, Health Canada will determine further steps necessary.	Recommendations, if any, based on findings from the review.	March 31, 2019	Executive Director, HPSI	No additional costs, as work will be completed with existing budget and human resources.
OTDT
Recommendation 3: With respect to the OTDT Program, improve consistency and clarity in the collection and reporting of performance data.	Health Canada agrees and will work with CBS to improve the consistency and clarity in the collection and reporting of performance data by CBS.	HC in collaboration with CBS-R&D and OTDT will develop a Performance Information Profile, in accordance with the Treasury Board requirements and use it as basis for new funding agreement.	Performance Information Profile. New funding agreement requires performance measurement in accordance with the Performance Information Profile.	March 31, 2018	Executive Director, HPSI	No additional costs as work will be completed with existing budget and human resources

1.0 Evaluation Purpose

The purpose of this evaluation was to assess the relevance and performance (effectiveness, efficiency and economy) of the Health Canada Canadian Blood Services (CBS) Contribution Programs, which include the Organ and Tissue Donation and Transplantation (OTDT) Program and the Blood Research and Development (R&D) Program. The evaluation focused on the period from 2013–14 to 2016–17, and updated findings from the previous evaluation, which covered the period from 2008–09 to 2012–13. The evaluation was undertaken in accordance with the requirements of the Financial Administration Act and the Treasury Board Policy on Results (2016).

2.0 Program Description

2.1 Organ and Tissue Donation and Transplantation (OTDT) Program

2.1.1 Program Context

In April 1999, the House of Commons Standing Committee on Health tabled its report, Organ and Tissue Donation and Transplantation (OTDT): A Canadian Approach. Among the committee’s 18 recommendations was support for a comprehensive organ and tissue donation and transplantation system in Canada. In response to the report, federal, provincial and territorial (F/P/T) Ministers of Health agreed to formulate a “coordinated, comprehensive and integrated donation and transplantation strategy” to improve organ and tissue donation and transplantation in Canada. As part of this strategy, the Canadian Council for Donation and Transplantation (CCDT) was established as an advisory body to the Conference of Deputy Ministers in support of its efforts to coordinate F/P/T activities relating to organ donation and transplantation. The authority to make decisions with respect to OTDT remains with the P/T governments.

In June 2007, CBS proposed the creation of a national system for organ and tissue donation and transplantation which included the advisory functions previously undertaken by CCDT. On September 24, 2007, CBS’ proposal was accepted by provincial and territorial (P/T) Deputy Ministers of Health, excluding Quebec, and on April 1, 2008, CBS assumed the lead role of coordinating efforts for a national organ and tissue system.

In 2008 the CBS signed a Contribution Agreement with Health Canada for the improvement of OTDT. Health Canada’s Strategic Policy Branch (SPB) is responsible for administering the Contribution Agreement. According to the Agreement, CBS’ responsibilities include facilitating system performance improvement, facilitating public education and awareness, and supporting leading clinical practices. In 2012, CBS published the Call to Action, which identified numerous shortcomings of Canada’s donation and transplantation system and set out a plan to improve its performance (CBS, 2012). To date, the vision expressed in the Call to Action has not been endorsed by the federal or P/T governments. The current Contribution Agreement between CBS and Health Canada covers the period from September 1, 2014 to March 31, 2017. As of the writing of this report, the Contribution Agreement was extended and remains in place.

2.1.2 Program Profile

According to the OTDT Program’s Performance Measurement Strategy (PMS), Health Canada funding supports four main activities that deliver national services relating to OTDT.^{Footnote 1}

• National Governance, Policy, and Planning. This activity delivers partnership and engagement support, in order to build capacity in, and access to, a nationally coordinated and optimized OTDT system. It supports the development of high-performing OTDT interprovincial practices through policy, planning, coordination, collaboration and integration across jurisdictions and sectors.
• Leading Practice Development, Knowledge Translation, and Health Professional Education. This activity involves managing, coordinating, and administering leading practices from inception to consultation, reporting, and knowledge mobilization. It seeks to fill knowledge gaps and deliver education to enable professionals to apply the best available evidence in decision-making.
• Public Awareness and Education. This activity involves promoting organ and tissue donation to the public and educating the public about the need for organ and tissue donors, in an effort to improve public perceptions and understanding of OTDT.
• National Performance Analysis and Reporting. This activity involves establishing a performance measurement strategy and framework for effective and ongoing performance monitoring, analysis, and reporting as well as partnership engagement support to deliver high quality evidence-based results, reviewing data collected in the Canadian Transplant Registry (CTR) patient programs, and developing improved data analytics and reporting to optimize real-time organ listing and allocation across the country.

Outputs produced by these activities include strategic partnerships/collaborations, knowledge products, and knowledge exchange mechanisms.

2.1.3 Program Narrative

In the immediate term, OTDT Program activities and outputs are expected to result in the following outcomes: partners collaborate on OTDT, health professionals and decision-makers are knowledgeable about OTDT leading practices, knowledge, and evidence, and the public and patients understand OTDT processes. In the intermediate term, OTDT-related activities are expected to be integrated across jurisdictions, and relevant stakeholders are expected to use or adopt new or modified knowledge and practices. In the long term, the intermediate outcomes are expected to improve the quality of the OTDT system across jurisdictions, and to increase the confidence of relevant stakeholders in that system.

2.1.4 Program Alignment and Resources

The OTDT Program supports Health Canada’s Strategic Outcome #2: Canadians are informed of and protected from health risks associated with food, products, substances and environments, and are informed of the benefits of healthy eating. In particular, the OTDT Program supports the Biologics and Radiopharmaceuticals sub-Strategic Activity within the Health Products Strategic Activity.

Health Canada funding to CBS for the OTDT Program was $3.432 million in 2013–14 and $3.58 million per year between 2014–15 and 2016–17, for a total of $14.172 million over this period. P/T ministries of health (excluding Quebec) contribute an additional $3.58 million each year, and through Héma-Québec, the Government of Quebec contributes an additional $0.845 million.

2.2 Blood Research and Development (R&D) Program

2.2.1 Program Context

Blood safety was recognized as a major issue in Canada following the contamination of the Canadian blood supply with Human Immunodeficiency Virus (HIV) and Hepatitis C virus, from the late 1970’s through the 1980’s, which resulted in the public health crisis known as the “tainted blood scandal”. As a result, the Government of Canada established the Commission of Inquiry on the Blood System in Canada (the Krever Inquiry). The Krever Report was tabled in the House of Commons in 1997 and set out a series of recommendations that continue to guide improvements in the blood system in Canada.

Krever’s twenty-fourth recommendation stated that the national blood service should have the facilities and the competence to conduct in-house research and development, as well as collaborative work between the national blood service and other organizations. In late 1997, an F/P/T Memorandum of Understanding (MOU) was signed to create the National Blood Authority, an interim national authority. The MOU committed $5 million for research and development. When CBS was created in September 1998, it inherited the mandate for coordinating a national research and development program. In September 1998, Treasury Board approved the payment of a grant of $5 million from 2000–01 and ongoing to CBS. Health Canada’s SPB is responsible for administering the funding agreement.

Since the publication of the Krever Report, progress has been made in improving the safety of the blood system in Canada. According to the World Health Organization (WHO), in a speech given at the World Blood Donor Day on June 14, 2007, Canada has developed the highest standards of blood safety in the world. However, the WHO and international partners have recognized that maintaining the safety of the blood system in developed countries requires a strong resolve by governments to continue to support the regulatory and surveillance functions of the blood system for the future.

2.2.2 Program Profile

According to the Blood R&D Program’s PMS, Health Canada funding supports four main activities delivered by CBS, while Health Canada is directly responsible for two additional activities.

• Funding priority research and development programs and projects (CBS). This includes developing and implementing an R&D program that will lead to new discoveries in areas of specific focus that are relevant and timely for the blood system, identifying priority areas and projects to be funded, and monitoring and evaluating projects.
• Engaging in knowledge mobilization (translation and exchange) from creator to user (CBS). This includes developing and implementing a set of coordinated initiatives to develop and exchange knowledge that will enhance the capacity of researchers, health care professionals, staff in blood system service operations, and policy makers, as well as a commercialization strategy to facilitate the translation of research findings into new products.
• Training highly qualified personnel and building a transfusion science and medicine community of experts (CBS). This includes developing and implementing training programs for highly qualified personnel (HQP), providing operational funding to other blood and transfusion medicine research programs to leverage talent and infrastructure, and build capacity in the field, as well as deploying existing talent within an optimal structure.
• Facilitating collaborative working relationships (CBS). This includes leveraging and advancing collaborations with national and international partners, including other blood operators, universities, hospitals, and industry, in order to translate discoveries into changes in practice and policy.
• Effectively managing the Program (Health Canada). This refers to Health Canada’s responsibilities in setting priorities for funding that will support Health Portfolio needs, monitoring progress and performance to identify issues and showcase the Program’s achievements, and contributing to financial accountability and evaluation of the Program.
• Acting as a knowledge broker and supporting horizontal policy coordination (Health Canada). This refers to Health Canada’s work in acting as a knowledge broker between Program-generated knowledge and internal stakeholders in the Health Portfolio, coordinating and leading the effective delivery of the program across the Portfolio through horizontal collaboration and engagement, and acting as the single-window resource and Government of Canada link for CBS.

Outputs produced by these activities include grants and awards, knowledge exchange mechanisms, knowledge products and, collaborative working arrangements.

2.2.3 Program Narrative

In the immediate term, Blood R&D Program activities and outputs are expected to result in the following outcomes: HQP participate in transfusion science and medicine in Canada, and key stakeholders in the transfusion and transplantation community are knowledgeable about the evidence and knowledge generated by R&D projects. In the intermediate term, key stakeholders in the transfusion and transplantation community are expected to apply knowledge created by R&D projects. In the long term, changes in knowledge and application of that knowledge are expected to contribute to the safety, quality, and supply of blood and blood products.

2.2.4 Program Alignment and Resources

The Blood R&D Program aligns with and supports Health Canada’s Strategic Outcome #2: Canadians are informed of and protected from health risks associated with food, products, substances and environments, and are informed of the benefits of healthy eating. In particular, the Blood R&D Program supports the Biologics and Radiopharmaceuticals sub-Strategic Activity within the Health Products Strategic Activity.

Health Canada funding to CBS for the Blood R&D Program was $5.0 million per year between 2013–14 and 2015–2016 and $5.175 million in 2016–2017, for a total of $20.175 million over this period.

3.0 Evaluation Description

3.1 Evaluation Scope, Approach and Design

The evaluation focused on, but was not confined to, the period from 2013–14 to 2016–17, and included all activities within the Organ and Tissue Donation and Transplantation (OTDT) Program and the Blood Research and Development (R&D) Program. The evaluation issues were aligned with the Treasury Board of Canada’s Policy on Results (2016), and considered the five core issues under the two themes of relevance and performance. Corresponding to each of the core issues, specific questions were developed based on program considerations, and these guided the evaluation process.

Methods included literature review, document and data review, key informant interviews (n=32) with 47 key informants representing Canadian Blood Services, Health Canada, federal departments, P/T representatives, and other external stakeholders (such as academics or researchers, organ donor registries, and non-profit organizations), and surveys of OTDT Program stakeholders (n=104) and Blood R&D Program stakeholders (n=56). The evaluation took place over a six-month period between March 2017 and August 2017. More detail on the data collection and analysis methods is provided in Appendix 2.

3.2 Limitations and Mitigation Strategies

Most evaluations face constraints that may have implications for the validity and reliability of evaluation findings and conclusions. The following table outlines the limitations encountered during the implementation of the selected methods for this evaluation, along with mitigation strategies used to ensure that the evaluation findings can be used with confidence to guide program planning and decision-making.

Table 1: Limitations and Mitigation Strategies

Limitation	Impact	Mitigation Strategy
Stakeholder survey samples were compiled by CBS from its contact databases, and do not constitute a random or representative sample of the stakeholder populations.	Survey findings cannot be interpreted as representative of all program stakeholders.	Survey findings are used in conjunction with other lines of evidence.
Key informants were identified based on purposive sampling. Budget considerations placed constraints on the number of external key informant interviews that could be completed.	External key informant interview findings cannot be interpreted as representing the views of all external stakeholders.	Interview findings are used in conjunction with other lines of evidence. No conclusions are drawn solely on the basis of interview data.
Limited quantitative information was available to support the analysis of efficiency and economy. In particular, activity- and output-based costing information was not available.	Quantitative analysis of efficiency and economy is limited primarily to comparing planned and actual spending.	Analysis is supplemented by qualitative information from interviews.

4.0 Findings

4.1 Relevance: Issue #1 – Continued Need for the Programs

There is a continued need to address the ongoing imbalance between demand for organs/tissues in Canada, and available supply, as well as a continued need for ongoing blood research and development (R&D) to respond to emerging threats and support the safety of the Canadian blood system.

Organ and Tissue Donation and Transplantation (OTDT) Program

Available data indicate ongoing challenges within the OTDT system in Canada, highlighting in particular the persistent shortfall that exists between the demand for organs, and the available supply. Over the past decade, the number of Canadians waiting for an organ transplant has exceeded the number of transplants performed in a given year, and since 2010, has exceeded 4,500 individuals (CBS, 2016c; CIHI, 2016b), as shown in Figure 1. In 2015, for example, 2,580 transplants were performed, but there were 4,713 patients on organ waitlists at the end of that year, and 242 patients died while waiting for a transplant (CIHI, 2017b).^{Footnote 2}

Canada’s deceased donation rate^{Footnote 3} increased by 29% between 2006 and 2015 to 18.2 donors per million population (DPMP), exceeding the OTDT Program’s target of 17 DPMP. However, performance was variable across provinces, and the overall rate lagged significantly behind that of peer nations, such as the United States and Spain (25 and 40 DPMP, respectively, in 2015) (CBS, 2016c). By 2016, Canada’s deceased donation rate reached 21.8 DPMP, similar to Australia (20.8) and the United Kingdom (21.4); however, the United States and Spain continued to exceed Canadian rates with 30.8 and 43.4 DPMP, respectively (IRODaT, 2017).^{Footnote 4} Furthermore, Canada’s living donation rate actually decreased by 8% between 2006 and 2015, with variable performance across provinces (CBS, 2016c), although it should be noted that similar downward trends in living donation rates occurred in the United States and Australia during this period.

In a 2016 progress report, Canadian Blood Services (CBS) characterized progress over the period from 2006 to 2015 as “incremental” (CBS, 2016c). CBS noted that, while advances had been made in some areas, national and provincial programs were not yet formally coordinated or aligned. From CBS’ perspective, additional effort was required to further improve coordination in order to produce improved outcomes for patients. A brief comparison of OTDT systems in eight of Canada’s peer nations^{Footnote 5} (some of which, including the United States, the United Kingdom, and Spain, have significantly better OTDT system performance), reveals that centralized control or coordination of the organ donation process is found in these nations, implying that Canada’s performance might improve with better coordination.

Input from stakeholders who participated in this evaluation indicates a considerable degree of support for CBS’ efforts to improve coordination. For example, a large majority (86%) of respondents to the OTDT stakeholder survey agreed that CBS should continue its efforts to achieve a nationally coordinated and integrated OTDT system in Canada. Somewhat smaller majorities agreed that CBS’ activities relating to OTDT are well-aligned with the priorities of their own organization (74%), and with the priorities of their province or territory (64%).

Furthermore, most key informants across all categories agreed that overall performance of the Canadian OTDT system remains sub-optimal, even as improvements have been made in some areas. Many believe there is ongoing need to strengthen and ensure a nationally-consistent OTDT system to enhance access to organs and tissues, and reduce the disparities in access to, and utilization of, organs and tissues across Canada that stem, in part, from differences in capacity among P/Ts. As a specific example, one interviewee indicated that, whereas provinces like Ontario and British Columbia have well-designed provincial OTDT systems with access to substantial financial and human resources (e.g., transplant fellows), its counterparts in other jurisdictions are less well-equipped and, in some instances, must rely on significantly fewer resources and personnel. In this context, national coordination is viewed as a means of “leveling the playing field” by ensuring all Canadian jurisdictions share (for example) leading practices, educational materials for health care professionals, and public awareness messaging. Overall, among P/T stakeholders who were interviewed, varying degrees of support were expressed regarding the need for a national OTDT system.

Moreover, some key informants, including, but not limited to, those representing P/Ts, cautioned that existing P/T limits to coordination should be respected in efforts to create a national system, or that some aspects of coordination should be prioritized over others. As shown in Table 2, while respondents to the OTDT stakeholder survey see all of CBS’ current activities as important, they did not prioritize all of them equally. Overall, development of leading practices, minimum datasets, and the Canadian Transplant Registry (CTR), along with public awareness and education about OTDT, were seen as the most important activities, while development of a clinical governance structure and an integrated data management and analytics service were seen as somewhat less important, although both of the latter activities were still seen as important by more than 80% of respondents.

Table 2: Proportion of Survey Respondents Rating CBS’ OTDT Activities as Important

	(n=104)
Activity	Very important	Somewhat important	Total of Somewhat or very important
Development and dissemination of leading practices and clinical practice guidelines in donation and transplantation	70%	26%	96%
Development of minimum datasets to provide a standardized Canadian approach to guide data collection	66%	29%	95%
Development and enhancement of the Canadian Transplant Registry	73%	20%	93%
Public awareness and education about OTDT	71%	22%	93%
Professional education about OTDT	57%	36%	93%
Development of guidance, recommendations, reports, or reviews of the OTDT system in Canada	53%	39%	92%
Development of national guideline documents that aim to clarify the roles and responsibilities of donation physicians	59%	32%	91%
Development of a formal, integrated clinical governance structure for the OTDT system in Canada	59%	28%	87%
Development of an integrated data management and analytics service	58%	26%	84%
Source: Survey of OTDT Stakeholders.

Blood R&D Program

The previous evaluation confirmed the ongoing relevance of the Blood R&D Program, recognizing that known pathogens, as well as new and developing ones, would continue to pose risks during blood transfusion. As a result, there was an ongoing need for scientifically accurate and up-to-date information for evaluating proposed policy or operational changes and developing improved practices to avoid contamination (Health Canada, 2013).

The need for ongoing blood R&D to inform the safety of the blood system has not diminished since the last evaluation was completed. Both key informants and the available literature point to a need for ongoing blood R&D to identify, understand, and develop responses to continuously evolving pathogens. In recent years, for example, West Nile, Dengue, Chikungunya, and Zika viruses have emerged as potential threats to the blood supply. Ongoing blood R&D is also needed to address risks associated with blood transfusion, facilitate the increasing personalization of interventions involving transfusion, and take advantage of new opportunities to promote the safety, supply, and quality of the blood supply.

An R&D function is a common feature of the national blood supply systems maintained in jurisdictions comparable to Canada. Among eight peer nations^{Footnote 6} examined, six have a unified blood system similar to Canada’s, wherein a single organization is responsible for all blood-related activities. Of these six, four include a research and development function. Furthermore, although the United States has multiple organizations that collaboratively manage the blood supply, it also funds basic blood research through the National Heart, Lung, and Blood Institute (US Department of Health and Human Services, 2017). It is also worth noting that access to independent laboratory facilities and scientific expertise were identified as mandatory requirements by the World Health Organization’s (WHO) Expert Committee on Biological Standardization when it adopted assessment criteria for national blood regulatory systems in 2013 (WHO, 2013). The Blood R&D Program helps to provide these functions for CBS.

4.2 Relevance: Issue #2 – Alignment with Government Priorities

Both programs align with, and support, departmental strategic outcomes. The Blood R&D Program aligns with the Government of Canada’s commitment to the safety of the blood system, and both programs can be seen as aligning with the Government’s current emphasis on health system innovation. Most recently, the Government highlighted its commitment to work with the provinces and territories to “develop a long-term vision for blood services that ensures safety and non-discrimination in donation policies, and facilitate collaboration on an organ and tissues donation and transplantation system that gives Canadians timely and effective access to care”.

Both the OTDT and Blood R&D Programs align with and support Health Canada’s Strategic Outcome #2: Canadians are informed of and protected from health risks associated with food, products, substances and environments, and are informed of the benefits of healthy eating. In particular, the programs support the Biologics and Radiopharmaceuticals sub-Strategic Activity within the Health Products Strategic Activity. In particular, the Blood R&D Program helps to protect Canadians from health risks associated with the use of biological products by conducting research that informs evidence-based practices aimed at ensuring the safety of the Canadian blood system. Similarly, by developing leading practices and public awareness activities relating to OTDT, the OTDT Program helps to inform and protect Canadians from health risks associated with organ and tissue transplantation.

While federal Budgets and Speeches from the Throne since 2013 have made no explicit mention of either OTDT or blood R&D, the 2015 Budget expressed a commitment to innovation in the health care system through the funding of health innovation research (Government of Canada, 2015b), and in the 2016 Budget, the government committed to pan-Canadian innovation in health services delivery (Government of Canada, 2016b). The OTDT Program’s activities relating to national system optimization and standardized data enhancements can be seen as aligning with this emphasis on innovation, as can the R&D activities of the Blood R&D Program.

Most recently, the October 2017 Minister’s Mandate Letter highlighted the Government of Canada’s commitment for the Minister of Health to work with the P/Ts to “develop a long-term vision for blood services that ensures safety and non-discrimination in donation policies, and facilitates collaboration on an organ and tissues donation and transplantation system that gives Canadians timely and effective access to care” (Prime Minister of Canada, 2017).

With respect to the Blood R&D Program, Health Canada completed development of a new regulatory framework for blood and blood products in 2013 (Health Canada, 2015). The new Blood Regulations, which came into force in October 2014, constitute Health Canada’s final response to the recommendations of the Krever Commission, indicating that the integrity and safety of the Canadian blood supply has remained a federal priority. Almost all key informants representing Health Canada viewed the Government of Canada’s funding of the Blood R&D Program as evidence of its commitment to a safe blood system in Canada.

4.3 Relevance: Issue #3 – Alignment with Federal Roles and Responsibilities

4.3.1 Alignment with federal roles and responsibilities

Both programs align with federal roles and responsibilities, as these are defined by federal legislation. Health Canada’s role in the Blood R&D Program is largely uncontroversial. While the Government of Canada’s role in and funding of the OTDT Program was originally intended to be a temporary support to the P/Ts for the development and implementation of a coordinated OTDT system for Canada, external stakeholders, including P/Ts, support an ongoing, or even an enhanced role for the department, despite a lack of consensus on the precise nature of that role.

Both the OTDT Program and the Blood R&D Program align broadly with federal roles and responsibilities as these are defined by federal legislation. In particular, Health Canada’s interest and activity in the areas of OTDT and blood safety derive from its mandate and role, as set out in the Department of Health Act, the Canada Health Act, the Food and Drugs Act, and related regulations.

• The Department of Health Act sets out Health Canada’s mandate. It defines the Minister’s duties to include, among other things, the preservation of Canadians’ health and well-being, the protection of Canadians against risks to health and the spread of diseases, investigation and research into public health, the establishment and control of safety standards and safety information on requirements for consumer products, and the collection, analysis, interpretation, publication and distribution of information relating to public health.
• Under the Canada Health Act, Health Canada is responsible for setting and administering national standards for the health care system and providing funding to the provinces and territories for their health care services, with the objective of protecting, promoting and restoring the physical and mental well-being of residents of Canada, and facilitating reasonable access to health services without financial or other barriers.
• Finally, Health Canada is responsible for regulating biological products (including blood and blood products, as well as tissues, organs and xenographs) under the authority of the Food and Drugs Act, and its related regulations, including the Food and Drug Regulations, the Blood Regulations, and the Safety of Human Cells, Tissues and Organs for Transplantation Regulations.

Notwithstanding the broad legislative basis for federal activity in the area of the OTDT, Health Canada representatives explained that the Government of Canada’s role in and funding of the OTDT Program was originally intended to be a temporary support to the P/Ts for the development and implementation of a coordinated OTDT system for Canada. At the present time, almost ten years after the Government first began funding CBS for OTDT-related activities, Health Canada key informants raised concerns about the extent to which the program has delivered on federally-funded activities.^{Footnote 7}

Conversely, representatives of CBS, the P/Ts, and other external stakeholders agreed that Health Canada should continue to play a role in the OTDT Program, although they did not necessarily agree on the precise nature of that role. CBS representatives and a few external stakeholders were of the view that Health Canada should increase its involvement in, and national leadership of, the OTDT Program, including clearly demonstrating its support and advocating for the work that CBS undertakes with federal funding, and taking a more active role in liaising with P/Ts to promote the Program’s objectives. Other external stakeholders suggested that Health Canada should adopt a formal role in the OTDT system, underpinned by federal legislation similar to that found in other developed nations, such as the National Organ Transplant Act in the United States, as well as The Human Tissue (Scotland) Act 2006 and The Human Tissue Act 2004 (England, Wales and Northern Ireland) in the United Kingdom. Among P/T representatives who participated in the evaluation, there was agreement that Health Canada should increase its involvement by clarifying its expectations and vision for an integrated OTDT system, committing more or longer-term funding, and developing and instituting performance measures to help assess the extent to which program objectives are being achieved.

Despite disagreeing on the precise nature of Health Canada’s role, key informants generally believe the OTDT Program would be seriously impacted if Health Canada withdrew its support. Some have implied that program activities might cease in the absence of Health Canada’s financial support, and they could also lose the legitimacy that its presence and involvement lend to OTDT-related activities. Several stakeholders argued that the P/Ts’ involvement in, and contribution to, the OTDT Program is bolstered by Health Canada’s own participation, and that the withdrawal of Health Canada’s support might well erode P/T commitment. If this were to occur, they suggested, smaller and less well-resourced jurisdictions would be more adversely affected than their larger and better-supported counterparts.

In contrast to questions raised concerning Health Canada’s role in the OTDT Program, the evaluation found little evidence of controversy with regard to the department’s role in the Blood R&D Program. Most external stakeholders perceived Health Canada as responsible for providing a combination of program funding and oversight and generally viewed this role as appropriate. Furthermore, interviewees noted that there are relatively few additional sources of dedicated funding for blood-related research in Canada at the present time. Some of these individuals suggested that were Health Canada to cease funding the Blood R&D Program, Canada’s ability to sustain a high-performing transfusion research program would be adversely affected. A few interviewees noted that such research might not be viewed as being of a high enough priority to compete for financial support from other funding bodies, such as the Canadian Institutes of Health Research (CIHR). One key informant expressed the opposite view, questioning the need and rationale for Health Canada to fund CBS for blood R&D, given that the Government of Canada already provides health research funding to CIHR.

From Health Canada’s perspective, the department’s involvement in the Blood R&D Program helps to demonstrate the Government’s commitment to sustaining the safety of the country’s blood supply, and its withdrawal from the program might elicit serious concerns from stakeholders who have not forgotten the aftermath of the tainted blood scandal and the Krever Inquiry.

4.3.2 Alignment with work of Health Canada and the Public Health Agency of Canada

Alignment of the OTDT Program’s work with that of Health Canada and the Public Health Agency of Canada is limited at the present time, but evolving. A focus on operational and regulatory issues was built into the Blood R&D Program’s current funding agreement with Health Canada, and in several instances over the period covered by the evaluation, CBS used R&D findings to inform submissions to Health Canada for changes to its internal policy or operational practice, which were subsequently approved by the department.

The extent to which the work that CBS undertakes through the OTDT Program is directly aligned with, or supports, the work of Health Canada and the Public Health Agency of Canada is relatively limited at the present time, although it is evolving. Health Canada key informants noted that the program’s work is highly operational in nature and therefore more directly relevant to the P/Ts than to the Government of Canada. However, they noted that the program’s current initiative to integrate the functions of the Canadian Institute for Health Information’s (CIHI) Canadian Organ Replacement Register (CORR) with those of the CTR will ultimately support the department. In addition, a few key informants reported that the program’s work relating to the bacteriological load in tissues had been incorporated into national tissue standards, and were subsequently incorporated into Health Canada regulations.^{Footnote 8} Finally, both CBS and Public Health Agency of Canada (PHAC) key informants reported that CBS has recently reached out to PHAC with regard to linking its surveillance activities for adverse events relating to cells, tissues and organs (i.e. the Cell, Tissue, and Organ Surveillance System) with those of the CTR.

With regard to the Blood R&D Program, key informants representing both CBS and Health Canada explained that a focus on operational and regulatory issues was built into the latest funding agreement, in response to a recommendation made by the previous evaluation. The evaluation found several documented instances in which CBS, as the blood operator, used R&D findings to inform submissions to Health Canada for changes to its own policy or operational practice, which were subsequently approved by the department. For example, based on evidence generated by the Blood R&D Program in partnership with Héma-Québec, Health Canada issued regulatory authorizations to CBS and Héma-Québec to change their blood donor deferral period for men who have sex with men from five years to one year (Government of Canada, 2016a). Pursuant to this, Health Canada provided $3,000,000 to further advance research to ensure non-discriminatory practices surrounding this issue, to be administered by CBS in partnership with Héma-Québec (CBS, 2017). Additional examples are provided in Section 4.4.5.

That said, key informants representing Health Canada’s Biologics and Genetic Therapies Directorate (BGTD), which is responsible for regulating biologics in Canada, reported that beyond reviewing submissions from CBS for changes to its own policy and practice, they do not use research findings generated by the Program to inform their work, relying instead on the Directorate’s internal research or research generated by PHAC or international regulators. Moreover, according to BGTD, the Directorate has no influence over the Blood R&D Program’s research priorities, although it is periodically approached by the Strategic Policy Branch (SPB) of Health Canada to give input into proposals advanced by CBS in relation to the program. BGTD representatives view their lack of influence over the program’s research priorities as appropriate, citing the need for the regulatory arm of Health Canada to preserve its neutrality.

4.3.3 Overlap, duplication and complementarity

Some activities of the OTDT Program may overlap with activities at the provincial or territorial level, particularly in the case of larger, better-resourced jurisdictions. However, these activities are also complementary, providing value-added by virtue of their national scope, and filling a gap for smaller jurisdictions with limited resources to undertake these activities on their own. Beyond the Blood R&D Program, there is no other program or source of funding specifically dedicated to blood R&D.

OTDT Program

Evidence available to the evaluation suggests that there may be some overlap and duplication between the work of the OTDT Program and that of other organizations. However, it is also possible to view CBS’ work in these areas as complementary to that of other organizations, particularly since activities that appear to duplicate work in larger well-funded jurisdictions may also be addressing gaps in smaller, less well-resourced jurisdictions that lack the capacity to sustain those activities themselves.

The two areas of potential overlap most commonly mentioned by informants were health professional education, and public awareness and education. With respect to health professional education, interviewees noted that OTDT stakeholders in individual jurisdictions are developing their own professional educational resources and programming, and thus CBS’ activities in this area could be seen as duplication of effort. However, it was also noted that CBS drew on and adapted professional educational materials originally developed by Trillium Gift of Life, rather than independently developing its own and these materials have since been offered to other jurisdictions. From this perspective, therefore, CBS’ work can also be viewed as complementary.

With respect to public awareness and education, interviewees reported that many organ donation and transplantation organizations administer their own public awareness and education campaigns, and charities such as the David Foster Foundation and the Organ Project have also begun to promote OTDT in Canada. However, key informants noted that less well-resourced jurisdictions may rely wholly, or in part, on educational materials developed by CBS, or suggested that CBS’ messaging may have an additive effect on public awareness of OTDT. From this perspective, CBS’ work in the area of public awareness and education is not duplicative, but rather complementary.

A few key informants viewed CBS’ development of leading practices as potentially duplicating work done by other stakeholders (e.g., at the P/T level). To this point, a small majority of respondents to the survey of OTDT stakeholders (56%) indicated that, in the absence of CBS, they would not be able to get similar information (i.e., information of the same quality or relevance to their work) on OTDT leading practices from another source. The remaining 44% identified a wide range of other sources of such information, with no one source clearly preferred. Key informants noted that the extent to which overlap and duplication may be occurring is difficult to assess with respect to leading practices, since CBS and other entities may appear to be developing leading practices independently but may actually be collaborating.

Several interviewees discussed issues of overlap, duplication and complementarity in the context of the relationship between CBS and the Canadian National Transplant Research Program (CNTRP). CBS currently provides a small amount of annual funding to the CNTRP (albeit through the Blood R&D Program),^{Footnote 9} it also contributes biologic materials for use in CNTRP research studies. In addition, many researchers who work with CBS are also affiliated with the CNTRP. CBS and CNTRP representatives reported that both organizations have taken deliberate steps to coordinate their activities in order to avoid overlap and duplication, and promote complementarity. Nevertheless, a few stakeholders perceive potential duplication of effort between the two organizations.

Some interviewees described activities relating to the integration of CIHI’s CORR with the CTR as an example of an explicit attempt to avoid overlap and duplication, while promoting complementarity. CORR is a national database that collects data to track long-term national trends in dialysis activity, organ transplantation, organ donation, and waitlist statistics that has been in operation since 1972. Once integration is complete, the CTR will become the common repository for data related to all donors, transplants and outcomes (CIHI, 2016a) and CIHI’s role in the collection of donation and transplantation data will be taken up by CBS.

Blood R&D Program

Overall, the Blood R&D Program complements, rather than overlaps or duplicates, the activities of other stakeholders. Although some research currently funded by the Blood R&D Program would be eligible for funding through CIHR, the Natural Sciences and Engineering Research Council (NSERC), provincial agencies, and other stakeholders, key informants suggested that the competitive nature of these alternative funding programs, combined with the relatively low profile of transfusion-related research, would limit the overall support available for this research in Canada in the program’s absence.

There is no other program specifically dedicated to blood R&D in Canada. Moreover, interviewees indicated that few, if any, other Canadian organizations conduct research similar to that being undertaken through the Blood R&D Program, although some engage in complementary work. For instance, while hospitals are unlikely to pursue the development of new blood products or pathogen-reduction technologies (as CBS does), they may engage in clinical research to demonstrate product effectiveness.

Finally, some interviewees argued that the Blood R&D Program reduces the risk of overlap and duplication of effort by sustaining robust collaboration and coordination with other stakeholders, such as Héma-Québec, the Canadian Society for Transfusion Medicine, and institutions of higher learning, such as the Centre for Blood Research at the University of British Columbia. That said, , the extent to which there may be overlap or duplication between the Blood R&D Program’s activities and BGTD’s internal research program is unclear. However, since BGTD’s research program is specific to research which can inform the regulatory process, the possibility of duplication is low.

4.4 Performance: Issue #4 – Achievement of Expected Outcomes

4.4.1 Immediate outcome #1: Collaboration among OTDT Program partners

Through various activities and its advisory committee structure, the OTDT Program collaborates with a diverse range of stakeholders, primarily on the front-line, but has encountered challenges in engaging effectively with P/Ts. CBS and the P/Ts have divergent perspectives regarding the need for an integrated, national OTDT system and, to date, the program has been unable to deliver on the Health Canada-funded activity to develop a national governance model and accountability framework for the OTDT system.

The OTDT Program’s performance measurement data indicate that the program reported 25 formal and informal strategic partnerships and collaborations in 2015–16 and 16 in 2016–17. Partnerships and collaborations involved a range of organizations, such as Héma-Québec and other provincial partners, the CNTRP, CIHI, PHAC, the Canadian Standards Association, Alberta Health Services, the Canadian Critical Care Society, the Ottawa Hospital Research Institute, the Canadian Society of Transplantation, and several US-based organizations, such as the United Network for Organ Sharing, and Philadelphia Gift of Life. Collaborative activities included working groups, committee and initiative meetings or planning sessions, workshops, site visits, webinars, and teleconferences.

Much of the OTDT Program’s collaborative work occurs through its advisory committee structure. This structure consists of the Organ Donation and Transplantation Expert Advisory Committee (ODTEAC), and 17 sub-committees and working groups focused on specialized areas. The ODTEAC is comprised of representatives from provincial organ donation programs (e.g., Transplant Québec, Gift of Life Manitoba), professional organizations and associations (e.g., the Alberta Transplant Institute at the University of Alberta; the Heart Transplant Program at the University of Calgary), chairs of other organ donation committees (e.g., Canadian Liver Transplant Network), and non-voting CBS members (with the chair and vice-chair appointed by CBS). Its Terms of Reference define its mandate as being to facilitate service delivery on the organ donation and transplantation system, to make recommendations to CBS about coordination based on a collaborative approach, and to liaise with other organ and tissue donation advisory bodies. The sub-committees and working groups carry out the ODTEAC’s core responsibilities in relation to their field of expertise, and provide reports to the main committee. 

Results from stakeholder feedback forms and self-assessment surveys of activities, as conducted through the OTDT Program’s advisory committees, working groups and other forums and symposia, were largely positive regarding the operation, value, mandate, and priorities of the OTDT committees, as well as committee meeting agendas, meeting materials, and opportunities to contribute to discussions. Moreover, key informants generally view these committees and working groups as effective in increasing collaboration. More specifically, the groups allow for sharing of research and best practices across provinces, and support interprovincial collaboration in areas such as policy development, public awareness, and reporting.

Likewise, positive feedback about the OTDT Program’s collaborative activities emerged from the OTDT stakeholder survey. Almost all respondents (95%) were aware of at least one collaborative activity, with the largest proportions aware of the development and enhancement of the CTR, the development of leading practices, and the program’s public awareness and education activities (77%, 67%, and 64%, respectively).

Furthermore, a majority of survey respondents (89%) reported having participated in CBS-led activities relating to OTDT since 2012 (including 44% who were members of one or more OTDT working groups, and 28% who were members of an advisory committee), and 71% reported that they had personally contributed time to at least one of CBS’ OTDT-related activities in the last five years. Most commonly, respondents had contributed time to developing leading practices (40%), professional education about OTDT (28%), and guidance, recommendations, reports, or reviews of the OTDT system in Canada (26%). A majority of those who contributed time were pleased with the collaborative experience: 82% agreed that they were satisfied with how CBS involved them in collaboration activities, while 78% agreed that their contributions were taken into account by CBS; in both cases, about half agreed strongly. Overall, 82% of those who had contributed time reported being very satisfied (47%) or somewhat satisfied (35%) with the experience.

Despite this positive feedback from those directly involved in collaborative activities, some key informants were of the view that CBS collaborates well with front-line groups through the ODTEAC structure and its various sub-committees and working groups, but has struggled to engage effectively with the P/Ts, and in particular, with the P/Ts through the Provincial Territorial Blood Liaison Committee (PTBLC). The PTBLC is directly linked to P/T Deputy Ministers and Ministers of Health and is, from the P/T perspective, the governance body for the OTDT Program. CBS, however, considers governance by the PTBLC to be problematic, and from CBS’ perspective, the ODTEAC is functioning in an interim governance role, pending endorsement of a formal governance model by the F/P/T Ministers of Health.

A governance model and accountability framework is an expected deliverable of the OTDT Program, according to CBS’ current funding agreement with Health Canada. CBS key informants explained that the program’s proposed “clinical governance model” is an attempt to clarify stakeholder roles and responsibilities in a national OTDT system, including those of CBS, the P/Ts, and other implicated stakeholders, as well as to identify decision-making authorities in instances of P/T disagreement or dispute. At present, CBS and the P/Ts have divergent perspectives on the need for an integrated, national OTDT system, and differing levels of commitment to it, To date, the program has been unable to deliver on the Health Canada-funded activity to develop a national governance model and accountability framework. CBS key informants reported that the program is working to advance relationships with P/T partners, and national accountability discussions are currently ongoing between CBS and the P/Ts.

4.4.2 Immediate outcome #2: Key stakeholders are knowledgeable about OTDT leading practices, evidence, and knowledge generated by R&D projects

Large majorities of surveyed stakeholders of the OTDT and Blood R&D Programs are aware of the knowledge and evidence generated by these programs. However, some key informants within the federal health portfolio reported low levels of awareness of the Blood R&D Program’s research activities and results.

OTDT Program

The OTDT Program’s performance data indicate that the program is making information about OTDT leading practices, knowledge, and evidence available to stakeholders through various means. Between 2014–15 and 2016–17, the OTDT Program produced a total of 140 knowledge products, exceeding its target of 30 per year in two of the three years. Of these knowledge products, 80% targeted health professionals. In addition, between 2013–14 and 2016–17, the OTDT Program produced 83 publications through online journals and databases, such as PubMed and used CBS’ Professional Education website to publish data reports, leading practices and clinical guidelines. About 18 professional education webpages were published during this timeframe. However, the available performance data are vague about the number of new leading practices developed and published since 2012, compared to the number of existing ones that were updated.

In addition to producing and publishing knowledge products, the OTDT Program conducted 140 knowledge exchange (KE) activities between 2014–15 and 2016–17, primarily targeting health professionals, committees or working groups, and transplant professionals and organizations.^{Footnote 10} Results from self-assessment surveys conducted after KE events show that many respondents believe they gained knowledge from the event (average 89%), agreed that the information from the event could be or will be used in practice (average 88%), and agreed that the knowledge obtained was relevant (average 95%).^{Footnote 11}

Results from the survey of OTDT stakeholders show that a large majority of respondents (89%) were aware of at least one of the 15 OTDT leading practices listed in the survey.^{Footnote 12} Respondents were most likely to be aware of Death Determination (69%), Donor Management (58%), and Living Donor Paired Exchange (57%). Fewer than half of respondents reported awareness of each of the other leading practices; see Table 3.

Table 3: Awareness of OTDT Leading Practices among Stakeholder Survey Respondents

Leading Practice	Percent of All Respondents (n=104)
Death Determination (2003, 2005, 2007, 2012, 2016)	69%
Donor Management (2004)	58%
Living Donor Paired Exchange (2005)	57%
Highly Sensitized Patient Registry (2005)	46%
Kidney Paired Donation Protocol for Participating Donors (2014)	45%
Effective Requesting: End-of-Life Family Conversations/Consent (2014)	40%
Eye and Tissue Banking in Canada (2012)	40%
Donation Physician Specialists (2011, 2015)	39%
OTDT Ethics Consultation (2011)	31%
Kidney Listing and Allocation (2007)	28%
Liver Listing and Allocation (2016)	28%
Enhancing Living Donation (2006)	26%
Allocation of Organs for Combined Transplantation (2012)	24%
Tissue Bioburden Reduction and Control (2016)	20%
Assessment and Management of Immunologic Risk in Transplantation (2005)	15%
None of the above	11%
Multiple-response question. Total may not sum to 100%. Source: Survey of OTDT Stakeholders.

While key informants generally viewed the dissemination of leading practices as successful, they noted that dissemination at the local level may not be equally effective in all jurisdictions or regions. Some key informants also pointed to challenges in accurately measuring stakeholder awareness, as well as uptake, of leading practices; doing so would entail both knowing the entire population of potential users and having the ability and resources to track awareness and uptake within this population.

Blood R&D Program

Available performance measurement data indicate that the Blood R&D Program is making evidence and knowledge generated by R&D projects available to stakeholders. The program produced about 1,297 knowledge products over the period covered by the evaluation, exceeding its targets in all three years. The majority of the products (85%) were peer-reviewed publications, such as published abstracts, journal articles, review articles, and letters or editorials. Non-peer reviewed publications such as CBS website publications, technical reports, theses, and “Fast Policy Facts” were also produced.^{Footnote 13} Overall, researchers (41%), health care providers (25%), and blood operators (18%) are the most common target audiences for these publications, although a few targeted industry and regulators.

The Blood R&D Program funds and uses KE mechanisms, such as conference presentations, webinars, seminars, symposia, and other mechanisms, to exchange information and connect key stakeholders, research users, and research producers within the blood system. Over the course of the evaluation period, approximately 1,480 KE activities were undertaken, exceeding the program’s annual targets each year. These activities primarily targeted researchers (58%) and health care providers (37%). Participant feedback surveys indicate that, in nearly all cases, at least 85% of respondents reported that the event enhanced their knowledge. All of the key events received high marks (ranging from 85% to 100% of respondents) for quality, and most received high marks (ranging from 45% to 95% of respondents) for relevance or applicability of the information to attendees’ workplace.

Similarly, the Blood R&D Program’s survey of stakeholders found high levels of awareness of its research among survey respondents. Survey results show that all respondents were aware of the CBS’ Centre for Innovation research activities pertaining to transfusion and transplantation science and medicine. A majority of respondents were also either extremely familiar or generally familiar with each of the Centre’s specific research areas, namely fresh blood products (93%), plasma protein products and plasma product replacements (82%), donor health and screening (82%), stem cell products (70%), and legal, health services, and policy research (54%).

The Blood R&D Program uses the h-index, a bibliographic measure of an author’s productivity and impact, as an indicator of the extent to which research users are aware of, and value the program’s published research evidence. The program’s performance measure strategy aims to have 90% of CBS staff researchers achieve an h-index of 10.6, which reflects the Canadian Science Standard (i.e., the average h-index of Canadian academic authors within the field of science in 2012). Performance data show that CBS researchers exceeded this target every year; moreover, the average h-index of CBS research staff is over double the average h-index of Canadian academic science authors (10.6), ranging from 23.9 in 2013–14 to 26.9 in 2016–17.

Overall, the evidence suggests that the Blood R&D Program is successful at generating and disseminating research knowledge, and that there is a high degree of awareness of this research among program stakeholders. That said, there are opportunities to increase awareness of the program’s research activities and research outputs among health portfolio partners. Key informants representing Health Canada noted that the department had recently taken steps, in response to the previous evaluation, to more actively promote the exchange of knowledge generated by the Blood R&D Program within Health Canada, and with other partners in the federal health portfolio. For example, a knowledge translation event to facilitate engagement between CBS, Health Canada, PHAC, and CIHR stakeholders took place in 2015. Health Canada key informants indicated that there are plans to continue promoting knowledge exchange in the future.

4.4.3 Immediate outcome #3: Public and patients are aware of, and understand OTDT

Public opinion research carried out by CBS indicates that most measures of public and patient awareness and understanding of OTDT have been largely stable between 2012 and 2017. The level of awareness varies depending on the measure, ranging from approximately 1% of Canadians who are aware that CBS is the organization responsible for national administration of the organ and tissue donation system, to 78% who are aware that organs and tissue can be donated at the time of death. Most importantly, from the perspective of addressing the gap in Canada between demand for organs and tissues and the available supply, the proportion of Canadians who plan to donate their organs and tissues at the time of death has hovered around 50% over this time period.

The OTDT Program conducts public education activities to promote organ and tissue donation in an effort to evolve public perception about OTDT, and improve understanding of OTDT. The program’s performance data show that, during the period covered by the evaluation, major public education activities included:

• Participating in National Organ and Tissue Donation Awareness Week (NOTDAW) annually from 2014–15 to 2016–17, along with F/P/T governments and other stakeholders. The goal of NOTDAW is to increase awareness of organ donation among members of the general public, increase awareness of CBS’ work in this area, and promote organ donor registration at blood donor clinics. In 2015, the event resulted in a total circulation of about 4.2 million information products through various media, which increased to over 107 million in 2016; in addition, the 2016 NOTDAW resulted in 6,403 donor registrations.
• Collaborating with BC Transplant in 2015–16 to promote the organ donor registry at blood donor clinics across the province, in an effort to increase the number of registered organ donors, and stimulate conversation about organ donation. This collaboration involved 57 clinics and resulted in 234 people returning their donor registration form to BCT. CBS reports that it is currently working to facilitate collaboration between provinces to implement a public awareness campaign around organ and tissue donation and transplantation, with a view to increasing the number of organ donors.
• Delivering a series of nine webinars in collaboration with the Canadian Transplant Association during 2014–15, which were communicated through Facebook, Twitter, and webpages.
• Reaching out to the public through online and print advertising and social media, including accumulating over 66,000 followers on Facebook, Twitter and Instagram, and distributing two short video advertisements through Facebook, Instagram, AOL, and Google Preferred, which were viewed over 3.5 million times.

Results from surveys of the general public conducted by CBS indicate that most measures of public awareness and understanding of organ and tissue donation were stable, or fluctuated slightly between 2012 and 2017.^{Footnote 14} In 2015, 55% of respondents indicated they had seen, read, or heard something or a lot, on the topic of organ and tissue donation in the last few years, the highest rate of recall recorded so far for this question and a significant increase over the previous wave (41% in 2014). This suggests that OTDT-related public education and awareness activities are reaching their target audience, although it is important to note that the question did not ask specifically about CBS’ activities. The available survey data also show that:

• In 2015, although 78% of respondents were aware that organs and tissue could be donated at the time of death, 84% did not know what organizations are responsible for the provincial management of organ and tissue donation programs. These proportions are similar to previous years these questions were asked.
• Most respondents are unaware that CBS plays a role in the organ and tissue donation system in Canada. When aided, 22% of respondents in 2017 identified CBS as the organization “that aids in the national administration of the organ and tissue donation system” ^{Footnote 15}, a slightly higher proportion than in 2015 (18%).
• Some respondents indicated they would officially express their intention to donate their organs and tissues at the time of their death by signing the donor card with their driver’s license (35% in March 2015, declining since April 2012) or signing the donor card with their health card (21% in March 2015, fluctuating since April 2012). Few respondents indicated they would officially express their intention to donate through a website (5% in March 2015, fluctuating since April 2012).
• Perhaps most importantly from the perspective of addressing the ongoing gap in Canada between the demand for organs and tissues, and the available supply, is that in 2017, approximately half (46%) of respondents stated their intention to donate their organs or tissues at the time of death, a slight decrease from 2015 (51%), but similar to prior years. Of these respondents, most had signed their donor card (82%) or spoken to family about their decision (77%), while about one-third (34%) had put their name on a registry. These proportions fluctuated slightly between 2012 and 2017.

Key informants who commented on the OTDT Program’s public awareness and education activities believe that raising public awareness is important to increasing organ donations, and that additional efforts are needed, perhaps with closer coordination between CBS and P/T programs, in order to avoid duplication of effort, real or perceived. Furthermore, they noted that it is difficult to disentangle the impact of CBS’ initiatives from those of other stakeholders, and indeed, the OTDT Program’s public opinion research does not specifically seek to assess the impact of CBS’ public education activities, relative to those of other stakeholders. Finally, some key informants suggested that public awareness of OTDT issues is only one component of a high-performing OTDT system, and that other elements of the Canadian system must also be addressed, even prioritized, to achieve desired improvements in system performance.

4.4.4 Immediate outcome #5: Highly qualified personnel participate in transfusion science and medicine in Canada

The Blood R&D Program has supported the training of highly qualified personnel (HQP) in transfusion science and medicine, through graduate fellowship programs, post-doctoral fellowship programs, and provision of training positions in CBS research laboratories.

The Blood R&D Program aims to train HQP and build a transfusion science and medicine community of experts. The program accomplishes this by developing and implementing training programs for HQP, such as graduate fellowship programs, post-doctoral fellowship programs, and provision of training positions in CBS research laboratories. The program’s performance measurement data show that between 2013–14 and 2016–17, 82 people completed training for HQP, almost half (44%) as summer interns. The program exceeded its target of training 15 HQP each year in three of these four years.

The Blood R&D Program also provides operational funding to other research programs to leverage talent, infrastructure, and build capacity; determine skill gaps; and deploy existing talent in an optimal structure. Performance data show that a total of 93 unique research projects with these objectives were funded between 2013 and 2017. Overall, key informants across all groups perceive CBS’ Blood R&D Program as successful in training HQP in the field of transfusion medicine and blood R&D.

4.4.5 Intermediate outcome #1: Key stakeholders use knowledge generated by R&D projects, and adopt new or modified OTDT knowledge and practices

Large majorities of surveyed stakeholders who are aware of the activities and knowledge generated by the OTDT and Blood R&D Programs have used this knowledge to inform their work. Knowledge generated by the programs has been used to develop or modify policy and practice, develop training or educational curricula, and design research, among other things. The perceived quality and relevance of the knowledge generated by the two programs is high.

OTDT Program

The OTDT Program last undertook research into uptake of leading practice recommendations in 2012.^{Footnote 16} The research found that most participants were aware of the leading practice recommendations, believed they were relevant and useful, and believed they contributed to improvements in their practice. However, most key informants also agreed that all leading practice recommendations could be more consistently used. Challenges and barriers to implementation included lack of funding and staff, lack of knowledge among practitioners, lack of “buy-in” among physicians and other OTDT stakeholders, inadequate internal communications among medical staff and external communications (e.g., media coverage and a champion for organ donation and transplantation), and challenges identifying potential donors. The OTDT Program has not conducted similar comprehensive research on uptake of leading practices since 2012.^{Footnote 17}

However, some information on uptake of leading practices was included in CBS’ 2016 system progress report, which noted that leading practices on donor management and neurological determination of death had been implemented or were in the process of being implemented in several provinces, including British Columbia, Alberta, Saskatchewan, Manitoba, Ontario, Quebec, New Brunswick, Nova Scotia, and Newfoundland and Labrador (CBS, 2016c).

In addition, the OTDT Program surveys KE event attendees about their intentions to use information from the event in their practice and in decision-making. During the period covered by the evaluation, among KE event attendees who responded to the survey, many (average 89%) agreed that the information from the event could be, or will be, used in practice. ^{Footnote 18}

The survey of OTDT stakeholders provides more recent information on uptake of leading practices. Results show that:

• A large majority (83%) of respondents who were aware of at least one of 15 leading practices listed on the survey, or 74% of all survey respondents, also reported having implemented or used at least one of them. Health care providers were significantly more likely to have used or implemented leading practices compared to other respondents.
• Rates of use varied across the leading practices. About 40% of all survey respondents have used Death Determination (42%) and Donor Management (40%), while about 30% have used Effective Requesting (30%), Living Donor Paired Exchange (30%), Highly Sensitized Patient Registry (29%), and Kidney Paired Donation Protocol (28%). Each of the remaining leading practices has been used by fewer than one-quarter of respondents; see Table 4.

Table 4: Use of OTDT Leading Practices by OTDT Stakeholder Survey Respondents

Leading practice	Percent of respondents who are aware of the LP (n=variable)	Percent of all respondents (n=104)
Death Determination (2003, 2005, 2007, 2012, 2016)	61%	42%
Donor Management (2004)	70%	40%
Effective Requesting: End-of-Life Family Conversations/Consent (2014)	74%	30%
Living Donor Paired Exchange (2005)	53%	30%
Highly Sensitized Patient Registry (2005)	63%	29%
Kidney Paired Donation Protocol for Participating Donors (2014)	62%	28%
Eye and Tissue Banking in Canada (2012)	60%	24%
Kidney Listing and Allocation (2007)	69%	19%
Donation Physician Specialists (2011, 2015)	48%	18%
OTDT Ethics Consultation (2011)	56%	17%
Enhancing Living Donation (2006)	59%	15%
Tissue Bioburden Reduction and Control (2016)	67%	14%
Liver Listing and Allocation (2016)	48%	14%
Allocation of Organs for Combined Transplantation (2012)	44%	11%
Assessment and Management of Immunologic Risk in Transplantation (2005)	44%	7%
Source: Survey of OTDT Stakeholders. Note: Respondents were only asked about their use of a specific leading practice if they indicated awareness of it. For the purposes of this table, the proportion using each leading practice was calculated out of the proportion who were aware of the leading practice (second column), as well as out of the total number of respondents (third column). When interpreting the data on use presented in the table, it is important to note that a minority of respondents (between 9% and 36%) who were aware of each leading practice indicated that it was not applicable to their work.

• Respondents who have used one or more leading practices most often indicated using them for general reference purposes (78%), or reported that they promoted, disseminated, or distributed them within their organizations (74%). Slightly fewer, but still a majority, reported that they adopted the leading practice as standard or required practice for their organization (60%), or used the leading practices in teaching and educational curricula (58%). Less frequent uses included integrating the leading practices into written policies (48%), adapting or customizing the leading practices for their organization (44%), using the leading practices to inform policy or regulatory decisions (42%), incorporating the leading practices into mandatory or optional training (33%), and promoting, disseminating, or distributing the leading practices outside their organizations (30%). However, among all respondents to the survey, only 55% agreed that overall use of OTDT leading practices has become more widespread in Canada, although it is important to note that 35% of respondents did not know.
• Respondents had positive opinions of the quality and relevance of the leading practices. Almost all (98%) of those who had used at least one leading practice believe they are useful, including 64% who perceive them as very useful and 34% who see them as somewhat useful. Furthermore, a majority of respondents believe the leading practices are based on best available evidence (81%), are useful to their work (79%), help to fill a knowledge gap (77%), and are of high quality (77%). Furthermore, 75% are confident in using the leading practices in their work.
• More than half of respondents (56%) said that in the absence of CBS, they would not be able to get similar information (i.e., information of the same quality and relevance to their work) on OTDT leading practices from another source. The remainder identified a wide range of other sources of such information, with no one source preferred.
• Two-thirds (67%) of respondents ranked CBS among their top three sources of evidence-based leading practices relating to OTDT (after academic journals and databases at 75%), and CBS was the first choice of 26% of respondents for this type of information, following academic journals and databases (39%).

Similar to the 2012 research, the survey found that lack of resources (funding, time, and personnel) or organizational support is a significant barrier to implementation. In addition, key informants who participated in this evaluation observed that, while CBS can develop and disseminate leading practices, it lacks any authority to compel jurisdictions to adopt them. Therefore, leading practices may be, and often are, implemented at the organizational or facility level without being mandated provincially. Thus, implementation of leading practices is inconsistent across P/Ts and even within jurisdictions.

Blood R&D Program

Performance measurement data indicate that, between 2013–14 and 2015–16, knowledge generated by the Blood R&D Program was used in the creation or update of 46 policies, procedures, practices, products, or standards, with CBS being the most frequent user of this information, followed by Health Canada. CBS frequently used knowledge generated by the Blood R&D Program to update its standard operating procedures. As the blood operator, CBS also used the program’s findings to inform submissions to Health Canada for changes to its own policy or operational practice, which were subsequently approved by Health Canada (these are evidently being counted by the program as examples of Health Canada use). For example, submissions from CBS based on Blood R&D Program findings resulted in Health Canada approving license amendments in a variety of areas, including the preparation of washed red cells, donation criteria for men who have sex with men, elimination of mixing of B1s, discontinuation of the process of balancing whole blood prior to centrifugation, changes to arm scrub methods, and extension of cryosupernatant plasma storage. Additionally, in response to the rapid spread of the Zika virus, CBS implemented a temporary 21-day donation deferral period for international travellers in February 2016 that was subsequently officially approved by Health Canada. Also, a Health Canada-approved protocol for the collection of convalescent plasma to treat patients with Ebola in Canada was developed by CBS as a precautionary measure in 2015–16.

While the above examples certainly involved Health Canada review of research findings generated by the Blood R&D Program, it is important to note that CBS itself initiated this review through its submissions to Health Canada. BGTD representatives who were interviewed indicated that the Directorate relies on its own internal research and research generated by the Public Health Agency of Canada (PHAC) and international regulators to inform its policy and regulatory work. That said, BGTD representatives were open to learning more about the Blood R&D Program’s research activities, with a view to identifying potential opportunities for collaboration among CBS and BGTD researchers.

As another example of the use of knowledge generated by the Blood R&D Program, the Canadian Standards Association applied, in 2014–15, program-generated knowledge in amending standards related to the length of time that red blood cells can be exposed to uncontrolled temperatures.

Although results from feedback surveys for the Blood R&D Program’s KE events suggest that a minority of survey respondents intend to use the knowledge they gained to modify their current practices, results from the R&D stakeholder survey show that virtually all respondents with at least some familiarity with the program’s research activities (96%) have used the knowledge they gleaned from these activities in at least one of the ways examined by the survey. Most frequently, these respondents indicated that they used the knowledge to develop and/or deliver education (72%), while 65% used it to design and/or conduct research, and 63% used it to develop and/or implement policies or procedures. Other reported uses included developing and/or implementing recommendations or guidelines (52%), developing and/or implementing standards, legislation or regulations (46%); and developing new or improved products (41%).

As was also the case for the OTDT Program, the perceived quality and relevance of the Blood R&D Program’s research is high. Survey results show that at majority of respondents with at least a minimal level of familiarity with each of the program’s research areas considered these areas to be extremely or generally relevant to the current and future needs of their personal work, as well as to the needs of their primary organization and other organizations in the field. Furthermore, almost all respondents (98%) consider the quality of knowledge created by CBS’ research activities to be either excellent (54%) or good (44%).

4.4.6 Intermediate outcome #2: OTDT-related activities are integrated across jurisdictions

Some progress has been made in integrating OTDT activities across jurisdictions, most notably through the provincially-funded Canadian Transplant Registry (CTR) and related initiatives, but there is less evidence of integration resulting from federal funding. There is considerable support among respondents to the survey of OTDT Program stakeholders for CBS to continue its efforts to achieve a nationally coordinated and integrated OTDT system. Key informants across all groups emphasized the importance of respecting existing P/T’s OTDT activities and jurisdiction, in further efforts at integration.

The development of the CTR and related interprovincial patient programs, such as the Kidney Paired Donation (KPD) Program, the National Organ Waitlist, and the Highly Sensitized Patient (HSP) Program are important advances toward the goal of greater integration of OTDT-related activities across jurisdictions. According to CBS, most provinces are using the CTR directly, while others have built direct data feeds between the CTR and their local information technology systems in order to reduce duplication of data entry (CBS, 2016c). CBS reports that, of the transplants facilitated in 2014 by the KPD program, 51% occurred between donors and recipients who were registered through transplant centres in different provinces. Additionally, 56% of the transplants provided to highly sensitized kidney patients, as a result of the HSP program in that year, involved donors and recipients who came from different provinces (CBS, 2016c). ^{Footnote 19} However, it is important to note that the CTR and related interprovincial patient programs have been funded by the P/Ts, rather than the Government of Canada.

The evaluation found less evidence of integration resulting from Health Canada’s funding to the OTDT Program^{Footnote 20}. For example, agreement has been achieved on data elements and definitions for the minimum datasets for heart, liver, lung and kidney transplants, and some progress has been made toward the CBS-CIHI data transfer, although this initiative remains in the planning stages. As already noted, the Program has not delivered on the Health Canada-funded activity to develop a national governance model for the OTDT system, and P/Ts are questioning the very need for such a model.

Among respondents to the OTDT stakeholder survey, 65% believe that CBS’ activities have led to greater integration of OTDT-related activities across jurisdictions (14% disagreed and 21% did not know), while 86% agree that CBS should continue its efforts to achieve a nationally coordinated and integrated OTDT system in Canada. Moreover, 87% of respondents to the OTDT stakeholder survey believe that the development of a formal, integrated clinical governance structure for the OTDT system in Canada is somewhat or very important (including 59% who believe it is very important).

Similarly, key informants from all stakeholder groups generally agreed that progress has been made toward integration, particularly through the provincially-funded CTR. However, many emphasized the importance of respecting existing P/T’s OTDT activities and jurisdiction in further efforts at integration. It was noted that many P/Ts have already invested significantly in provincial systems or are facing fiscal pressure, and therefore may be reluctant to commit to initiatives that they believe might duplicate their efforts.

4.4.7 Longer term outcome #1: Contribution to the quality of OTDT systems across jurisdictions

Many OTDT stakeholders believe the quality of OTDT systems across jurisdictions has improved because of CBS’ activities. However, objective data are relatively limited.

Among OTDT survey respondents there is considerable agreement that the activities of the OTDT Program are leading to improvements in quality. 70% of survey respondents believe the quality of OTDT systems in Canada has improved because of CBS’ activities, and 61% believe these activities have reduced inconsistencies in OTDT systems across Canadian jurisdictions.^{Footnote 21} Similarly, many key informants believe that CBS’ activities are contributing to improved quality of OTDT systems, noting that provincially-funded programs such as the KPD, the HSP, and the CTR are helping to reduce wait times and improving access for patients.

There is some quantitative evidence to substantiate these assertions at the present time. Data on KPD program activity, reported by CBS in the system progress report for the period from 2009 to 2015, show that 391 of the 741 candidates registered with the program were transplanted, a number that CBS considers unlikely to have been achieved without the program (CBS, 2016c). According to the same report, 51% of the kidney transplants facilitated by the KPD program in 2014 involved interprovincial sharing of organs, which would not have been possible in the absence of the program. CBS also reports that the program has been successful in increasing transplants for blood-group O patients, having coordinated transplants for 140 of the 427 blood-group O patients who have participated (however, it is unknown how this compares to what would have been achieved in the absence of the program). In addition, as CBS acknowledges in the same report, the absolute number of living kidney donors has been relatively flat since 2006, and the number of pairs referred to the KPD program has been static for three years.

As another example, CBS reports that the number of transplants of highly sensitized patients through the HSP has gradually increased over time, with more than 200 patients transplanted through this program since 2013 (CBS, 2016b). Moreover, 57% of the transplants provided to highly sensitized kidney patients in 2014 as a result of the HSP involved donors and recipients who came from different provinces (CBS, 2016c). However, it is unclear how many of the remaining patients would have been transplanted if the program did not exist, or how their wait times for transplantation would have differed. These examples demonstrate the need for rigorous evaluation of these and other initiatives to improve the quality of OTDT systems, particularly in terms of their impact on patients. Even if undertaken, however, such evaluations would have difficulty disentangling CBS’ contributions from those of other implicated stakeholders.

It is also worth noting that overall trend data reported in Section 4.1 indicate ongoing opportunities to improve the quality of OTDT systems in Canada. As described in that section, demand for organs has greatly exceeded supply for every year over the past decade, and the number of individuals waiting for organs has remained above 4,500 since 2010. While donation rates at the time of death have increased over the past decade, it remains below the rate of comparable jurisdictions, and the living donation rate has actually declined over this period. Moreover, performance continues to vary across jurisdictions.

4.4.8 Longer term outcome #2: Contribution to the confidence of stakeholders in the OTDT system

Public opinion surveys conducted between 2014 and 2017 show moderately high levels of public trust in the OTDT system. Among OTDT stakeholders surveyed, just over half believe stakeholder confidence in the OTDT system has improved due to the activities of CBS.

Data from public opinion surveys carried out by CBS in 2014, 2015, and 2017 show that about 70% of respondents each year were confident that their wishes regarding organ and tissue donation would be acted upon by their family, although somewhat fewer – 60% – were confident that their wishes regarding organ and tissue donation would be acted upon by the hospital. The proportion of respondents who agreed that the organs and tissues donation system in Canada is administered in the best interest of the public has fluctuated over this time period, but exceeded 70% in all three years, and reached its highest level (76%) in March 2017. The proportion of respondents who agreed that CBS does what is best for the OTDT system in Canada followed a similar pattern, reaching 76% in March 2017. The surveys did not attempt to measure the extent to which CBS’ activities, in particular, may have influenced public attitudes.

Likewise, while key informants generally believe confidence among transplant program personnel and health professionals has improved, they were uncertain about the extent to which the OTDT Program may have contributed to this increased confidence. When asked directly, 54% of respondents to the survey of OTDT stakeholders agreed that stakeholder confidence in the OTDT system in Canada has improved because of CBS’ activities, while 17% disagreed and 29% did not know.

4.4.9 Longer term outcome #3: Contribution to the safety, quality and supply of blood and blood products

Canadian surveillance data indicate low prevalence rates of select blood-borne pathogens among first-time donors. While the extent to which the Blood R&D Program had a bearing on these long-terms trends is unknown, it is reasonable to assume that it is contributing to the safety, quality, and supply of blood and blood products, through its research activities and training of HQP in transfusion medicine.

Long-term surveillance data for select blood-borne pathogens (HIV, hepatitis B, hepatitis C, and syphilis) in Canada show that prevalence rates for first time donors have declined considerably since the mid-1990s. In 2015, these rates ranged between 0.3 positive donations per 100,000 donations for HIV, to 5.8 positive donations per 100,000 donations for hepatitis B (CBS, 2015). Overall, these low prevalence rates indicate that Canada’s blood supply is safe. The extent to which the Blood R&D Program’s research may have influenced long-term trends relative to other stakeholders could not be determined through this evaluation. However, it is reasonable to assume that the program has contributed to the safety, quality, and supply of blood and blood products by undertaking research to support regulatory and operational changes and by supporting the training of HQP in transfusion science and medicine.

4.5 Performance: Issue #5 – Demonstration of Economy and Efficiency

The OTDT Program has taken some steps to lower costs but has experienced challenges in operating efficiently. It has had difficulty generating intended outputs and outcomes in the activity areas for which it receives Health Canada funding, and in some cases, program achievements are difficult to discern due to limitations in performance reporting. The Blood R&D Program is operating efficiently and economically.

 

4.5.1 OTDT Program

Observations on Economy

Variances between planned and actual expenditures for the OTDT Program, based on financial data provided by CBS, are shown in Table 5. Between April 2013 and March 2017, the OTDT Program spent virtually 100% of the total Health Canada allocation.

Table 5: Variance between Planned Spending and Expenditures - OTDT Program 2013-14 to 2016-17

Year	Planned Spending				Expenditures				Variance ($)	% planned budget spent
	Operating and Maintenance (O&M)Table 5 Footnote 1	Salary & Benefits	Contractual  Personnel	TOTAL	O&M1	Salary & Benefits	Contractual  Personnel	TOTAL
2013-14	$828,688	$1,678,259	$925,232	$3,432,179	$560,758	$1,767,361	$1,103,420	$3,431,539	$640	99.9%
2014-15	$552,753	$1,024,932	$2,002,315	$3,580,000	$504,759	$1,140,525	$1,934,717	$3,580,001	-$1	100%
2015-16	$316,726	$2,086,473	$1,176,800	$3,579,999	$382,483	$1,988,943	$1,208,572	$3,580.000	$1	100%
2016-17	$558,340	$1,767,931	$1,253,729	$3,580,000	$693,275	$1,693,782	$1,192,943	$3,580,000	$0	100%
Source: CBS. Table 5 Footnote 1 O&M includes: travel and accommodation, meetings, training, materials, equipment, and other. Return to table 5 footnote 1 referrer

Program expenditures by category, as proportion of total spending, are shown in Table 6.

Table 6: Allocation of Expenditures - OTDT Program 2013-14 to 2016-17

Year	Percentage of Total Expenditures
	O&MTable 6 Footnote 1	Salary & Benefits	Contractual  Personnel
2013-14	16%	52%	32%
2014-15	14%	32%	54%
2015-16	11%	56%	34%
2016-17	19%	47%	33%
2013-14 to 2016-17	15%	47%	38%
Source: CBS. Table 6 Footnote 1 Due to rounding, totals may not sum to 100%. Return to tale 6 footnote 1 referrer Table 6 Footnote 2 O&M includes: travel and accommodation, meetings, training, materials, equipment, and other. Return to table 6 footnote 2 referrer

Operating and maintenance (O&M) costs were relatively low, 15% of the total program costs, yet showed small but consistent increases relative to planned expenditures. The OTDT Program has taken steps to control travel costs in order to improve economy. For example, approximately $50,000 in travel costs in Q2 of 2014–15 were saved by holding a number of meetings via teleconference rather than in person. In Q3 of that same year, additional travel costs were saved by some attendees of a Medical Examiner/Coroners donation event electing to drive rather than fly, and through the negotiation of a better rate to attend Canadian Organ Donation and Transplantation Network meetings.

Staffing costs represented 47% of total program expenditures, while contractual staff accounted for 38% of expenditures. Starting in Q4 of 2014–15 and continuing through 2015–16, staffing costs were underspent due to approved hires not yet being filled and staff absences. These resources were partly redirected to temporary contractual staff. In total, there are 51.5 full-time equivalents (FTEs) associated with the OTDT Program, funded by Health Canada and the P/Ts.

The significant amount of spending on contractual staff has evidently been an area of concern for Health Canada, which in 2016 suggested opening discussion with the OTDT Program about how to best stabilize spending on contractual staff in lieu of full time staff, i.e., increasing the number of permanent staff relative to contract staff as a cost saving measure.^{Footnote 22} While it is unclear if this discussion has taken place or what the outcome of it was, CBS key informants reported that the Program has recently begun assigning tasks to internal staff that had previously been carried out by consultants (primarily medical specialists), such as development of leading practices and document preparation and translation.

Observations on efficiency

The available evidence suggests that the OTDT Program has experienced challenges in operating efficiently. In particular, it has experienced challenges in generating some of its intended outputs and outcomes in the activity areas for which it receives Health Canada funding. According to data provided by Health Canada program officials, the OTDT Program received approximately $7.8 million over the period covered by the evaluation for activities relating to strategic plan development and clinical governance,^{Footnote 23} of which no less than $280,000 was intended for the development of a national governance model for the OTDT system.^{Footnote 24} To date, the Program has been unable to deliver on this commitment. Furthermore, at least some P/Ts are questioning the very need for such a model; although those who participated in this evaluation appear to be open to greater national coordination, and see national coordination as an appropriate role for CBS. The lack of progress with respect to governance, along with P/T questions about the need for a national model, may suggest a need for Health Canada to review this aspect of its funding of the OTDT Program. As another example, in relation to the Program’s activities relating to system performance improvement,^{Footnote 25} Health Canada key informants noted that the CBS-CIHI data transfer initiative is not as far advanced as had been hoped; as noted above, this initiative is still in the planning stages. The amount of Health Canada funding specifically for the CBS-CIHI data transfer initiative is unknown.

In other cases, it is challenging to assess the extent to which the OTDT Program is operating efficiently, as its achievements are difficult to discern due to limited performance data, or lack of clarity and transparency in performance reporting. One notable example relates to the Program’s work with respect to leading practices. As described earlier in this report, it is not clear in reporting how many leading practices were developed or updated (or even planned) during the period covered by this evaluation, nor has the OTDT Program conducted research on uptake of leading practices since 2012. This gap was filled to some extent by the OTDT stakeholder survey carried out as part of this evaluation, which showed that 74% of all survey respondents had used or implemented at least one leading practice. Data provided by Health Canada officials show that the Program received no less than $3.6 million in Health Canada funding over the evaluation period for the development of leading practices, and no less than $1.6 million for professional education and knowledge translation.

Key informants identified a wide range of suggestions to improve efficiency and/or economy of OTDT Program delivery, but no individual approach was generally preferred to any other. Some advocated for significant modifications, or even fundamental redesign, variously suggesting that Health Canada should focus on the following:

• funding targeted initiatives (e.g., scaling up successful regional projects);
• creating a new entity, distinct from CBS, to oversee or coordinate OTDT activities in Canada; or
• significantly increasing its own involvement in the OTDT system, possibly through the implementation of federal legislation such as exists in the United States and the United Kingdom.

Others argued for a variety of changes to the current program, such as better articulating the roles and responsibilities of program stakeholders, expanding support for OTDT research in Canada, and better integrating activities relating to blood, tissues, and organs. The diverse range of perspectives held by stakeholders, with respect to the appropriateness of the design and objectives of the OTDT Program, underscores the challenges that would be involved in identifying a widely-accepted alternative.

Observations on governance

The available evidence suggests that improvements are needed in OTDT Program governance and working relationships to facilitate more efficient delivery of program activities, and progress on program objectives. With respect to the working relationship between Health Canada and CBS, key informants from both organizations identified areas for improvement. CBS key informants suggested that Health Canada could more actively demonstrate its support for the OTDT Program to other stakeholders (including the P/Ts). Health Canada key informants suggested improved transparency and timeliness in the Program’s provision of information to the department. This suggests opportunities to further clarify roles and responsibilities within the narrow context of governance of the current funding agreement.

Within the context of the larger F/P/T OTDT Program, and the OTDT system in general, progress on matters related to governance, as noted above, has proven difficult to achieve. There are currently two governance structures existing in parallel: the Organ Donation and Transplantation Expert Advisory Committee (ODTEAC), and the Provincial Territorial Blood Liaison Committee (PTBLC). Neither one of these has the full support and recognition of all parties (CBS, P/Ts, and Health Canada), and, as noted above, CBS’ efforts to develop a national governance model have not been successful to date. Indeed, there are considerable ongoing differences of perspective among the implicated parties with regard to overall vision, roles, responsibilities, and accountabilities, which remain to be resolved. National accountability discussions are reportedly ongoing between CBS and the P/Ts, and Health Canada and the P/Ts have recently begun working more collaboratively on the OTDT file, in order to sort out differences among them and provide more consistent direction to CBS.^{Footnote 26}

Observations on adequacy and use of performance measurement data

The OTDT Program has a performance measurement strategy (PMS) in place dated May 2015 and collects performance information from a wide variety of sources. Review of this information suggests that it was often incomplete for the period covered by this evaluation, and numerous limitations emerged. For example, participant feedback surveys for the Program’s knowledge exchange (KE) events vary in question wording and response scales, do not consistently report on the number of participants or survey respondents, and do not consistently address participant intentions to implement information in practice. As another example, the Program does not clearly report how many leading practices have been developed or updated in the period covered by the evaluation, and has not undertaken research on uptake of leading practices since 2012. As a final example, although the Program regularly surveys the public to measure awareness and understanding of OTDT, these surveys do not link responses with exposure to OTDT messaging.

While acknowledging that the OTDT Program generates a substantial volume of reporting, Health Canada representatives identified opportunities to improve the clarity and transparency of this reporting, in order to enhance understanding of program activities and achievements. CBS key informants highlighted the challenges involved in attributing outcomes to activities undertaken by CBS through the OTDT Program, given the complexity of the system and the number of stakeholders involved. Some interviewees argued that CBS must be careful to avoid overstating its contribution to the OTDT system in Canada, relative to the contributions of other stakeholders. Overall, the OTDT Program may benefit from focusing additional resources or expertise on performance measurement, in order to address these limitations and challenges.

4.5.2 Blood R&D Program

Observations on economy

As shown in Table 7, based on financial information provided by CBS, the Blood R&D Program spent its entire allocation between 2013–14 and 2016–17. Over this period, the Program underspent slightly on O&M and salary costs relative to plan, and increased expenditures on grants and contributions (G&Cs) and contractual staff (Table 8).

O&M represented 18% of the total program costs. These costs consistently decreased over this time period, from 30% to 13%, although how these operational costs were reduced is unclear. In 2014–15, O&M savings were redirected primarily into increased G&Cs, and, in the following year, into salaried and contractual staffing costs. Spending on G&Cs was 58% of the overall program costs, while expenditures on staff and benefits were 17%. As of March 2017, the Blood R&D Program had a staff complement of 55.51 FTEs, including 20.5 in research, 22.61 in products and process development, 4 in knowledge mobilization and education, 4.2 in policy research and leading practices, 2.2 in international collaboration, and 2 for the director and her support. This staff is funded by Health Canada, P/T Ministries of Health, and CBS. Health Canada’s management of the Blood R&D Program requires minimal resources: 0.25 FTE, including the evaluation budgeted for 2017–18.

Table 7 : Variance between planned spending & expenditures - Blood R&D Program 2013-14 to 2016-17

Year	Planned Spending ($)					Expenditures ($)					Variance ($)	% planned budget spent
	G&CsTable 7 Footnote 1	O&MTable 7 Footnote 2	Salary & Benefits	Contractual Personnel	TOTAL	G&CTable 7 Footnote 1	O&MTable 7 Footnote 2	Salary & Benefits	Contractual Personnel	TOTAL
2013-14	$1,950,558	$1,386,560	$1,259,116	$403,766	$5,000,000	$2,231,425	$1,521,012	$853,974	$393,589	$5,000,000	$0	100%
2014-15	$3,000,000	$925,915	$744,985	$329,100	$5,000,000	$3,142,582	$806,626	$742,502	$308,290	$5,000,000	$0	100%
2015-16	$2,900,039	$1,034,963	$766,739	$298,259	$5,000,000	$2,908,683	$648,712	$974,582	$468,023	$5,000,000	$0	100%
2016-17	$2,903,926	$913,069	$822,121	$535,884	$5,175,000	$3,391,046	$665,064	$732,624	$386,266	$5,175,000	$0	100%
Source: CBS. Table 7 Footnote 1 An intramural grant program awards a portion of these funds internally to CBS researchers each year. Return to table 7 footnote 1 referrer Table 7 Footnote 2 O&M includes: travel and accommodation, materials and supplies, rent and utilities, training/recruitment/professional dues, legal fees associated with patents, and machinery and equipment. Return to table 7 footnote 2 referrer

Table 8 : Allocation of expenditures - Blood R&D Program 2013-14 to 2016-17

Year	Percentage of Total Expenditures (Percentage change from planned spending)
	G&Cs	O&MTable 8 Footnote 1	Salary & Benefits	Contractual Personnel
2013-14	45%	30%	17%	8%
2014-15	63%	16%	15%	6%
2015-16	58%	13%	19%	9%
2016-17	68%	13%	15%	8%
2013-14 to 2016-17	58%	18%	17%	8%
Source: CBS. Table 8 Footnote 1 Due to rounding, totals may not sum to 100% Return to table 8 footnote 1 referrer Table 8 Footnote 2 O&M includes: travel and accommodation, materials and supplies, rent and utilities, training/recruitment/professional dues, legal fees associated with patents, and machinery and equipment. Return to table 8 footnote 2 referrer

Observations on efficiency

Program representatives (CBS and Health Canada) and external key informants generally believe the Blood R&D Program operates efficiently. They cite numerous factors, including the following:

• the relatively large number of projects undertaken with a modest amount of funding (137 projects in 2014–15; 160 in 2015–16; and 151 in 2016–17);^{Footnote 27}
• the small proportion of program funds spent on O&M (13% by 2015–16);
• strong program leadership;
• purposeful selection of research projects aligned with stakeholder priorities;
• changes instituted following a recent program review; and
• collaboration with and leveraging of activities and projects undertaken by other stakeholders.

Between 2013–14 and 2016–17, performance data show that the Blood R&D Program engaged in a total of 141 distinct partnership or collaboration activities with 75 unique organizations, primarily industry and private sector partners, and academic institutions. The Blood R&D Program also collaborates frequently with the Canadian Institute of Health Research (CIHR), and delivers one of its two grant programs in partnership with the organization. As another example of leveraging, the Blood R&D Program and Mitacs, a non-profit organization that supports applied and industrial research in mathematical sciences and associated disciplines, co-fund graduate fellows for research projects to address CBS’ key business challenges. Collaborations and partnerships contribute to efficiency, since they enable the Program to access in-kind contributions, as well as (to a more limited extent) financial support to supplement the resources it receives from Health Canada.

In addition to the above factors, a few key informants commented positively on the availability of funding for small-scale applied studies in the area of transfusion medicine which might not otherwise be eligible for financial support, as well as the value of continuing to offer “seed funding” to support preliminary or exploratory research that could serve as the basis for a more comprehensive funding application.

Key informants identified relatively few opportunities to further promote efficiency and economy within the context of the Blood R&D Program’s existing resources. Some argued that CBS could invest additional effort in collaborating, communicating and partnering with other stakeholders, both domestically and internationally, and further leveraging their activities. For example, some Health Canada and Public Health Agency of Canada representatives expressed interest in improving or expanding information-sharing with CBS, as it relates to blood research. Others suggested there may be opportunities to further promote and publicize research funded through the Blood R&D Program, as well as its outcomes.

Few interviewees described alternatives to the current approach. Health Canada representatives noted that program funding could conceivably be reallocated to support similar research within the department, or through CIHR. However, external key informants were doubtful that a dedicated program of blood R&D, which they perceive as necessary to maintaining the safety of the blood supply, would be sustained under such a scenario.

Many key informants made a case for additional funding for the Blood R&D Program, citing the increasing costs of doing research. Indeed, the previous evaluation found that the Program’s research costs were increasing without commensurate increases in funding (Health Canada, 2013). Although it was not a formal recommendation, the evaluation suggested that funding would need to be increased, or alternative sources leveraged. Although CBS has not identified alternative sources of funding, it has increased its leveraging and collaborative work with other organizations, resulting in more in-kind contributions, as well as, in a limited number of instances, direct financial support from partners. It also participated in efforts, through the Canadian Blood Utilization Collaborative (CBUC), to review and prioritize clinical research programs in blood utilization, and determine the funding avenues to best support these initiatives.^{Footnote 28}

Observations on governance

Both Health Canada and CBS believe program governance and working relationships are effective. The governance structure involves oversight from Health Canada, the CBS Board and Executive, the Scientific and Research Advisory Committee (SRAC) (which consists of scientific experts in transfusion from Canadian and international organizations and academic institutions), a Research Ethics Board, an external independent peer review panel, and an internal product development committee. The SRAC is consulted with respect to priority-setting and strategy development for the Blood R&D Program. In addition, CBS key informants indicated that the Program becomes aware of federal priorities through informal channels. At the present time, there is no formal mechanism by which federal partners (and more specifically BGTD, as the regulator) are consulted on research priorities.^{Footnote 29} On the other hand, this does not appear to be an issue from the perspective of BGTD. In addition, while acknowledging that its subject matter expertise puts CBS in the best position to identify priorities for transfusion research, some P/T representatives suggested that it would be helpful to have a forum through which to request or recommend research to address needs within their respective jurisdictions. CBS representatives noted that the Blood R&D Program uses informal mechanisms for obtaining P/T input on research priorities, including sharing information about its research activities with the P/Ts several times per year, and conducting direct outreach activities.

Observations on adequacy and use of performance measurement data

The Blood R&D Program has a PMS in place that was last updated in January 2016. Review of performance measurement data suggests the Program is systematic and complete in its tracking of output and outcome information. However, a few minor improvements could be implemented, such as tracking other types of bibliometric measures, in addition to the h-index score, to measure the impact of the Program’s published work.^{Footnote 30} As another example, although the Blood R&D Program is systematic in its collection of feedback information from key KE events, some of these surveys do not collect information on knowledge gained as a result of the events. CBS recently undertook a survey pertaining to the uptake and application of knowledge generated and disseminated through the Blood R&D Program (selected results have been incorporated into this report), and intends to administer such surveys on a biennial basis going forward.

5.0 Conclusions

5.1 Relevance

5.1.1 Continued Need

Evidence available to this evaluation indicates a continued need to address a persistent imbalance in Canada between demand for organs/tissues, and available supply. The transplant waitlist has exceeded 4,500 persons since 2010, and in 2015, although 2,580 transplants were performed, there were 4,713 patients on organ waitlists at the end of that year, with an average wait time of 430 days. To the extent that centralized coordination (the main objective of the OTDT Program) is a common feature of OTDT systems in Canada’s peer nations with significantly better OTDT system performance, Canada’s performance might likewise improve with better coordination, although it is important to acknowledge the possibility that some elements of coordination may be more critical than others to improved system performance. Similarly, the evaluation found a continued need for ongoing blood research and development (R&D) to respond to emerging threats, while taking advantage of opportunities to further promote the safety of the blood system in Canada. Moreover, an R&D function is a common feature of the national blood supply systems maintained in jurisdictions comparable to Canada.

5.1.2 Alignment with government priorities

Although federal Budgets and Speeches from the Throne since 2013 have not explicitly mentioned either OTDT or blood R&D, both the OTDT and Blood R&D Programs align with the Government’s commitment to innovation in the health care system, through funding of health innovation research and innovation in health services delivery, and the Blood R&D Program aligns with the Government of Canada’s commitment to the safety of the blood system. In addition, both programs align with the October 2017 Minister’s Mandate Letter, which highlighted the Government’s commitment for the Minister of Health to work with the provinces and territories to “develop a long-term vision for blood services that ensures safety and non-discrimination in donation policies, and facilitate collaboration on an organ and tissues donation and transplantation system that gives Canadians timely and effective access to care”.

Both programs align with and support Health Canada’s Strategic Outcome #2: Canadians are informed of and protected from health risks associated with food, products, substances and environments, and are informed of the benefits of healthy eating. In particular, the programs support the Biologics and Radiopharmaceuticals sub-Strategic Activity within the Health Products Strategic Activity. The Blood R&D Program helps to protect Canadians from health risks associated with the use of biological products by conducting research that informs evidence-based practices aimed at ensuring the safety of the Canadian blood system. Similarly, by engaging in leading practice development and public awareness activities relating to OTDT, the OTDT Program helps to inform and protect Canadians from health risks associated with organ and tissue transplantation.

5.1.3 Alignment with federal roles and responsibilities

Both the OTDT and Blood R&D Programs align with federal roles and responsibilities as defined by federal legislation, including the Department of Health Act, the Canada Health Act, the Food and Drugs Act, and related regulations.

Health Canada representatives noted that the federal role in the OTDT Program was intended to be a temporary support to the P/Ts for the development and implementation of a coordinated OTDT system. , In contrast, external stakeholders, including P/Ts, support an ongoing, or even an enhanced role for Health Canada in the OTDT Program, although they did not agree on the precise nature of that role. Furthermore, many argued that P/T involvement in the Program is bolstered by Health Canada’s participation, and that the withdrawal of Health Canada’s support might well erode P/T commitment, with negative consequences for the OTDT system as a whole.

While some of the OTDT Program’s activities, such as public awareness messaging, professional education, and development of leading practices, may overlap with activities undertaken by other stakeholders; they may equally be seen as complementary. In particular, activities that appear to duplicate work in larger, well-funded jurisdictions may also be addressing gaps in smaller, less well-resourced jurisdictions that lack the capacity to sustain those activities themselves. Furthermore, by virtue of their national scope, the activities of the OTDT Program are intended to add value beyond the benefits resulting from analogous P/T activities.

Health Canada’s role in the Blood R&D Program is largely uncontroversial from the perspective of stakeholders. Although some research currently funded by the Blood R&D Program would be eligible for funding through the Canadian Institutes of Health Research (CIHR), the Natural Sciences and Engineering Research Council (NSERC), provincial agencies and other organizations, stakeholders believe competition for these funds is such that it could be difficult to sustain ongoing Canadian research in this area without a dedicated program.

In response to a recommendation from the last evaluation, a focus on operational and regulatory issues was built into the Blood R&D Program’s current funding agreement. In several instances over the period covered by the evaluation, Canadian Blood Services (CBS) used program findings to inform submissions to Health Canada for changes to its internal policy or operational practice, which were subsequently approved by the department. That said, representatives of BGTD, which is responsible for regulating biologics in Canada, reported that they relyi on the Directorate’s internal research, research undertaken by the Public Health Agency of Canada (PHAC), and international regulators to inform their work. However, BGTD representatives were open to learning more about the Blood R&D Program’s research activities, with a view to identifying potential opportunities for collaboration among CBS and BGTD researchers.

5.2 Performance

5.2.1 Achievement of Expected Outcomes

Both the OTDT and Blood R&D Programs are making progress toward their immediate, intermediate, and longer-term outcomes. However, the OTDT Program has encountered some challenges in a number of areas.

Immediate outcomes

• Collaboration among OTDT partners. Through its various activities and advisory committee structure, the OTDT Program collaborates with a diverse range of stakeholders, primarily on the front-line, but has encountered challenges in engaging effectively with P/Ts. At present, CBS and the P/Ts have divergent perspectives regarding the need for an integrated, national OTDT system and differing levels of commitment to it. To date, the Program has been unable to deliver on the Health Canada-funded activity to develop a national governance model and accountability framework. CBS key informants reported that program are working to advance relationships with P/T partners, and national accountability discussions are ongoing between CBS and the P/Ts.
• Stakeholders are knowledgeable about leading practices, evidence, and knowledge generated by the programs. Large majorities of surveyed stakeholders of the OTDT and Blood R&D Programs are aware of the knowledge and evidence generated by these programs. However, there are opportunities to increase  awareness of the Blood R&D Program’s research activities among certain partners within the federal health portfolio, in particular BGTD and PHAC. Health Canada has begun taking steps, in response to the previous evaluation, to more actively promote the exchange of knowledge generated by the Blood R&D Program within the federal health portfolio, and plans to continue doing so in the future.
• Public and patient awareness and understanding of OTDT. In collaboration with other stakeholders, the OTDT Program has undertaken various initiatives to improve public and patient awareness and understanding of OTDT. Public opinion research carried out by CBS indicates that most measures of public and patient awareness and understanding of OTDT have remained stable between 2012 and 2017. The level of awareness varies depending on the measure. Most importantly from the perspective of addressing the gap in Canada between demand for organs and tissues and the available supply, the proportion of Canadians who plan to donate their organs and tissues at the time of death has hovered around 50% over this time period.

Intermediate outcomes

• Highly qualified personnel (HQP) participate in transfusion science and medicine. The Blood R&D Program has supported the training of HQP in transfusion science and medicine through graduate fellowship programs, post-doctoral fellowship programs, and provision of training positions in CBS research laboratories. Over the period covered by the evaluation, 82 people completed training for HQP through the Blood R&D Program.
• Stakeholders use R&D generated knowledge and adopt new or modified OTDT knowledge and practices. Large majorities of surveyed stakeholders who are aware of the activities and knowledge generated by the OTDT and Blood R&D Programs have used this knowledge to inform their work. Knowledge generated by the programs has been used to develop or modify policy and practice, develop training or educational curricula, and design research, among other things. The perceived quality and relevance of the knowledge generated by the two programs is high.
• OTDT activities are integrated across jurisdictions. Some progress has been made in integrating OTDT activities across jurisdictions, most notably through the provincially-funded Canadian Transplant Registry (CTR) and related initiatives, but there is less evidence of integration resulting from federal funding. Respondents to the survey of OTDT Program stakeholders and many key informants have expressed considerable support for CBS to continue its efforts to achieve a nationally coordinated and integrated OTDT system, while respecting P/T activity and jurisdiction in the area of OTDT.

Longer-term outcomes

• Contribution to the quality of OTDT systems across jurisdictions. Many OTDT stakeholders believe the quality of OTDT systems across jurisdictions has improved because of CBS’ activities. However, objective data are relatively limited at the present time, and disentangling CBS’ contribution from that of other stakeholders is challenging.
• Contribution to the confidence of stakeholders in the OTDT system. Public opinion surveys conducted between 2014 and 2017 show moderately high levels of public trust in the OTDT system. Among OTDT stakeholders surveyed, just over half believe stakeholder confidence in the OTDT system has improved due to the activities of CBS.
• Contribution to the safety, quality and supply of blood and blood products. Canadian surveillance data indicate low prevalence rates of select blood-borne pathogens among first-time donors. While the extent to which the Blood R&D Program had a bearing on these long-terms trends, relative to other factors is unknown, it is reasonable to assume that it is contributing to the safety, quality and supply of blood and blood products through its research activities and training of HQP in transfusion medicine.

5.2.2 Demonstration of Economy and Efficiency

The OTDT Program has taken some steps to lower costs but has experienced challenges in operating efficiently. In some cases, progress in generating outputs and outcomes has been slower than expected. For example, the CBS-CIHI data transfer initiative is not as far advanced as had been hoped, and the Program has been unable to deliver on the Health Canada-funded project to develop a national governance model and accountability framework for the OTDT system. The lack of progress with respect to governance, along with P/T questions about the need for a national model, may suggest that Health Canada needs to review this aspect of its funding to the OTDT Program. In other cases, program achievements are difficult to discern due to limited performance data, and a lack of clarity and transparency in performance reporting. For example, there is a lack of performance measurement data on uptake of leading practices, as well as a lack of clarity with regard to the number of leading practices developed and updated during the evaluation period.

Improvements in OTDT Program governance and working relationships could facilitate more efficient delivery of program activities, and progress on program objectives. Within the narrow context of the funding agreement between Health Canada and CBS, key informants representing both organizations identified areas for improvement. CBS believes Health Canada could more actively demonstrate its support for the OTDT Program to other stakeholders, including the P/Ts. Health Canada would like improved transparency and timeliness in CBS’ provision of information to them. Within the broader context of governance of the F/P/T OTDT Program and the OTDT system in general, as noted above, there are a number of unresolved differences of perspective among the implicated parties with regard to overall vision, roles, responsibilities, and accountability.

Internal and external stakeholders believe the Blood R&D Program operates efficiently and economically, citing factors such as the large number of projects undertaken with a modest amount of funding, the relatively small proportion of program funds spent on O&M, strong program leadership, and selection of research projects aligned with stakeholder priorities. In addition, although CBS has not identified alternative sources of funding, it has increased its leveraging and collaborative work with other organizations (e.g., CIHR, Mitacs). This has resulted in more in-kind contributions, as well as a limited number of instances of direct financial support from partners. Both Health Canada and CBS believe current program governance and working relationships are effective. The Blood R&D Program is systematic and complete in its tracking of output and outcome information.

6.0 Recommendations

Recommendation 1

With respect to the Blood R&D Program, improve knowledge exchange of research projects and results among partners in the federal Health Portfolio to optimize awareness and uptake.

There are opportunities to increase awareness of the Blood R&D Program’s research activities among certain partners within the federal health portfolio, in particular BGTD and the Public Health Agency of Canada as there is interest among these partners in learning more. While Health Canada has begun taking steps, in response to the previous evaluation, to more actively promote the exchange of knowledge generated by the program within the federal health portfolio, further efforts in this area should optimize awareness and uptake of research findings, as well as identify areas for potential collaboration.

Recommendation 2

With respect to the OTDT Program, and in fulfillment of the October 2017 Minister’s Mandate Letter, Health Canada should facilitate collaboration with key stakeholders on the long-term objective of a comprehensive Canadian OTDT system and define Health Canada’s role in developing such a system.

While the federal role in the OTDT Program was intended to be a temporary support to the P/Ts for the development and implementation of a coordinated OTDT system, external stakeholders, including P/Ts, support an ongoing or even an enhanced role for Health Canada in the OTDT Program. Given the challenges the OTDT Program has encountered in delivering on some Health Canada-funded activities, and in light of the recent direction given to the Minister of Health to work with the provinces and territories to facilitate collaboration on an OTDT system that gives Canadians timely and effective access to care, Health Canada should consult with key stakeholders on the objective of a comprehensive Canadian OTDT system and define Health Canada’s role in developing such a system.

Recommendation 3

With respect to the OTDT Program, improve consistency and clarity in the collection and reporting of performance data.

In many cases, the evaluation was challenged in determining what the OTDT Program has achieved, due to limited performance data and/or lack of clarity/transparency in performance reporting. One notable example is the lack of performance measurement data on uptake of leading practices as well as a lack of clarity with regard to the number of leading practices developed and updated during the evaluation period. The OTDT Program should take steps to improve consistency and clarity in the collection and reporting of performance data in order to better demonstrate progress toward expected outcomes.

Appendix 1 – Logic Models

The logic model for the Canadian Blood Services’ OTDT Program is shown in Figure 2.

Figure 2 : CBS OTDT Program Logic Model

Inputs	Health Canada Funding		In-Kind support from Canadian Blood Services
↓
Key Activities	National governance, policy and planning	Leading practice development, knowledge translation, health professional education	Public awareness and education	National performance analysis and reporting
↓
Outputs	Strategic partnerships/collaborations		Knowledge products/knowledge exchange mechanisms
↓
Immediate Outcomes	Partners collaborate on OTDT	Health professionals and decision makers are knowledgeable about OTDT leading practices and knowledge/evidence	Public and patients are aware of and understand OTDT
↓
Intermediate Outcomes	OTDT related activities are coordinated and integrated across jurisdictions		Relevant stakeholders use/ adopt new/ modified knowledge and practices
↓
Longer-term Outcomes	Contribution to the quality of OTDT system across jurisdictions		Contribution to the confidence of relevant stakeholders in the Canadian transplant system

The logic model for the Canadian Blood Services’ Blood R&D Program is shown in Figure 3 .

Figure 3 : CBS Blood R&D Program Logic Model

Inputs	CBS Centre of Innovation Funding and HR dedicated to R&D Program activities		CBS Operations Expertise dedicated to the blood system supply chain	CBS Shared Services (IT, Finance)	External Partnerships and Collaborations Funding and HR dedicated to the blood system	Funding (HC)
↓
Activities	Funding priority research and development programs and projects	Knowledge mobilization Translate and Exchange Knowledge	Training HQP and building a community of experts in transfusion science and medicine	Networking Facilitate collaborative working relationships	Program Management (HC)	Partnerships Collaborative working arrangements (CBS/HC)
↓
Outputs	Grants and awards (CBS)	Knowledge Exchange mechanisms (CBS/HC)	Knowledge Products (CBS)	Partnerships/ collaborative working arrangements (CBS/HC)
↓
Immediate Outcome	Highly Qualified Personnel (HQP) participate in transfusion science and medicine in Canada	Key stakeholders in the transfusion and transplantation community are knowledgeable about the evidence/ knowledge generated by R&D projects
↓
Intermediate Outcome	Key stakeholders in the transfusion and transplantation community apply knowledge created by R&D projects
↓
Longer-Term Outcome	Contribution to the safety, quality and supply of blood and blood products

Appendix 2 – Evaluation Description

Evaluation Scope

The evaluation focused on, but was not confined to, the period from the period from 2013–14 to 2016–17, and included all activities within the Organ and Tissue Donation and Transplantation (OTDT) Program and the Blood Research and Development (R&D) Program.

Evaluation Issues

The evaluation issues were aligned with the Treasury Board of Canada’s Policy on Results (2016) and considered the five core issues under the two themes of relevance and performance. Corresponding to each of the core issues, specific questions were developed based on program considerations and these guided the evaluation process.

Table 1: Core evaluation issues and questions

Core Issues	Evaluation Questions
Relevance
Issue #1: Continued Need for Program	Has the need for the Canadian Blood Services (CBS) contribution programs changed since 2012-13?
Issue #2: Alignment with Government Priorities	Do the objectives and activities of the CBS contribution programs continue to align with federal government priorities? Do the objectives and activities of the CBS contribution programs continue to align with the strategic outcomes of Health Canada?
Issue #3: Alignment with Federal Roles and Responsibilities	Has there been a change in how the CBS contribution programs align with federal roles and responsibilities? Is the role of the various partners clear? To what extent do the CBS contribution programs link with policy work at Health Canada? Are there other programs or initiatives that complement, overlap or duplicate the objectives of these funded programs?
Performance (effectiveness, economy, and efficiency)
Issue #4: Achievement of Expected Outcomes (Effectiveness)	To what extent have the CBS contribution programs produced expected immediate, intermediate, and long-term outcomes?
Issue #5: Demonstration of Economy and Efficiency	Have the CBS contributions been implemented in the most efficient and economical manner? How, and in what ways, can economy and efficiency be improved? To what extent are governance and working relationships clear, effective and efficient for achieving expected results (e.g., within Health Canada, and between Health Canada and its contribution recipients) Are there alternative ways to achieve similar results at a lower cost? Have alternative funding models for R&D activities been explored and implemented? Are the CBS contribution agreements clear on what they are trying to achieve and how they assess success? How, and to what extent, was performance assessed and the resulting information used?

Data Collection and Analysis Methods

Evaluators collected and analysed data from multiple sources, including literature review, document review, review of administrative and performance measurement data, surveys of OTDT and Blood R&D Program stakeholders, and key informant interviews.

Literature review. The literature review examined information from peer-reviewed (academic) sources as well as grey literature external to the Government of Canada. The scope of the literature review was fairly limited, and focused primarily on assessing the extent to which there is a continued need for the OTDT and Blood R&D Programs.

Document review. The document review provided historical and contextual information for the programs, and responded directly to the majority of the evaluation questions, as indicated in the evaluation matrix. The review encompassed documents and files provided by the Office of Audit and Evaluation, as well as publicly available information.

Analysis of administrative and performance measurement data. This task included analysis of financial information to support the analysis of efficiency and economy, as well as analysis of performance information collected by the programs.

Surveys of OTDT and Blood R&D Program stakeholders. A bilingual, web-based stakeholder survey was conducted for each program.

The survey of OTDT Program stakeholders was conducted as part of the evaluation data collection activities. The survey sample was provided by CBS. After cleaning to remove duplicates and invalid addresses, the final sample consisted of 566 email addresses. The survey was launched July 10, 2017 and closed on July 31, 2017. Three rounds of reminders were issued to increase the response rate. The survey achieved 104 completions, representing a completion rate of 18%. The survey results were analysed using SPSS.

The survey of Blood R&D Program stakeholders was conducted outside of the context of the evaluation. As with the OTDT survey, the survey sample was compiled by CBS, and the final list consisted of 114 stakeholders. The survey was launched on April 20, 2017 and closed on May 17, 2017. The survey achieved 56 completions, representing a completion rate of 49%.

Key informant interviews. A total of 32 interviews involving 47 key informants were conducted. The table below provides a breakdown by key informant category.

Key Informant Categories

Category of key informant	Number of interviews	Number of key informants
External stakeholder	20	20
CBS representative	4	6
Federal partner	4	6
P/T representative	1	9
Health Canada SPB representative	3	6
Total	32	47

Interviews were recorded with the permission of key informants, and interview notes were returned to them for review and sign-off. The interview data were analysed using NVivo.

Approach to analysis

Data were analysed by triangulating information gathered from the different sources and methods listed above, which included the following:

• systematic compilation, review, and summarization of data to illustrate key findings;
• quantitative analysis of administrative/financial data, including trend analysis over time;
• thematic analysis of qualitative data; and
• comparative analysis of data from disparate sources to validate findings.

Reference List

• CBS. (2012). Call to action: A strategic plan to improve organ and tissue donation and transplantation performance for Canadians. Retrieved from https://blood.ca/sites/default/files/otdt-indx-final-c2a.pdf
• CBS. (2015). Surveillance Report 2015.
• CBS. (2016a). Bioburden Reduction and Control in Tissue Banking: Leading Evidence Based Practice Guidelines (Final Report). Retrieved from https://professionaleducation.blood.ca/sites/msi/files/bioburden_leading_practice_guidelines_final_report_november_2016.pdf
• CBS. (2016b). Highly Sensitized Patient (HSP) Program. Retrieved August 24, 2017, from https://professionaleducation.blood.ca/en/organs-tissues/program-overview/highly-sensitized-patient-hsp-program
• CBS. (2016c). Organ donation and transplantation in Canada: System progress report 2006-2015. Retrieved from https://blood.ca/sites/default/files/ODT_Report.pdf
• CBS. (2017). MSM Research Grant Program Guidelines. Retrieved from https://blood.ca/sites/default/files/MSMResearch_Grant_Program_2017Guidelines.pdf
• CIHI. (2014). Deceased organ donation potential in Canada. Retrieved from https://secure.cihi.ca/free_products/OrganDonorReport_ENweb.pdf
• CIHI. (2016a). Canadian organ replacement register update. Retrieved from https://www.cihi.ca/sites/default/files/corr_summer_update_en02mc_0.pdf
• CIHI. (2016b). Organ Donations Fall Short of Meeting Demand. Retrieved August 24, 2017, from https://www.cihi.ca/en/organ-donations-fall-short-of-meeting-demand
• CIHI. (2017a, February 27). OECD Interactive Tool: International Comparisons, Peer Countries, Alberta | CIHI. Retrieved April 7, 2017, from https://www.cihi.ca/en/health-system-performance/performance-reporting/international/oecd-interactive-tool-peer-countries-ab
• CIHI. (2017b, March 2). e-Statistics On Organ Transplants, Waiting Lists And Donors | CIHI. Retrieved March 21, 2017, from https://www.cihi.ca/en/e-statistics-on-organ-transplants-waiting-lists-and-donors
  Government of Canada. (2015a). Safety of Human Cells, Tissues and Organs for Transplantation Regulations (SOR/2007-118). Retrieved May 11, 2017, from http://laws-lois.justice.gc.ca/eng/regulations/SOR-2007-118/
• Government of Canada. (2015b, April 21). Budget 2015 - Budget Plan: Chapter 4.2 - Strong Communities. Retrieved March 30, 2017, from http://www.budget.canada.ca/2015/docs/plan/ch4-2-eng.html#Investing_in_the_Health_of_Canadians
• Government of Canada. (2016a). Statement from the Minister of Health on one year blood donor deferral period for MSM. Retrieved from http://www.newswire.ca/news-releases/statement-from-the-minister-of-health-on-one-year-blood-donor-deferral-period-for-msm-583664611.html
• Government of Canada. (2016b, March 22). Budget 2016: Chapter 5 - An Inclusive and Fair Canada. Retrieved March 30, 2017, from http://www.budget.canada.ca/2016/docs/plan/ch5-en.html#_Toc446106794
• Health Canada. (2013). Evaluation of Canadian Blood Services (CBS) Grant and Contribution Programs: Evaluation Management Response and Action Plan.
• Health Canada. (2015, February 5). Guidance Document: Blood Regulations [notice]. Retrieved March 31, 2017, from http://www.hc-sc.gc.ca/dhp-mps/brgtherap/applic-demande/guides/blood-reg-sang/blood-guid-sang-ligne-2014-10-23-eng.php#a1.1
• HLWIKI Canada. (2017). HLWIKI Canada: Author impact metrics. Retrieved May 11, 2017, from http://hlwiki.slais.ubc.ca/index.php/Author_impact_metrics
• IRODaT. (2017). International Registry on Organ Donation and Transplantation. Retrieved October 30, 2017, from http://www.irodat.org/?p=database
• Prime Minister of Canada. (2017, August 28). Minister of Health Mandate Letter. Retrieved from http://pm.gc.ca/eng/minister-health-mandate-letter
• US Department of Health and Human Services. (2017). Hematology Branch - NHLBI, NIH. Retrieved April 6, 2017, from https://www.nhlbi.nih.gov/research/intramural/organization/center-branch/hematology-branch
• WHO. (2013). Assessment criteria for national blood regulatory systems. Retrieved March 31, 2017, from http://apps.who.int/medicinedocs/documents/s21094en/s21094en.pdf