Evaluation of Canadian Blood Services Grant and Contribution Programs 2008-2009 to 2012-2013

September 2013

List of Acronyms

CBS

Canadian Blood Services

CCDT

Canadian Council for Donation and Transplantation

CIHI

Canadian Institute for Health Information

CIHR

Canadian Institutes of Health Research

F/P/T

Federal/Provincial/Territorial

FY

Fiscal Year

G&C

Grant and Contribution

GoC

Government of Canada

HC

Health Canada

HLA

Human Leukocyte Antigen

HPFB

Health Products and Food Branch

KT

Knowledge Transfer

LDPE

Living Donor Paired Exchange (registry)

netCAD

Network Centre for Applied Development

ODTEAC

Organ Donation and Transplantation Expert Advisory Committee

OPO

Organ Procurement Organization

OTDT

Organ and Tissue Donation and Transplantation

PAA

Program Alignment Architecture

PHAC

Public Health Agency of Canada

PMD

Program Management Division

P/T

Provincial/Territorial

R&D

Research and Development

SPB

Strategic Policy Branch

Table of Contents

• Executive Summary
• Management Response and Action Plan
• 1. Introduction
• 2. Program Description
  • 2.1 Program Profile
  • 2.2 Program Logic Model
  • 2.3 Program Alignment and Resources
    
• 3. Evaluation Description
  • 3.1 Evaluation Scope
  • 3.2 Evaluation Issues
  • 3.3 Evaluation Approach
  • 3.4 Evaluation Design
  • 3.5 Methods
  • 3.6 Limitations
    
• 4. Evaluation Findings
  • 4.1 Relevance
  • 4.2 Performance
    
• 5. Conclusions
  • 5.1 Relevance Conclusions
  • 5.2 Performance Conclusions
    
• 6. Recommendations
• Annex A: Program Description

List of Tables

• Table 1 CBS OTDT Contribution Planned and Actual Expenditures ('000), FY 2008-09 to 2012-13
• Table 2 CBS R&D Grant Planned and Actual Expenditures ('000), FY 2008-09 to 2012-13
• Table 3 Core Evaluation Issues and Questions
• Table 4 Overview of Data Collection Methods
• Table 5 Limitations and Mitigation Strategies
• Table 6 How appropriate were the CBS activities undertaken in the following areas in order to improve the OTDT system?
• Table 7 Based on your knowledge of the CBS R&D Program, to what extent do the following statements apply to the Program?
• Table 8 Based on your knowledge of the OTDT-related activities, to what extent have these activities contributed to the following?
• Table 9 Were the following results generated by CBS-supported research since 2008?
• Table 10 Distribution of CBS OTDT Expenditure by Component (Administration and Pillars), FY 2008-09 to 2011-12 and Planned Expenditures (FY 2012-13)
• Table 11 Distribution of CBS R&D Actual Grant Expenditure by Component (Schedule), FY 2008/09 to 2011/12 and Planned Expenditures (FY 2012-13)

Executive Summary

This evaluation covered the Canadian Blood Services Grant and Contribution Programs for the period from 2008-09 to 2012-13. The evaluation was undertaken in fulfillment of the requirements of the Financial Administration Act and the Treasury Board Policy on Evaluation (2009).

Evaluation Purpose and Design

The purpose of the evaluation was to assess the relevance and performance of the Programs. The evaluation conclusions and recommendations were developed to support evidence-based decision-making with regard to Health Canada's Canadian Blood Services Grant and Contribution Programs. Preliminary results were made available to Health Canada to inform the renewal process of the Programs in December 2012.

The methodology used in the evaluation included multiple lines of evidence, including a document review, literature review, key informant interviews with stakeholders from Health Canada, Canadian Blood Services (CBS) and external groups (e.g., provincial/territorial representatives, partner organizations, subject matter experts), and two surveys with program stakeholders and funding recipients.

Program Description

Health Canada supports two specific CBS Programs through a grant and a contribution: the Organ and Tissue Donation and Transplantation (OTDT) Program and the Blood Research and Development (R&D) Program.

For the OTDT Program, CBS signed a Contribution Agreement with Health Canada in 2008, representing $17.9 million over five years ($3.58 million per year). The overall goal of Health Canada is to support the development of a national organ and tissue donation and transplantation system that will improve and extend the quality of the lives of Canadians, while respecting the federal role and interest in organ and tissue donation and transplantation. Under this funding agreement, CBS's work has been conducted within three pillars of activity: (1) Strategic Plan Development and Implementation; (2) Support Leading Practices / Public Awareness and Education; and (3) System Performance Improvement.

Provinces and territories (P/Ts) (except Quebec) support a fourth initiative, the development and operation of three national patient registries, for an additional $3.58 million per year. These registries are not covered by this evaluation.

For the R&D Program, Health Canada has provided a grant of $5 million per year for a total of $25M over five years, with approximately the same amount provided by P/Ts ($25M over five years). As a result, CBS has received a total of $50M in funding for R&D. With that funding, the CBS R&D Program seeks to drive continuous improvement and knowledge creation through research and development work into blood safety, alternatives to transfusions, and blood substitutes. This work has been conducted across five functions: (1) Research; (2) Development; (3) Intellectual Property & Business Development; (4) Quality Monitoring; and (5) Research and Education (as of 2011-12).

Conclusions and Recommendations

Relevance

Both components of the CBS Grant and Contribution Programs continue to address a demonstrable need and remain responsive to the needs of Canadians.

The range of activities and priority areas pursued by the OTDT Program are appropriate and well aligned with the ongoing need to improve the performance of the Canadian OTDT system. This includes improving health outcomes, as well as improving access, fairness and transparency for all patients on organ waitlists. As a result, there is a continued need to strengthen and ensure a nationally consistent Canadian OTDT system to enhance access to organs and tissues and improve public awareness of the transplant system.

For the CBS R&D Program, although the Canadian blood system is now widely recognized as safe, there continues to be a need for scientific and evidence-based information to maintain and increase the safety, supply and efficiency of the Canadian blood system. R&D Program activities are generally well aligned with related operational, regulatory, knowledge, and capacity needs.

CBS is widely seen as the organization best positioned to respond to the needs associated with both the OTDT and R&D Programs. However, the OTDT Program is currently in a transition phase, as it awaits direction from P/Ts on the implementation of recommendations, outlined in the strategic plan called the Call to Action (2011) developed by the CBS. The Call to Action is an integrated, cost-shared plan that identifies gaps in the current Canadian system and provides a set of 25 recommendations which will impact its future role and mandate, particularly for the tissue donation and transplantation program. Meanwhile, the R&D Program is under increasing pressure from stakeholders to focus to a greater extent on addressing operational and regulatory needs.

Both components of the CBS Grant and Contribution Programs are aligned with government priorities.

Both programs are aligned with the Government of Canada's priority and related work to protect the health and safety of Canadians, as well as Health Canada's strategic outcome to support access to safe and effective health products and protect Canadians from health risks. The OTDT Program aligns with Government of Canada and Health Canada health care system priorities, such as working with P/Ts to ensure the sustainability and accountability of the health system, and ensuring the system responds to the needs of Canadians.

Both components of the CBS Grant and Contribution Programs are aligned with federal roles and responsibilities but there is room for further development.

The CBS Grant and Contribution Programs' activities fall within the federal role, including Health Canada's mandate, regulatory role, and jurisdictional responsibilities for the regulation of biological health products. They also align with the federal role to ensure that all Canadians are provided fair and equitable access to health care, in collaboration with the provinces and territories, in accordance with the five principles of the Canada Health Act.

The nature of Health Canada's role for the OTDT Program (other than providing funding and oversight of the Contribution Agreement) was subject to conflicting expectations and different views with respect to the appropriate level of direct involvement and provision of "national leadership". On the other hand, closer involvement and knowledge sharing of Health Canada with CBS' R&D Program was generally desired. In addition, greater opportunities for discussions of research results, agendas and needs, that involve both key blood R&D researchers and users (i.e., federal, provincial and territorial regulators and blood operators, including Héma-Québec), was seen as beneficial.

Performance

Both components of the CBS Grant and Contribution Programs have produced expected outputs and are making progress toward their expected immediate and intermediate outcomes. Health Canada's management of the OTDT Program has been effective, while that of the CBS R&D Program has improved significantly in recent years.

The CBS OTDT Program has generated a large volume of outputs in accordance with its work plans and intended activities. These outputs were necessary in helping the program to achieve progress towards its immediate and intermediate outcomes including: raising awareness and understanding of the OTDT Program and priorities, developing improved system performance metrics, and supporting an enhanced accreditation framework. Increased coordination and integration of OTDT activities was also observed. Similarly, progress was observed toward the adoption of new/modified knowledge, policies, standards and regulations, as well as adoption of improved health-care practices and procedures. The extent to which longer-term outcomes will be achieved is largely dependent on the direction given by P/Ts about the implementation of the recommendations in the Call to Action.

The CBS R&D Program has generated numerous outputs in the important areas of basic and applied research that support the mission of the CBS. The Program has played a key role in building and maintaining research capacity in transfusion science and medicine. Several examples of application and adoption of R&D results within and outside CBS were identified; however, the findings suggest there is room for further improvement in knowledge exchange and collaboration with research users, to facilitate greater adoption and application of knowledge or innovation results in policy and practice. CBS has already begun to focus its efforts on these areas, but it is too early to assess the success of these efforts.

Health Canada's management of the CBS OTDT has been effective over the evaluation period, while that of the CBS R&D Program has improved significantly in recent years. The outputs of these management activities have contributed to enhanced accountability and more focused program delivery for the OTDT Program and, more recently, for the R&D Program. The management activities have also helped enhance knowledge sharing and strengthened relationships within Health Canada, with other government partners in the Health Portfolio and with the CBS around issues associated with these two Programs. The evaluation found evidence that HC has mainly engaged in program management of both programs and was moving towards a more active role beyond that of funder, e.g. knowledge sharing and strengthening relationships between CBS and HC.

Both components of the CBS Grant and Contribution Programs have demonstrated economy and efficiency in their operations and management. Alternative funding models have been identified that may help CBS explore additional funding sources to support its R&D activities and meet growing expectations of its stakeholders.

The CBS grants and contributions (G&C) Programs were viewed as providing value for money, based on low program management input costs (less than 1%), favourable OTDT cost comparisons with other countries, leverage obtained from other sources (e.g., provinces/territories, universities, etc.), efficiencies gained by building on previous efforts and promoting measures to save costs, as well as stakeholder views that the Programs produce tangible results for modest costs.

There was consensus that CBS is the most appropriate organization to deliver the Programs and that contribution funding is seen as an efficient instrument for achieving the federal goals.

Alternative funding models for the R&D Program have been identified that may help CBS explore additional funding sources to meet growing expectations (e.g., to increase development or applied research efforts).

Recommendations

Two recommendations have been identified for the CBS Grant and Contribution Programs.

Recommendation 1 (OTDT and R&D):

The two CBS programs are experiencing a time of transition. The OTDT Program has developed the Call to Action (April 2011) and is awaiting a response from P/Ts. Many of the intended outcomes for the OTDT Program rely on the P/Ts to respond in full or in part to the Call to Action. Therefore, determining appropriate program objectives, taking into account possible responses to the Call to Action, will be necessary.

Health Canada has decided to strengthen its oversight by moving from a grant to a contribution agreement. Therefore, this would be an opportune time for Health Canada to refine and clarify its role as well as the program objectives.

Health Canada should determine the departmental results to be achieved with the two CBS Programs going forward and, based on this, clearly define the federal role and program objectives.

Recommendation 2 (R&D):

The current strategic direction for the CBS R&D Program is to increase knowledge mobilization activities, mainly through its Research & Education function, and to conduct more applied/operational research. The impact of these efforts should lead to an increase in the use of the R & D results.

Health Canada should look for opportunities to facilitate CBS' shift toward research and development which is focussed on operational and regulatory issues.

Management Response and Action Plan

1. Introduction

Health Canada's Canadian Blood Services Grant and Contribution Programs is a platform that the Government of Canada uses to enhance and maintain the safety of Canada's blood system and to facilitate the implementation of an organs and tissues donation system. The Canadian Blood Services has received funding from Health Canada through two streams: a contribution agreement representing $17.9 million for a five-year period ($3.58 million per year)^{Footnote 1} for the Organ and Tissue Donation and Transplantation (OTDT) Program and a grant of $25 million over five years to support the Blood Research and Development Program.^{Footnote 2}

The OTDT Program supports the development of a National System Design (NSD) for organ and tissue donation and transplantation that will improve and extend the quality of the lives of Canadians while respecting the federal role and interest in organ and tissue donation and transplantation. The NSD is subject to approval by the F/P/Ts. Provincial and territorial OTDT funding supports complementary initiatives within each jurisdiction, such as administration of registries and service delivery activities. The grant supports the Canadian Blood Services (CBS) Research & Development (R&D) program in conducting research towards the elimination of threats of insufficient supply and transfusion-borne diseases.

The evaluation of the Canadian Blood Services Grant and Contribution Programs was a scheduled evaluation on the Public Health Agency of Canada/Health Canada's Five-Year Evaluation Plan. The evaluation assessed the relevance and performance (effectiveness, efficiency, and economy) of the Program, in accordance with the Treasury Board Policy on Evaluation (2009) and to fulfill requirements of the Financial Administration Act. The evaluation covered the 2008-09 to 2012-13 fiscal years (FY). The results will inform the implementation of current and future activities of the Canadian Blood Services Grant and Contribution Programs.

The evaluation findings and recommendations were developed to support evidence-based decision-making with regard to Health Canada's Canadian Blood Services Grant and Contributions Programs. Preliminary results were made available to Health Canada for the renewal process of the Programs in December 2012.

2. Program Description

2.1 Program Profile

A brief summary of the two programs under evaluation is provided below with further details in Annex A.

2.1.1 Organ and Tissue Donation and Transplantation (OTDT) Program

The overall goal of Health Canada is to support CBS in the development of a national organ and tissue donation and transplantation system that will improve and extend the quality of the lives of Canadians while respecting the federal role and interest in organ and tissue donation and transplantation.

The objectives of OTDT Program are to:

• Develop a coordinated F/P/T strategy on organ and tissue donation and transplantation;
• Support the enhancement of standards, clinical practice guidelines, and best practices;
• Provide a forum to discuss issues including: information sharing; provincial/territorial initiatives related to donation and transplantation; ethical issues related to donation and transplantation;
• Recommend practice guidelines based on an assessment of best practices;
• Provide advice on program and system linkages and interoperability with respect to information management systems, and educational resources for interdisciplinary professionals involved in donation and transplant processes;
• Develop social marketing strategies and their implementation;
• Monitor, in accordance with its mandate only, the implementation of an F/P/T strategy and identify areas of emergent interests; and
• Monitor, in accordance with its mandate only, donation and transplant outcomes, including both quantitative and qualitative measures against international and Canadian benchmarks; as well as on the outcomes of the F/P/T strategy established by provinces/territories.

Under this funding agreement, CBS's work concerning the OTDT program has been conducted in three pillars of activity:

1. Strategic Plan Development and Implementation: development and implementation of a national coordinated strategic plan on organ and tissue donation and transplantation;
2. Support Leading Practices / Public Awareness and Education: the development and evaluation of clinical practice guidelines and recommendations, organ sharing algorithms, and supporting knowledge transfer to health practitioners; as well as developing and disseminating education materials on OTDT for the Canadian public; and
3. System Performance Improvement: identify performance metrics and work with Accreditation Canada to develop accreditation standards that will be incorporated into the general accreditation standards for all Canadian hospitals with a donation program, as well as develop guidelines and recommendations for improved OTDT measures and definitions.

Concurrently, the provinces and territories (P/Ts) signed a Letter of Intent with CBS providing a matching amount (an additional $3.58 million per year) for the development and operation of three national patient registries. These activities are funded exclusively through the P/Ts (excluding Quebec).^{Footnote 3} Although Health Canada funding was used to support some policy and coordination work associated with the registries, the registries are not covered by this evaluation. The evaluation focuses on the three pillars funded by Health Canada.

2.1.2 Research and Development (R&D) Program

The CBS R&D Program seeks to drive continuous improvement and knowledge creation through research and development work in the areas of blood safety, alternatives to transfusions, and blood substitutes. Indeed, the R&D Program aims to advance Canadian innovation in transfusion^{Footnote 4} primarily through discovery research, training of highly qualified researchers, and development projects.

The overall objectives of the R&D program are to support basic, applied and clinical research on blood safety and effectiveness issues by:

• Conducting innovative research towards the elimination of threats of insufficient supply and transfusion-borne diseases;
• Providing in-house scientific support to business operations in an effort to maximize blood and blood product safety, quality and innovation, and to provide a means to address the national shortage of skilled personnel in transfusion science and medicine; and
• Supporting programs to recruit and train scientists and physicians either independently or in partnership with the Canadian Institutes of Health Research, to meet Canada's needs for an effective blood system.

The program encompasses several laboratories based in universities and research institutes across Canada that collaborate with a range of external partners. This expertise is concentrated in five research centers, or "hubs", each of which specializes in a particular area of transfusion science within CBS priorities: Vancouver - Blood Product Processing and Storage; Edmonton - Stem Cells, Nanotechnology and Cryobiology; Hamilton - Clinical Research; Toronto - Transfusion Immunology; Ottawa - Infectious Diseases.^{Footnote 5}

CBS's work concerning the R&D program has been conducted across five functions which are aligned with the program's objectives:

1. Research Function: provides leadership and support for scientific research related to blood transfusions. This includes peer review of applications and the provision of operational grants and personnel funding awards in relevant areas for scientists, as well as postdoctoral fellows and graduate students (note that some grants are delivered in partnership with the Canadian Institutes of Health Research (CIHR))^{Footnote 6};
2. Development Function: provides primary scientific support to the operational arms of CBS and helps with the development of research innovations. Much of this work is carried out by the Vancouver Network Centre for Applied Development (netCAD) facility^{Footnote 7};
3. Intellectual Property & Business Development Function (Operational Support); works on the protection of intellectual property arising from CBS-supported innovative research through patents;^{Footnote 8}
4. Quality Monitoring Program Function (Operational Support): examining the effects of clinical blood preservation and storage through regular product testing to provide CBS researchers (e.g., within netCAD) and operations with data to assist with product and process validation and troubleshooting;^{Footnote 9} and
5. Research and Education (Operational Support): a new function established in 2011-12 to enhance mobilization of CBS knowledge, products and tools, including collaborating with key public and private organizations involved in the Canadian health care system.^{Footnote 10}

2.1.3 Stakeholders and Beneficiaries

Both OTDT and the CBS R&D Program have a broad range of stakeholders, especially those external to CBS.^{Footnote 11} These stakeholders include:

• Health Canada, units responsible for managing the CBS Grant and Contribution Programs and those involved in blood and OTDT policy and regulatory activities, and Federal/Provincial/Territorial (F/P/T) relations;
• the provincial and territorial governments (particularly the Ministries of Health);
• federal agencies in the Health Portfolio (CIHR and the Public Health Agency of Canada);
• Canadian Institute for Health Information (CIHI), Accreditation Canada, and other national organizations with whom CBS collaborates;
• Héma-Québec;
• researchers from the fields of organ and/or tissue transplantation, transfusion, blood and blood products;
• other Canadian health sector organizations and the professionals working within these organizations (e.g., national/provincial patient organizations, organ transplant organizations, tissue banks, organ procurement organizations (OPOs)); and
• international organizations active in organ and/or tissue transplantation or transfusion science and safety.
• Finally, and more broadly, all Canadians, in particular blood, organ and tissue donors, transfusion and transplantation recipients and their support networks are the ultimate beneficiaries of both these programs.

2.1.4 Governance and Administration

Canadian Blood Services

The structure of CBS allows the organization to operate relatively independently ("at arm's length") from the F/P/T governments.^{Footnote 12Footnote 13} The Ministers of Health of nine provinces (excluding Quebec) and three territories participate as corporate members in the organization (similar to shareholders).^{Footnote 14} CBS is governed by a Board whose members are appointed for a four-year term by the provincial and territorial Ministers. However, Health Ministers do not have the power to direct operational decisions of the Board or CBS staff. This ensures that accountability rests with the organization and its Board. The R&D program is housed in the Medical, Scientific & Research Affairs division, led by a Vice-President.^{Footnote 15} To fulfill its mission and enhance its work, the CBS Board uses several Committees. In particular, CBS has an independent Scientific Research Advisory Committee composed of international experts associated with blood systems and blood operations. This Committee reports directly to the CEO on emerging matters and provides advice on the activities of CBS.

Health Canada

Health Canada's Strategic Policy Branch (SPB) plays an oversight role in accordance with the OTDT Contribution Agreement and also manages the R&D Grant Agreement. Two program management staff in SPB - one responsible for each program - ensure the terms and conditions of the funding agreements are met (e.g., review the plans and reports submitted by CBS for each program for this purpose). Program staff members also take an active role in knowledge brokering through the relationship they establish with CBS and with relevant units within Health Canada and within the Health Portfolio.

2.2 Program Logic Model

The integrated logic model for the evaluation of the CBS Grant and Contribution Programs and detailed description are provided in Annex A. Importantly, this integrated logic model was specifically designed for this evaluation and has not been used by CBS or Health Canada in the planning, management, delivery or oversight of either program. The integrated logic model was developed in 2012 based on the OTDT logic model^{Footnote 16}, a review of available documentation and consultations with Health Canada representatives responsible for oversight of both programs. Although the logic model contains both programs, their activities, outputs and most outcomes are presented in parallel rather than combined because the consultations clearly indicated that, despite both being housed in CBS, the OTDT and R&D programs are operated completely separately. It also includes the activities conducted by Health Canada's SPB with regard to the CBS Grant and Contribution Programs.

The connection between the influencing factors, inputs, activity areas, outputs and the expected outcomes is depicted in the logic model. The evaluation assessed the degree to which the defined outputs, immediate outcomes and intermediate outcomes were being achieved over the evaluation timeframe. The ultimate outcomes for the CBS Grant and Contribution Programs are:

• OTDT: Improved Health Outcomes for Patients
• R&D: Safe, Effective and Efficient Blood System Responsive to Canada's Current & Future Needs

2.3 Program Alignment and Resources

2.3.1 Program Alignment

The CBS Grant and Contribution Programs are managed within the Strategic Policy Branch (SPB) and fall within Health Canada's Program Alignment Architecture (PAA), specifically under Program Activity 2.1, "Health Products". Note that, before 2008, the CBS Grant and Contribution Programs were managed by Health Canada's Health Products and Food Branch. Currently, all grant and contribution programs are managed by SPB regardless of where they are situated in the PAA. Regulatory activities associated with this Program Activity are undertaken by the Health Products and Food Branch (HPFB).

2.3.2 OTDT Program Resources

The OTDT Program is supported by a $17.9 million five-year Contribution Agreement with Health Canada ($3.58 million per year).^{Footnote 17Footnote 18} The planned and actual expenditures of the OTDT Program by cost type are presented in Table 1 for each fiscal year (FY) and as a total across the evaluation period. These expenditures are those funded by the Health Canada contribution, except for the "P/T Allocation" line. This line represents provincial and territorial funding that was used to support OTDT activities included under the Health Canada Contribution Agreement (i.e., national system design, leading practice development, and system performance) in FY 2009-10 and FY 2010-11.^{Footnote 19}

Source: Compiled by Science-Metrix from CBS OTDT Program documents.

2.3.3 R&D Program Resources

Since 2000-01, the R&D Program has been funded by a grant of $5 million per year provided to CBS by Health Canada, for a total of $25 million over the evaluation period (2008-09 to 2012-13). In addition to expenditures shown in Table 2, some operating grants under the CBS-CIHR partnership are funded by CIHR. In other words, through this partnership, additional funding is leveraged from CIHR to support R&D work on research areas relevant to the CBS R&D Program.

Note: No planned expenditures available for 2008-09 to 2010-11. For Federally Funded CBS-CIHR Grants (Schedule 6), this represents grants from the CBS-CIHR partnership that are funded by CBS using Health Canada funds for grant recipients outside CBS, whereas Schedule 4 covers CBS-CIHR grants to recipients who are salaried researchers at CBS; additional funds (not shown here) are provided by CIHR to other grant recipients from the same competitions.

Source: Compiled by Science-Metrix from CBS R&D Program Annual Financial Summaries 2008-09 to 2012-13.

Besides the Health Canada grant, P/T Corporate Members provided an additional $5 million per year to CBS's R&D activities. The P/T R&D funding is determined each year as part of the annual budget negotiations. This funding thus also contributes to R&D Program achievements, such that no distinction is made in this evaluation between specific activities funded by Health Canada and those funded by the P/Ts. For instance, some items in Table 2 (e.g., Schedule 2) have been partially or totally supported by P/Ts in various years instead of by Health Canada.

3. Evaluation Description

3.1 Evaluation Scope

This is the first federal evaluation of Health Canada's CBS Grant and Contribution Programs and is a single evaluation of two distinct programs within CBS, each of which receives grant or contribution funding from Health Canada. The OTDT Program and R&D Program are distinct in their context/history (including within CBS), stakeholders, management, delivery and outcomes.

The scope of the evaluation covered both the OTDT and R&D Programs from 2008-09 to 2012-13. The evaluation covered activities carried out by CBS under the federal components of the OTDT and R&D Programs. This excluded the national registries developed by the CBS OTDT Program with P/T funding.

Evaluation Issues

The specific evaluation questions used in this evaluation were based on the five core issues prescribed in the Treasury Board Policy on Evaluation (2009). These are noted in Table 3 below. These five evaluation issues were addressed through a series of specific evaluation questions, which are answered in Section 0. A data collection matrix was used to align evaluation questions with the associated indicators, data sources and data collection methods.^{Footnote 20}

3.3 Evaluation Approach

A goal-based evaluation approach was used for the conduct of the evaluation to assess the progress made towards the achievement of the expected outcomes, whether there were any unintended consequences, and what lessons were learned. The evaluation approach for this project was developed in consultation with the Public Health Agency of Canada/Health Canada Evaluation Directorate and an external expert advisor. It was designed to draw on various sources of data to ensure that the combined lines of evidence resulted in an in-depth and comprehensive analysis.

3.4 Evaluation Design

The Treasury Board Policy on Evaluation guided the identification of the evaluation design and data collection methods so that the evaluation would meet the objectives and requirements of the Policy. The specific non-experimental design used was based on the data collection matrix developed for this evaluation which provided consistency in the collection of data to support the evaluation. The data collection matrix was used to cross-link evaluation questions with associated performance indicators, data collection methods and data sources, allowing data to be triangulated and compared for each question during the analysis. This practice helped ensure the findings and conclusions are valid and reliable.

3.5 Methods

The evaluation collected data between September 2012 and August 2013. The key findings, conclusions and recommendations presented in this report were drawn from multiple lines of evidence collected and analyzed using the following four methods. Table 4 below presents additional information on each data collection method.

1. Document review (internal documents): a review of documents and files pertinent to the OTDT and the R&D programs.
2. Literature review: a review of several types of documents (peer-reviewed articles, grey literature, media reports and websites) retrieved using bibliographic database and web searches, or extracted from the websites of targeted organizations.
3. Key informant interviews (internal and external stakeholders): telephone and in-person interviews with a total of 26 key informants (via 22 interviews).
4. Surveys: two web surveys were administered to CBS OTDT program stakeholders and CBS R&D program funding recipients.

3.6 Limitations

The following table (Table 5) outlines the limitations, their impact/potential impact on the evaluation and the mitigation strategies employed in this evaluation to limit these impacts. For the most part, mitigation strategies had been proactively built into the evaluation design, most notably by ensuring that multiple lines of evidence were collected for each evaluation question to allow triangulation and validation of findings from several sources.

4. Evaluation Findings

4.1 Relevance

4.1.1 Continued Need for the Program

Both components of the CBS Grant and Contribution Programs continue to address a demonstrable need and remain responsive to the needs of Canadians.

Question: Do the CBS Grant and Contribution Programs continue to address a demonstrable need?

CBS OTDT Program

Multiple lines of evidence, particularly the document and literature reviews and the interviews, indicated that there is a continued need to strengthen the Canadian OTDT system as several core issues that triggered the establishment of the Canadian Council for Donation and Transplantation (CCDT) in 2001 remain largely unchanged. For example, there have been few improvements with regard to organ and tissue waiting lists, donation and transplantation statistics since 2008.^{Footnote 22} This shows that the supply clearly does not meet the demand (about 2000 transplants per year versus 4000 patients on waitlists). In addition, the number of patients reported to have died while on a waitlist increased from 215 to 285 over this time. Probability analyses showed that Canadians are more likely to need an organ in their life than to become a deceased donor.^{Footnote 23}

Another issue behind the original mandate of the OTDT program under the CCDT is the fragmented national system. Examples of this fragmentation include inconsistent practices across provinces and territories (e.g., to express one's intent to donate) and various barriers to effective inter-provincial organ sharing. The need to develop and implement a more coordinated and integrated national system for OTDT was noted by the Barrington Research Group in the 2006 summative evaluation of the CCDT.^{Footnote 24} To address this need, the CBS OTDT Program has developed a strategy to achieve a more coordinated and integrated national system in the 2012 Call to Action.

The Call to Action report is a national strategy on organ and tissue donation and transplantation that was submitted to the provincial and territorial governments in which CBS outlined the key problems and issues they identified within the Canadian OTDT system at that time. CBS also established three committees (Steering Committee, Organ Expert Committee and the Tissue Expert Committee), who prepared the resulting strategic plan. This plan, called the Call to Action included 25 recommendations (12 for organs and 13 for tissues) to address the identified barriers and significantly enhance the donation rates and health outcomes of patients in Canada.

The development of the national strategy benefitted from a collaborative approach through which CBS collected and integrated the perspectives of several groups of OTDT stakeholders. When the report was first released, the report came with an aura of a "big bang" impact for OTDT. In reality though, adoption of the recommendations will likely be more incremental and will vary across jurisdictions, mainly due to the fact that some jurisdictions have already implemented some form of the recommendations outlined in the Call to Action; whereas, other have not. Further collaboration will be necessary to implement this strategy.

The continued need for the program is outlined in the Call to Action which notes the Canadian OTDT system has been significantly underperforming compared with some other developed countries (the most commonly cited high performers are Spain and the United States), especially with deceased donation rates.^{Footnote 25} As for tissues, Canada was largely dependent on imports from the United States, although individual jurisdictions exhibit a diverse range in wait times for ocular tissue transplantation. Additionally, there has been a lack of public awareness in Canada about organ donation and a need to increase transparency, fairness and information sharing practices (e.g., donor information) associated with organ donation and transplantation. Finally, there has been a need to better identify potential donors in Canada and to address the growing global problem of "transplant tourism", which is the practice of travelling to another country for organ transplantation.

CBS R&D Program

The CBS R&D Program was established as a response to the Krever Commission. Krever investigated the causes behind thousands of cases of transfusion-transmitted HIV and hepatitis C in the 1980s and 1990s. The resulting Krever Report (1997) made several recommendations to enhance the safety of the Canadian blood system, some of which directly or indirectly concerned the need for research and evidence-based information. Notably, one recommendation (#24) was that the blood operator (i.e., CBS or Héma-Québec^{Footnote 26} under the current Canadian system) should allocate 10% of its operating budget to a research program.^{Footnote 27}

Nevertheless, the evidence reviewed for this question (i.e., documents, literature and interviews) clearly indicated that, because of the biological nature of blood, there will be continued risks associated with blood transfusion from various known, new and emerging pathogens. Improving practices to avoid contamination and spoilage are important from both safety and cost-effectiveness perspectives. Therefore, there is a continuous need to keep abreast of the latest developments not only to sustain blood safety (e.g., testing and developing new technologies to address these threats), but also to identify changes in practice that help ensure an adequate supply of quality blood and blood products and to improve the performance of blood operations (including effectiveness, cost-effectiveness and affordability) in a way that minimizes risk.^{Footnote 28} From a regulatory perspective, science-based evidence is also needed to support proposed operational changes that are prepared and submitted by blood operators (i.e., CBS and Héma-Québec) to Health Canada' Biologics and Genetic Therapies Directorate for approval, in accordance with the Food and Drugs Act and its regulations.

Notably, the large demand for transfusable products such as red blood cells and platelets continues to grow over time and underscores the need for R&D to ensure an ongoing supply of safe and high-quality blood products.^{Footnote 29}

Question: Are the CBS Grant and Contribution Programs responsive to the needs of Canadians?

CBS OTDT Program

Overall, the evidence indicates that the CBS Contribution Program has been responsive to some needs of Canadians by seeking to improve the Canadian OTDT system in order to improve health outcomes for patients in Canada and to improve access, fairness and transparency for all patients on organ waitlists. About 90% of OTDT stakeholders (individuals who are currently or have recently been engaged with OTDT projects) who responded to the survey agreed that the OTDT program objectives and activities funded by Health Canada were important or very important. At least 70% of survey respondents also indicated that each of the main CBS OTDT activities were appropriate or very appropriate (Table 6) to improve the performance of the OTDT system given the issues summarized above. These survey results were consistent with the views expressed across the interviewee groups consulted (key informants within Health Canada, CBS and P/T governments).

Note: This question was asked of all OTDT respondents (N=39 respondents).

Besides the interviewees and OTDT stakeholders consulted for this evaluation, recent public opinion surveys^{Footnote 30} and documents on public fora held across Canada by the OTDT Program,^{Footnote 31} literature on international best practices, as well as media reports and expert commentary, collectively confirmed that the program is targeting the appropriate objectives and priorities. In other words, the OTDT Program objectives and activities are well aligned with the identified needs. In particular, there is broad stakeholder and documented support for the development and implementation of leading practices, for greater education about OTDT for the public and for health care professionals, as well as for a pan-Canadian approach to help coordinate and standardize OTDT practices across jurisdictions (including improved performance measurement) to address ongoing national system-level issues affecting Canadian OTDT performance. All lines of evidence also indicated that CBS's extensive consultation process during the development of the Call to Action helped ensure that the plan meets the needs of the broad OTDT communities.

Notably, some interviewees saw similarities between the national system design proposed in the Call to Action and the approach and goals expressed by Premiers in setting up a Health Care Innovation Working Group (2012). All P/Ts are participating in this initiative which aims to "drive a collaborative process for transformation and innovation to help ensure the sustainable delivery of health care services", including through greater inter-jurisdictional collaboration and cooperation.^{Footnote 32}

Interviewees were therefore highly supportive of ongoing work of the CBS OTDT Program to achieve further improvements, with leading practices and education/awareness activities receiving the most vocal support across stakeholder groups (i.e., internal and external to CBS). The stakeholder views (through interviews and survey) also indicated the OTDT Program at CBS is well positioned to contribute to these improvements and to help support inter-provincial coordination.

CBS R&D Program

All lines of evidence, including documents, interviews and the web survey, confirmed that the CBS R&D Program addresses the current and future needs of Canadians by helping ensure the safety, supply, quality, cost-effectiveness and affordability of blood, blood products and their alternatives. Indeed, the R&D Program is well aligned with these needs: it is designed to advance innovation and maintain Canadian capacity in transfusion science and medicine in order to address risks and threats to the safety of blood transfusion in a timely manner. According to CBS, it also develops improved procedures, products, and technologies relating to the safety, quality and efficiency of the blood system.^{Footnote 33}

The R&D Program was found to address a broad range of needs, including operational, regulatory, knowledge, and capacity-related needs. More specifically, the R&D Program supports the CBS operations and mandate^{Footnote 34} and provides evidence to support decision-making at the regulatory level (e.g., by Health Canada). It also contributes to the advancement of knowledge in this field in Canada (and internationally) and helps build and maintain capacity through funding and training of students and researchers in transfusion science and medicine. As such, it addresses some of the underlying issues identified by the Krever Commission, namely, the lack of knowledge and capacity in transfusion medicine in Canada (within and outside the blood operator) that contributed to the "tainted blood" event, and the need to maintain the necessary competence to stay abreast of changes in medical techniques and technology.^{Footnote 35}

In this context, the vast majority of interviewees agreed that the R&D Program remains relevant and appropriate. More than 80% of survey respondents (CBS-funded researchers) agreed or strongly agreed that the R&D Program is addressing the current and anticipated needs of the Canadian blood system (Table 7). As shown in this table, these stakeholders also indicated that the Program is generally relevant to the intended users and, to a slightly lesser extent, is successful in reaching them.

Health Canada's continued involvement/funding was seen as key by interviewees across stakeholder groups (Health Canada, CBS and most external stakeholders). In their view, if Health Canada were to withdraw its funding, CBS would suffer a severe reduction in its ability to support its research activities given that alternative sources of funding specifically dedicated to blood R&D are limited in Canada.

Note: This question was asked of all R&D respondents (N=68 respondents).

However, similar to the findings of the 2010 KPMG performance review,^{Footnote 36} some considered that the Program could better serve operational and regulatory needs by focusing more on developmental work (e.g., applied/operational research). For example, it was suggested that CBS devote a larger portion of their R&D resources/efforts to projects whose results are directly transferred to CBS operations (in response to particular needs/questions), to support regulatory submissions, or to other uses. It is important to note here that in the last two years CBS has been shifting some of its activities and resources toward supporting more applied/operational research (e.g., netCAD projects). CBS has also launched the Research & Education function, which has a clear focus on increasing knowledge mobilization to enhance the use and commercialization of research results. This transition was generally well received by stakeholders across varied groups interviewed for this evaluation, but was met with mixed views among supported researchers and trainees in the survey.

Moreover, CBS has seen the launch of a first spin-off company (LightIntegra Technologies), which is commercializing the ThromboLUXTM technology invented by a CBS scientist.^{Footnote 37} Commercializing and/or licensing products are in line with the practices of other international organizations conducting blood research. Indeed, several organizations have internal intellectual property programs, although few reported that this was a major focus or source of revenue for their organizations.^{Footnote 38}

The need for the Program to continue to support the knowledge generation and capacity building aspects was also highlighted by interviewees who indicated there were a limited number of other opportunities and organizations dedicated to blood safety and supply R&D in Canada. The Program annually reviews its research priorities through its Scientific and Research Advisory Committee - an independent group of 14 experts from the national and international transfusion science and medicine community - to help ensure that the R&D work conducted by CBS remains relevant to current and emerging issues.

4.1.2 Alignment with Government Priorities

Summary

Both programs are aligned with the Government of Canada (GoC)'s priority and related work to protect the health and safety of Canadians, as well as Health Canada's strategic outcome to support access to safe and effective health products and protect Canadians from health risks.

Question: How do CBS Grant and Contribution Programs' objectives relate to GoC priorities?

Current statements of the GoC priorities (e.g., Budget, Speech from the Throne) do not specifically mention the OTDT system or blood safety, but these are covered in the broader GoC priority to "protect the health and safety of Canadians" proclaimed in the 2012 Budget^{Footnote 39} and 2011 Speech from the Throne.^{Footnote 40} Indeed, this priority is consistent with CBS Grant and Contribution Programs' objectives to ensure and/or improve the safety and supply of blood and blood products (R&D Program) and to improve access to organs and tissues and to ultimately improve health outcomes (OTDT Program). Moreover, both programs were maintained at current funding levels in Budget 2013, while several other programs were reduced as part of the GoC's efforts to reduce expenditures; this suggests that each program aligns with ongoing federal priorities. Other evidence that these areas and objectives are aligned with GoC priorities is presented below.

CBS OTDT Program

While there has not been specific mention of OTDT for HC, the Minister of Health announced related funding for a national transplant research program (via the CIHR), anticipated to start in the spring of 2013. It is also worth highlighting that the 2011 Speech from the Throne included the following statement on health care priorities: "Our Government is committed to respecting provincial jurisdiction and working with the provinces and territories to ensure that the health care system is sustainable and that there is accountability for results."^{Footnote 41} The OTDT Program's ultimate goals of improving the health outcomes of Canadians through improved F/P/T cooperation on improving the performance of the OTDT system nationally, as well as improving access, fairness and transparency for all patients on organ wait-lists, appears to be well aligned with GoC priorities.

CBS R&D Program

In terms of other current GoC efforts associated with blood safety and/or supply, ongoing work from Health Canada shows that the Government continues to support efforts to ensure Canadians have access to safe blood for themselves and their families. For example, Health Canada developed the proposed Blood Regulations in 2012, which modernize existing regulations.^{Footnote 42}

Question: Do the CBS Grant and Contribution Programs align with Departmental strategic priorities?

Both components of the CBS Grant and Contribution Programs are aligned with Health Canada's Strategic Outcome 2: "Canadians are informed of and protected from health risks associated with food, products, substances and environments, and are informed of the benefits of healthy eating".^{Footnote 43} Specific evidence associated with each program is provided below.

CBS OTDT Program

The OTDT program supports access to safe and effective health products and protection from associated health risks by working to improve access to organs and tissues (through increased donations and transplantations), to promote public awareness and education, and to help coordinate/support improvements to the national OTDT system. With regard to the latter, it also contributes to supporting Health Canada's Strategic Outcome 1: "A Health System Responsive to the Needs of Canadians".

CBS R&D Program

Much of Health Canada's efforts have focused on health protection through the regulation of blood and blood products. For instance, in 2012, Health Canada proposed new Blood Regulations, delivering on the last recommendation of Justice Krever's comprehensive report.^{Footnote 44} Funding blood research is another way that Health Canada supports protection of Canadian from health risks associated with health products (e.g. blood and blood products) through evidence-based information and practices to ensure the safety and supply of the Canadian blood system.

4.1.3 Alignment with Federal Roles and Responsibilities

Summary

The CBS Grant and Contribution Programs' activities fall within the federal role, including Health Canada's mandate, regulatory role and jurisdictional responsibilities for the regulation of biological health products. They also align with the federal role in ensuring that all Canadians are provided fair and equitable access to health care, in collaboration with the provinces and territories, in accordance with the Canada Health Act^{Footnote 45}.

Question: Are the CBS Grant and Contribution Programs' activities aligned/congruent with federal and Health Canada's roles?

Under the authority of the Food and Drugs Act and its Regulations, and the Department of Health Act, the HPFB^{Footnote 46} evaluates and monitors the safety, quality and efficacy of biologic and genetic therapies. Blood and blood products (i.e., relevant to the R&D Program) and tissues, organs and xenographs (i.e., relevant to the OTDT Program) are defined as "biological products" (products derived from living sources) for human use. Given that the objectives and activities of the CBS Grant and Contribution Programs relate to these regulated biological/therapeutic products, there is a clear federal and Health Canada role in the subject areas addressed by these programs. Specifics for each program are noted below.

CBS OTDT Program

The CBS OTDT Program operates in a complex environment. There are a great variety of stakeholders with distinct jurisdictional, mandated and/or legislated roles. As a result, the question of roles and responsibilities is highly relevant. The documents reviewed, including the Call to Action report, clearly outlines Health Canada's role in the OTDT system as the regulator, in accordance with its legislated role through the Department of Health Act, Food and Drugs Act and its Regulations (particularly the Safety of Human Cells, Tissues and Organs for Transplantation Regulations^{Footnote 47}). In particular, the departmental specific role is to "establish and maintain regulations for organ and tissue donation and transplantation, and monitor and enforce compliance."^{Footnote 48}

In contrast, the provinces have jurisdictional authority over health care delivery, such that the roles and responsibilities of the P/T governments specific to the OTDT system as stated in the Call to Action include: to appropriately resource OTDT activities (i.e., funder); to determine the roles, responsibilities and reporting structures of the public organizations that provide OTDT services; and to formalize the OTDT accountability framework.^{Footnote 49} OTDT services are provided by public organizations such as hospitals, organ procurement organizations (OPOs), and tissue recovery and processing organizations.

The federal and provincial governments work in partnership to support the health care system. Health Canada's website notes: "Working in partnership with provinces and territories, Health Canada also supports the health care system through initiatives in areas such as health human resources planning, adoption of new technologies and primary health care delivery."^{Footnote 50} More broadly, the federal government has contributed to shaping health care services across the country using its spending power (e.g., transfer payments, shared cost) in the interest of providing fair and equitable access to health care for all Canadians, in accordance with the five principles of the Canada Health Act^{Footnote 51}.

Given the above, the reviewed documents indicate that the Health Canada contribution funding supports OTDT activities aligned with the federal role related to: (1) national leadership in coordination of the OTDT system; (2) knowledge brokering; and (3) providing leadership for clinical practices, standards and guidelines.

Interview and survey evidence also confirmed that Health Canada's role in funding the OTDT Program and overseeing the activities included in the Contribution Agreement (to monitor progress and for accountability purposes) are appropriate and congruent with federal roles and responsibilities. In particular, interviewed representatives from P/Ts saw a continued need for Health Canada funding for ongoing national-level work by CBS on OTDT, especially for issues that would be difficult for P/Ts to advance alone (e.g., coordination of national standards or awareness campaigns); representatives from CBS agreed that P/Ts were unlikely to fund the initiatives currently funded by Health Canada under the Contribution Agreement. Note that interviewees from Health Canada and CBS agreed that a Contribution Agreement was the appropriate funding mechanism for the CBS OTDT Program.

Of interest, in terms of the roles taken by other federal governments, the most effective systems were generally seen in countries that have a national structure and official body overseeing OTDT activities at the national (e.g., Spain) and/or federal level (e.g., Australia). Indeed, this is becoming the international standard, particularly in developed countries (which also have the highest "deceased donor" rates).

Interviewees and survey respondents often expressed conflicting expectations as to Health Canada's roles, other than funding, on this file. On one hand, some interviewees from CBS consider that Health Canada can and should provide "national leadership" and provide a more active role including in the implementation phase for the national OTDT strategy, as well as being involved in monitoring of progress and financial accountability of the OTDT Program. On the other hand, some (but not all) Health Canada and P/T government representatives suggested that the Department's regulatory and jurisdictional role was not consistent with a more active role in implementation of the national strategy.

Most interviewees saw the role of CBS as appropriate although, moving forward, Health Canada and external stakeholders perceived a need for greater clarification of the differences between CBS and P/T roles and responsibilities, particularly for governance (i.e., accountability framework, as proposed in the Call to Action^{Footnote 52}), coordination and around areas of potential overlap during implementation of the national strategy.

The future role and mandate of the CBS OTDT Program will depend largely on the decision of the P/Ts on the recommendations in the Call to Action. The OTDT Program is currently in a transition phase as it awaits this decision (the strategy and recommendations developed have been under review by the P/Ts for more than 18 months). Moreover, some provinces have moved forward with investments and changes to their OTDT systems,^{Footnote 53} such that these provincial investments could be duplicated in some regions if some national-level recommendations in the Call to Action are implemented as formulated. For example, CBS implementing new inter-provincial registry systems could potentially overlap with new registries recently implemented by some provinces. Ongoing and enhanced interaction between CBS and P/T governments would be necessary to support next steps (i.e., decision-making, potential implementation) for the Call to Action.

Assuming that at least some recommendations in the Call to Action will be endorsed by P/Ts, interviewees also saw a continued need to fund the CBS OTDT Program to support more fulsome implementation of the national OTDT system. There was a general sense that federal investments to date would be lost if the OTDT Program did not continue and if some form of implementation did not occur. In addition, several members of the OTDT community consulted through interviews and surveys worded concerns that the consultation process leading up to the development of the Call to Action would be meaningless if there was no follow-up on decisions and policies. In sum, a particular level of momentum and expectation has been created around the future national OTDT system, but, again, most changes depend on pending P/T decisions regarding implementation of the Call to Action.

CBS R&D Program

Health Canada fulfils the federal role of blood regulator in accordance with the Food and Drugs Act and its Regulations. Meanwhile, CBS, as a blood operator, conducts R&D activities that help Health Canada meet its objectives of delivering a safe and efficient blood system for Canada. The Grant Agreement^{Footnote 54} signed between Health Canada and CBS and other lines of evidence (e.g., interviews, survey) confirm that Health Canada's financial support to the CBS R&D Grant is consistent with the Departmental mandate and complements its regulatory role for the blood system.

Also supporting the relevance of providing government funding (F/P/T) to a blood operator for its R&D activities, similar funding models for blood R&D are found in international jurisdictions (e.g., Australia), although some alternative models are also observed (these are discussed in Section 4.2).^{Footnote 55}

Overall, views on the appropriateness of Health Canada's involvement in the CBS R&D Program were found to vary according to the group and associated level of awareness of stakeholders. For example, individuals receiving grants and awards through the CBS R&D Program (i.e., primarily researchers) were often not aware of Health Canada's involvement and many considered that the Department should only be providing funding. In contrast, Health Canada's representatives expressed an interest in being more involved with CBS to increase their awareness of planned work within CBS' R&D research areas in order to help support the Department's regulatory role and priorities^{Footnote 56}. The opportunity for greater coordination and knowledge sharing with Health Canada was also well received within CBS, who saw benefits in terms of a better understanding of Health Canada's needs and expectations, as well as greater knowledge mobilization across the Health Portfolio (i.e., involving PHAC and CIHR). As a result, there is potentially a larger role for HC to play in CBS' R& D agenda beyond that of funder.

More broadly, various lines of evidence (e.g., documents, interviewees) suggest there would be mutually beneficial opportunities for Health Canada, CBS and Héma-Québec to work more closely with one another with regard to blood R&D to discuss complementarities or synergies and to avoid overlap in their R&D activities/projects.

Finally, the role of CBS was generally perceived as appropriate, including its presence in the international blood community. However, some interviewees within Health Canada and organizations external to CBS questioned the current balance between basic and developmental (including operations- and/or regulatory-focused) research, and supported recent efforts by CBS - such as through netCAD - to increase the focus of the program's R&D activities in priority areas that clearly involve operational considerations (e.g., to improve the efficiency or effectiveness of CBS blood operations or to address specific operational problems or issues).^{Footnote 57}

4.2 Performance

4.2.1 Achievement of Expected Outcomes

Summary

The CBS OTDT Program has developed standards, guidelines, leading practices, training programs and the Call to Action report in accordance with its work plans and has achieved progress towards the achievement of its outcomes related to improved performance metrics, enhanced standards and public awareness of OTDT.

The CBS R&D Program has generated numerous knowledge products, i.e. peer reviewed papers and conference presentations, in the important areas of basic and applied research that support the mission of CBS. The Program has also shown achievement across most of its outcomes, especially in the areas of building and maintaining research capacity and developing/revising technologies, tools and procedures for blood manufacturing, transportation, storage and use.

Health Canada's management of the CBS OTDT has been effective over the evaluation period, while that of the CBS R&D Program has improved significantly in recent years. The outputs of these activities have contributed to enhanced accountability, more focused program delivery, and enhanced knowledge sharing and strengthened relationships.

Question: Is the CBS OTDT Program producing the expected outputs and outcomes?

Immediate Outcomes and contributing Outputs

All lines of evidence support the Program's success in raising awareness and understanding of the OTDT Program and priorities among diverse groups of OTDT stakeholders across Canada. These stakeholders include relevant experts and those involved in health care delivery, as well as interested members of the public. The extensive consultation and engagement activities carried out by the Program in its first three years of the evaluation period were particularly credited.

These activities produced outputs relating to:

• knowledge tools and productssuch as leading practices, guidelines/standards, toolkits, reports, reviews of international practices;
• information-sharing and dissemination outputs such as the delivery of dozens of presentations, engagement in national and international conferences, fora and symposiums, and the organization and conduct of a range of workshops, meetings and expert groups;
• a website dedicated to organ and tissue donation and transplantation in Canada;
• participation in recent public education efforts (e.g., 2012 Facebook campaign for National Organ and Tissue Donation Awareness Week) to raise awareness;
• work that informed the development of the Call to Action and the national registries (which also contributed to understanding OTDT priorities).

OTDT stakeholders external to Health Canada and CBS consulted for this evaluation through the web survey and interviews were asked to comment on specific Health Canada-funded activities within the CBS OTDT Program. These stakeholders were most aware of the priorities and work conducted by the OTDT Program relating to the barriers, opportunities and recommendations for the national system, followed by awareness of the development of leading practices by CBS. The tabling and subsequent release of Call to Action has had a considerable effect on increasing awareness among P/T governments. Individual P/T governments have since conducted jurisdictional reviews of their capacities, needs and resources as part of their efforts to determine their response to the strategy.

Similarly, representatives from Health Canada reported a greater level of awareness within the Department and within the federal government more broadly, given the increased attention paid to the OTDT issues after the tabling of the national plan. Note that other factors were identified - such as provincial OTDT communication efforts and the Hélène Campbell's story^{Footnote 58} - which have helped increase awareness of OTDT issues among government representatives, among the OTDT community, and among the general public.

There are indications that general public awareness of the OTDT issues has increased because of the Program, particularly through the public consultations. Public opinion surveys suggest that most Canadians consider that improving access to organs and tissues (83% and 80%, respectively) is a priority issue.^{Footnote 59} However, the same surveys indicate that Canadians continue to have limited awareness of which organizations are responsible for this area of health care, both on national and P/T levels, or how to indicate their intention to donate organs and tissues.

The Program's activities and outputs have also resulted in increased coordination and integration of OTDT activities across stakeholder groups in Canada. Indeed, more than 50% of survey respondents, who represent a wide range of OTDT stakeholders across Canada, indicated that CBS OTDT activities had contributed to this outcome to a large or great extent (Table 8). An additional 32% perceived this contribution to be to a moderate extent. Overall, CBS was seen by interviewees and survey respondents to have brought key OTDT stakeholder groups together, such as through consultations, collaborative efforts and workshops on various OTDT topics (various information-sharing and dissemination outputs). This enhanced communication and working relationship within the national OTDT community has directly contributed to greater collaboration and standardization of practices within and between jurisdictions (e.g. improved consistency between provinces, coordinating the development of organ allocation algorithms). Better coordinated data collection was also noted as a result of CBS OTDT efforts, particularly in the area of tissue donation.

Note: This question was asked of all OTDT respondents (N=38 respondents). Items a) to c) represent immediate outcomes and items d) to f) represent intermediate outcomes.

Health Canada's ongoing contribution funding to the CCDT and CBS OTDT Program supported several types of nationally coordinated activities and outputs (e.g., development and continued promotion of leading practices, policy work, and standardization of laboratory practices) that indirectly supported the development and implementation of the three national patient registries.^{Footnote 60} Supported by P/T funding, the national patient registries were often cited by stakeholders as a key element of the CBS OTDT Program. This synergistic relationship between Health Canada and P/T-supported activities was highlighted by several survey respondents and interviewees.

The Program also invested considerable efforts and resources in identifying the main barriers and associated strategies to improve the performance^{Footnote 61} of the Canadian OTDT system. Outputs associated with these activities included consultations with health care professionals, the public and international experts, as well as reviews of international practices to identify strategies that could be applied in the Canadian context. This led to the production of a document called the Case for Change (2010).

Furthermore, in response to opportunities for further improvement and standardization of OTDT practices, CBS has been leading continuing work to develop guidelines/leading practices (e.g., determination of death) and professional training programs in a manner to promote collaboration, cooperation and implementation across Canada.

The immediate outcome associated with the improved system performance metrics has been achieved, as existing metrics have been refined or new ones defined following reviews and consultative processes. The Program delivered on the development of new system performance metrics^{Footnote 62} and accreditation standards for hospitals with donation and transplantation programs. To date, through the work of the Organ Expert Committee, four detailed clinical metrics have been developed.^{Footnote 63} In addition, other system-level performance measures and targets specific to organ donation/transplantation and to tissue donation/transplantation were proposed in the Call to Action.^{Footnote 64} Furthermore, CBS continued its collaboration with Accreditation Canada to support and provide advice in their development of new accreditation standards (Organ and Tissue Donation Standards for Deceased Donors, Organ Donation Standards for Living Donors, Organ and Tissue Transplant Standards) and in the enhancement of existing standards (Emergency Department and Critical Care Services Standards).

OTDT-related performance measures are also included in the improved accreditation framework for OTDT. As part of this framework, new standards that have been developed were piloted (with six performance indicators) in at least a dozen hospitals across Canada. Meanwhile, the enhanced standards are being used in 114 emergency departments and 72 critical care departments across the country.

Intermediate Outcomes

Progress was observed across several lines of evidence toward the achievement of the intermediate outcome related to the adoption of new/modified knowledge, policies and health care practices(e.g., clinical, laboratory recommendations). This outcome often built on past work initiated under the previous contribution agreement, for example, on issues associated with the neurological determination of death.^{Footnote 65} Progress towards achievement of this outcome is supported by the survey conducted for this evaluation (Table 8), as well as a survey commissioned by CBS on leading practice recommendations. Although CBS's survey had a small sample (26 respondents) of OTDT practitioners across Canada, the results showed that the leading practices developed by CBS were seen as relevant, useful, and that they have generally made positive contributions to the performance of the OTDT system through increased donation rates and a higher number of successful transplants.^{Footnote 66}

Evaluation evidence pointed to several other examples associated with this outcome, including:

• the single policy document for Living Donor Paired Exchange (LDPE) listing and allocation in Canada. This policy has been used to develop the registry algorithm for matching and allocating pairs and chains and is updated annually, based on modeling and outcome data. To date LDPE has matched nearly 200 patients as a result of this algorithm.
• the standardization of donor reimbursement policies in various provinces;
• the development of practices for donation after cardio-circulatory death; and
• the implementation of donation physicians specialists in some provinces (e.g. Manitoba) based on best practices identified and communicated by CBS.

The intermediate outcome associated with the adoption and use of performance metricsappears to have been achieved to a modest extent (Table 8). Broader adoption of these metrics is largely dependent on the P/T direction on the Call to Action recommendations and ongoing coordination work with key partners, such as CIHI and the Canadian Organ Replacement Register, Organ Procurement Organization (OPO) and transplant centres. In terms of increased use of an accreditation framework for OTDT, the results of the pilot study on new standards had not been released at the time of this evaluation while anecdotal evidence (reported by interviewees) indicates that the enhanced standards have been well accepted or received.

This evaluation identified that the OTDT Program has fulfilled to a large extent its annual and revised work plans in accordance with the Contribution Agreement with Health Canada. However, the achievement of the Program's intended intermediate and long-term outcomes will probably take a longer time-frame than originally anticipated. Several challenges exist facing ongoing work towards the implementation of a national OTDT system in Canada, e.g. the complexity of the Canadian OTDT system, the variability in the OTDT capacity and needs of individual jurisdictions, as well as the need to rely on the willingness of P/Ts to realize Program outcomes.

In this context, P/Ts have taken at least 18 months to review the Call to Action after the report was tabled by CBS with P/Ts, which is longer than expected by CBS. This situation has contributed to the current modest level of progress for many intermediate outcomes of the OTDT Program. For example, certain activities CBS had planned for the implementation of the Call to Action could not be undertaken as expected. CBS therefore needed to adjust its work plan accordingly for the last two years of the Health Canada Contribution Agreement. Thus, a more successful achievement of both intermediate and longer-term outcomes depends on the direction ultimately given on the implementation of the recommendations in the Call to Action.^{Footnote 67}

At the request of P/Ts, CBS recently conducted further analysis to prioritize individual recommendations in the Call to Action and to propose options for the sequenced implementation of recommendations over a longer period, which takes into account the current fiscal environment. Meetings between CBS and P/Ts held in early 2013 are expected to provide clearer direction on key issues around the Call to Action. This could include deciding which of the recommendations should be prioritized, as well as clarifying the accountability of CBS and P/T governments within an integrated, inter-provincial system such as that envisioned in the Call to Action. Overall, the interviews and surveys clearly indicate that, for next steps on the Call to Action, there will need to be continued and sustained interaction between CBS and the P/Ts, including enhanced communication, consultation and coordination.

Longer-term Outcomes

The results for the long-term outcomes of improved access, fairness and transparency for all patients on organ and tissues waitlists and increased public confidence in the Canadian transplant system were weak because there has been no movement on the Call to Action. That said, there are many areas of demonstrated success of the OTDT Program. In this context, most interviewees and 70-80% of survey respondents were generally positive about the potential of the OTDT Program as a whole to achieve its longer-term outcomes. This optimism primarily rests on:

1. the perceived high level of quality of the strategy presented in the Call to Action^{Footnote 68} and its responsiveness to Canadian needs (assuming some of its recommendations will be implemented in the short or longer term);
2. the fact that P/Ts have reiterated their commitment to these long-term outcomes;^{Footnote 69} and
3. the early success of the CBS OTDT Program relating to increased standardization and/or use of best practices.

Question: Is the CBS R&D Program producing the expected outputs and outcomes?

Immediate Outcomes and contributing Outputs

The evaluation evidence suggests that there has been increased awareness of the CBS-generated or supported research, knowledge products and tools within CBS and among the blood science and medicine community, including government stakeholders.

In terms of outputs, CBS-supported research has resulted in a large number of knowledge and dissemination products, particularly peer-reviewed journal articles and conference presentations (Table 9). On average, each CBS staff scientist publishes 3-4 peer reviewed papers per year, which some external expert interviewees noted was high in this context.^{Footnote 70} In addition, CBS has organized (sometimes jointly with Héma-Québec) or participated in several symposiums and forums to further disseminate and exchange the latest knowledge and research on issues related to blood operations and transfusion science and medicine. CBS-supported scientists also regularly produced technical reports (about 30 per year) and educational materials for both internal and non-CBS audiences (50-100 per year). Some CBS-supported research has been featured in the media.^{Footnote 71}

Note: This question was asked to all respondents (N=71 respondents).

Most surveyed CBS-funded researchers (nearly 80%) acknowledged that CBS was focusing on the appropriate research priorities. This finding is supported by the evidence presented in the 2010 performance review of the R&D Program as well as by key informants consulted for this evaluation. Moreover, the survey, interviews and documents indicated that CBS-supported research is well-perceived at the national and international levels, particularly in the United States.^{Footnote 72}

However, the degree to which these outputs have increased awareness and understanding is assessed mainly from anecdotal evidence and based on stakeholder perceptions. For example, when asked to demonstrate increased awareness, stakeholders pointed to proxy indicators, such as the publication record of CBS scientists and CBS-funded researchers, their presence at national and international (US) meetings of professional organizations gathering scientists and practitioners, as well as recent efforts by CBS and Health Canada's SPB to increase awareness within Health Canada about the R&D Program. The evaluation did not identify more tangible evidence that measured increases in awareness and understanding. In particular, there were no data available to provide a baseline level of awareness (i.e., before the commencement of the period under evaluation). The blood R&D community appears to be relatively close-knit and CBS is already seen as a key player in this community, so there might be few opportunities to further increase this awareness among researchers and practitioners already active in the field.

In contrast, the evidence (i.e., CBS' retention tracking database of scientists and technicians that it trains in blood science research) identified that building and maintenance of research capacity in transfusion science and medicine was one of the most significant impacts of the CBS R&D Program. This outcome was achieved through the training and personnel components (e.g., facilitating learning opportunities, offering grants and awards, including the CBS-CIHR partnership) of the R&D Program. Evidence of building and maintenance of research capacity in transfusion science and medicine was first found in the findings of the 2010 performance review.^{Footnote 73} As a result, more scientists and technicians are trained in blood science and staying in the field of blood science. In addition, the majority of consulted stakeholders (more than 75% of surveyed researchers and many interviewees) reported that the Program has contributed to attracting researchers to blood-related fields in order to maintain Canadian capacity/expertise. Survey evidence also showed that CBS graduate and post-doctoral awards have had a high level of impact on the career advancement of award recipients, such as through networking opportunities and the development of new partnerships related to blood research. About 70% of recipients of CBS fellowships who responded to the survey also reported that findings from their research were used within the health care sector.

Several lines of evidence indicated outcomes associated with collaborationand partnerships with diverse research organizations within Canada such as CIHR, universities, research hospitals, provincial and international organizations, such as with the World Health Organization. Survey results showed that the R&D Program enhanced the ability of CBS-supported researchers to collaborate with their colleagues, participate in or create partnerships and networks. Two-thirds of CBS-funding recipients credited their CBS-supported research as having generated collaboration networks and partnerships (Table 9). It is also worth noting that the revised "engagement" requirements for CBS intramural grants since 2011 (i.e., grants applications must include a CBS Adjunct Scientist, a CBS staff scientist and a CBS medical staff member) is intended to further encourage the use of networks and partnerships. Interviews and the 2010 performance review of the R&D Program indicated CBS's ongoing partnership with CIHR and its research hub model, including formal and informal partnerships with universities and hospital institutes across Canada (e.g., Ottawa General Hospital, McMaster University), have fostered collaborations seen as essential for CBS R&D's ability to deliver on its mandate.^{Footnote 74}

However, it was difficult to objectively assess or measure an increasein collaboration at either the national or international levels over the last five years as a result of the R&D Program because of the lack of available baseline data on pre-existing collaborations. In addition, the small number of researchers in blood R&D limits the potential for large numbers of new collaborations in this field. Nevertheless, evaluation findings show that collaborations and partnerships enabled the leveraging of resources and expertise, particularly the research hub model and the partnership with CIHR, and contributed to knowledge exchange and capacity building.

The most tangible evidence regarding collaboration comes from the survey of CBS-funding recipients. More than 50% of respondents credited the CBS R&D Program with facilitating both new and ongoing collaborations primarily with CBS scientists and academic researchers outside CBS (in the same or other disciplines), and to a slightly lesser extent (40%) with the health care sector. Hardly any collaboration was reported with Health Canada, which might have implications for the use of blood-related R&D to inform regulatory policy and science. Meanwhile, a modest number of respondents (13%) reported collaborations with the private sector, at least half of which were said to be new.

All lines of evidence indicated that the R&D Program supported the development of new/revised technologies, tools and procedures on blood manufacturing, transportation, storage and use. About one-third of survey respondents reported generating such outputs (Table 9). These outputs generated by CBS-supported research have to some extent been adopted in practice and applied in evidence-based decision-making. Based on the interviews, the survey and documents reviewed, this uptake has mostly been from within CBS. For example, research results have been used by CBS to troubleshoot or to improve operations, by making products/processes safer and more efficient. Notably, netCAD, which operates as a self-contained miniature blood centre as part of CBS' development function, was seen as a key element of the Program. It was found to have helped resolve operational issues and challenges and increased the safety and effectiveness of CBS' blood operations by examining modified procedures or testing new technologies.

CBS' intellectual property portfolio continues to grow. For example, in 2011-12, five patents were issued to CBS by the United States Patent Office and three from the European Patent Office. In addition, a few surveyed researchers who are not CBS staff scientists reported filing or obtaining patents on the results of CBS-funded research since 2008.

Some R&D-supported results have also reportedly been used and applied outside CBS (e.g., in the health sector). For example, CBS R&D knowledge has been used in areas such as clinical practices and blood and blood product storage and use.

Some key examples of the application and adoption of R&D results within and outside CBS include:

• the optimized buffy coat method for blood manufacturing (a centrifugation process used in blood manufacturing), which resulted in safer and more economical internal practices;
• selective testing for pathogens with rare occurrence that led to significant savings for Canadian taxpayers;
• extended storage time of thawed plasma that significantly reduced wastage of the product in hospitals; and
• the spin-off biotechnological company related to the ThromboLUX^TM Technology.

Work of the CBS R&D Program has also informed regulatory submissions made to Health Canada^{Footnote 75} for the approval of changes in blood manufacturing, blood products storage time, and other changes. These changes have led (or are expected to lead in the cases of submissions currently awaiting approval) to more effective and efficient practices, such as limiting inefficient use (losses) without compromising the safety or quality of the blood or blood products.

Intermediate and Longer-term Outcomes

Overall, the findings relating to immediate outcomes indicate that the CBS R&D work has contributed to supporting the safety, quality and supply of blood and blood products. However, it is too early to tell the degree to which intermediate and longer-term outcomes can be achieved.

Some evidence suggests that the uptake and use of the CBS-funded information/products could be greater to achieve outcomes. In particular, the survey results suggested low to moderate use of CBS-funded research results both within and outside CBS, although the survey data did have some limitations for this outcome.^{Footnote 76} It is likely that some R&D results were at too early a stage or that various conditions (e.g., regulatory approvals, technology transfer, commercialization) had not yet been met to enable their application in practice. Indeed, the time frame for the occurrence of these types of outcomes should not be underestimated, especially in fundamental research. Moreover, based on other sources of information (e.g., interviewees, document review), CBS-supported researchers are not always aware of how their research is used or what areas of research would be needed by potential users because of limited interaction and exchange with research users.

The 2010 performance review^{Footnote 77} of the CBS R&D Program noted a "lack of a clear strategic plan, driven by scientific and operational needs". Similarly, some interviewees suggested that closer alignment of R&D activities with operational, regulatory and scientific needs could lead to increased use of R&D results. However, stakeholders generally recognized the importance of remaining flexible and maintaining broad areas of R&D expertise/capacity to adequately respond to emerging issues and changing contexts (e.g., pathogens, new technologies, scientific breakthroughs, regulatory and operational needs).

In response to the issues identified above, the evaluation found several areas where CBS is already taking action. CBS is responding to the 2010 performance review through a variety of program improvements^{Footnote 78} and by identifying overall R&D priority areas^{Footnote 79}. CBS developed a project/program-based work plan outlining proposed activities, schedule, anticipated benefits and responsible groups for 2012-13 to meet reporting requirements for Health Canada. This plan and subsequent annual work plans are expected to contribute to strengthened program planning and delivery, as well as ensure that CBS achieves its targeted outcomes for the R&D Program. At this time, the work plan is not aligned with a logic model or performance measurement framework, but Health Canada incorporated these tools in the current funding agreement. Finally, the establishment of the new Research and Education function is also expected to contribute to greater achievement of changes in policy and practice through a broader adoption of knowledge and/or innovation (long-term outcome). Overall, although the recent adjustments to the R&D Program were generally seen by stakeholders as positive signs of progress, it is too early to assess their impact on this long-term outcome.

Question: Is Health Canada producing the expected outputs and outcomes?

All lines of evidence confirm that Health Canada has engaged in active program management of both the R&D and OTDT Programs. In accordance with respective funding agreements, staff at Health Canada have carried out performance monitoring, provided feedback (as appropriate), and conducted risk assessments for both programs annually. This assessment concluded that the risk associated with the R&D Program has consistently been low, while the OTDT Program is at medium risk.^{Footnote 80} The evaluation identified that the program management of both Programs has improved over the period under evaluation, which has led to enhanced relationships, accountability and program delivery.

CBS OTDT Program

The SPB took over the oversight of the OTDT Program in December 2008 from HPFB. Shortly after this date, SPB staff began working closely with CBS to develop and agree on CBS's five-year work plan, performance measurement framework, and reporting structure and templates. Subsequently, the Department has been diligently monitoring the OTDT performance and providing detailed feedback to CBS on the Program's quarterly reports, as shown by several documents reviewed for this evaluation.^{Footnote 81} These activities, with a good working relationship established between the two organizations, were reported by both groups of stakeholders (i.e., Health Canada and CBS staff) to have been beneficial. More specifically, they have contributed to improved performance reporting by CBS for accountability purposes, and perhaps more importantly, improved the management and delivery of the Program. For instance, CBS has benefited from this exercise by ensuring that the activities conducted under the Contribution Agreement stayed focused on priority areas and outputs were delivered in accordance with the Contribution Agreement.

Moreover, SPB staff have circulated CBS's quarterly performance reports to relevant Divisions within Health Canada (e.g., federal-provincial-territorial relations, policy priorities and analysis) to obtain broader context on OTDT issues and thus improve the feedback provided to CBS on OTDT activities. The initiation and coordination of this process by SPB has also contributed to internal relationship building and knowledge exchange within the Department. For example, Health Canada representatives outside SPB reported that this knowledge sharing approach has been useful to inform their policy and regulatory work (e.g., to help prepare briefings).

CBS R&D Program

Historically, the reporting requirements on federal grant recipients have been minimal (e.g., annual financial summary). As such, Health Canada originally had minimal expectations and engagement with the CBS R&D Program. After the formalization of the grant agreement with CBS in 2010-11, SPB's management of the CBS R&D Program became more active, and improvements were noted both in reporting and interactions, as described below. Moreover, the level of engagement of the Department, not to mention more stringent reporting requirements, will increase with the transition of the grant to a contribution starting in 2013-14.

Various stakeholders involved indicated that this recent enhanced program management approach has had some positive impacts. For example, they reported a strengthened working relationship and greater two-way communication between Health Canada and CBS, more comprehensive annual reporting to Health Canada (e.g., planned budget, work plan), and ongoing work to develop reporting templates (suitable for both parties) and clarified reporting requirements.

SPB recently began to share the annual CBS R&D Activity Reports with relevant divisions identified within the Department (e.g., Biologics and Genetic Therapies Directorate) for information-sharing purposes. Input from other divisions on these Activity Reports provides SPB staff with insight into scientific issues to help manage the CBS R&D Program, and reviewing these Activity Reports provides other divisions with information that can be useful to inform their regulatory and policy work. Staff members have initiated relationships within the Health Portfolio interested in the work conducted by CBS in blood R&D (e.g., CIHR, PHAC). In particular, the process to set up a Health Portfolio working group has already been initiated by SPB staff for the R&D Program. Subject matter experts participating in this working group provide input on work plans and performance reports under the Contribution Agreement to help the Program Management Division (PMD) better manage and oversee the R&D Program.

Finally, SPB staff has helped coordinate knowledge exchange sessions where CBS presented the work of the R&D Program to the relevant regulatory unit within Health Canada (Biologics and Genetic Therapies Directorate, HPFB). The first such session was in 2012 and a second was held in early 2013. These events were recognized by participating parties as having opened opportunities for further and closer collaboration between the two organizations. These efforts, and the implementation of the new performance measurement strategy under the Contribution Agreement, are expected to facilitate the knowledge exchange between CBS and Health Canada to ensure that the funded R&D work aligns to a greater extent with Health Canada's regulatory and policy needs and increases the use of R&D to inform policy work.

4.2.2 Demonstration of Efficiency and Economy

Summary

The CBS Contribution and Grant Programs were assessed as providing value for money, based on low program management input costs, favourable OTDT cost comparisons with other countries, leverage from other sources (e.g., P/Ts, universities), efficiencies gained by building on previous efforts and promoting measures to save costs, as well as views that the Programs produced tangible results for modest costs.

There was consensus that CBS is the most appropriate organization to deliver the Programs and that contribution funding is seen as an efficient instrument for achieving the intended goals. Suggestions were identified on potential adjustments through which the R&D Program could improve its resource use and effectiveness.

Question: What are the costs in relation to delivery of the CBS Grant and Contribution Programs?

Health Canada

The management of both Programs by the SPB was found to be lean, requiring minimal resources from Health Canada. The oversight costs over the period subject to this evaluation were estimated at 0.6 FTE, equalling $350,000 for both Programs.^{Footnote 82} The costs for SPB represented about 0.7% of the total annual funding allocated to the CBS Grant and Contribution Programs (1.1% for OTDT Contribution; 0.4% for R&D Grant). Note that this estimation does not include SPB operating and administrative costs and the salaries of individuals from other Divisions that collaborated with SPB staff for the delivery of the Program (e.g., input on quarterly reports and annual plans).

Interviewees recognized that more time and effort on behalf of SPB staff would be needed to manage the R&D Program after the anticipated transition from a Grant to a Contribution. In addition, some extra tasks would be conducted by representatives from Health Canada, the Public Health Agency of Canada and CIHR to participate in a working group for the R&D Program. However, these increases are not anticipated to generate significant additional costs to the Department (or other governmental departments/agencies). This change is expected to result in enhanced program planning, management, accountability (performance and financial reporting) and, consequently, greater effectiveness of the R&D Program.

CBS OTDT Program

From 2008-09 to 2011-12, CBS OTDT Program spent nearly the entire annual funding from HC of $3.58 million to deliver the planned activities under this contribution. These activities are categorized under four components: i) National System Design, ii) Leading Practices, iii) System Performance, and iv) Administration.^{Footnote 83} In addition, the OTDT Program used about $3 million from P/Ts (excluding Quebec) to support these activities over this period. The Program also reported obtaining about $395,000 in funding from other sources in 2009-10 and 2010-11. Note that regular annual P/T funding for OTDT is equal to the Health Canada contribution amount (i.e., $3.58 million per year) and is allocated to build and implement the national patient registries.^{Footnote 84}

Based on the distribution of expenditures by Program component (Table 10), the National System Design (i.e., development of the Call to Action) represented more than half (57%) of the total OTDT Program expenditures.^{Footnote 85} Overall, the National System Design component exceeded the planned budget, while other components expended funds below the planned budget.

Source: Compiled by Science-Metrix from data provided by CBS.

Despite annual fluctuations, administration costs over the period remained in line with the planned budget and represented 23% of expenditures. The first year focused on establishing the Program within CBS, resulting in the higher allocation for administrative costs. Subsequently, resource allocation focused on the National System Design, peaking in 2010-11 (78% of annual expenditures) with a shift the next year (2011-12) from administrative costs and National Design to Leading Practices which represented 34% of planned annual expenditures in 2012-13.

CBS R&D Program

Between 2008-09 and 2011-12, the CBS R&D Program operated with an annual budget of $10 million, composed almost equally of Health Canada and P/T funding. The P/T R&D funding is determined each year as part of the corporate annual internal budget negotiations and determined by CBS senior and R&D management. Historically, P/T funding for R&D has been set at $5 million, but for the 2013-14 budget, CBS management negotiated a small increase, up to $5.1 million.

As shown in the Program profile (Section 2.4), the annual R&D expenditures funded from the Health Canada grant consistently exceeded the annual allocation of $5 million. Indeed, these were $6.1 million per year, on average. This is because the R&D Program has reportedly used federal funding that was deferred from previous fiscal years. As such, expenses may not reflect actual funds received in any given year. From 2008-09 to 2011-12, R&D expenses reported in the CBS Annual Reports (financial summaries) have increased from about $11 million to about $12 million.

Typically, the P/T funds have been used by CBS to support research staff costs (i.e., salaries), administrative costs, and a small amount for the replacement of research equipment. The Health Canada grant funding therefore usually covered remaining costs to support CBS' research program and projects, including grants and awards. CBS can use funding from Health Canada or from other sources in a flexible manner across specific activities depending on needs. As such, some activities might be funded through Health Canada one year and not the next.

Some expenditure data were available for Health Canada grant funding, but no data was available for funding from P/Ts or from other sources.^{Footnote 86} Given the flexible use of funds described above, this made it impossible to examine trends in the distribution of total R&D Program funds across specific activities/components during the evaluation period. Therefore, the evaluation could not conduct an unambiguous assessment of the total costs of the CBS R&D Program by component.

Nonetheless, it remains possible to discuss the allocation of Health Canada funding. When looking at the funding by program component (see Table 11), CBS allocated nearly equal amounts of Health Canada funding to CBS-CIHR grants/projects (Schedule 4 and 6) and to internal R&D projects (Schedule 2, including fellowships).

Source: Compiled by Science-Metrix from data provided by CBS

Based on its planned expenditures, CBS intends to allocate a greater amount and proportion of its Health Canada funding in 2012-13 for "Other R&D (Schedule 2)", which includes the Laboratories, R&D management (R&D Office and Legal R&D), and the newly established Research & Education function. Again, because P/T funding is also used to support these activities, it is impossible to determine whether this represents an actual increase in funding to these components.

Question: Are the CBS Grant and Contribution Programs the most efficient means of achieving intended objectives?

The evaluation evidence was not fully conclusive in terms of assessing the efficiency of resource utilization by the CBS Grant and Contribution Programs. This is largely because both programs rely on multiple funders and there was a lack of detailed financial reporting across the funding sources, particularly with the R&D Program. Nevertheless, there is a general perception across stakeholders that both programs are cost-effective policy instruments. They are seen to use a relatively modest contribution or grant amount, while generating tangible results of value to these stakeholders and to Canadians in general. In addition, both programs have a high potential for long-term impact to improve the health and safety of Canadians.

CBS OTDT Program

Overall, the evaluation evidence suggests that CBS used the contribution funding efficiently to develop and table the Call to Action. In particular, the CBS OTDT Program built extensively on past CCDT work, progress and success. The corporate memory and relationships gained at the time of the CCDT have benefited CBS because key CCDT personnel transferred to CBS to form the current iteration of the OTDT Program in 2008.

Timing issues were a determining factor around the efficient use of resources. As shown in Table 10, resource allocation in the first year of the OTDT Program at CBS (2008-09) appeared to focus largely on establishing the management and administration of the Program. The main focus of the two subsequent years was on the development of the strategic plan for the national system (i.e., the Call to Action). In the last two years, the terms and conditions of the Contribution Agreement affected CBS' ability to make optimal use of funds while waiting for direction from P/Ts on the Call to Action. Because the deferral of funds from one FY to the next was not allowed for contributions agreements, the OTDT Program had to shift some planned resources for the national system design and implementation to other initiatives (e.g., Leading Practices). Although this benefited the other initiatives, some interviewees noted that more flexibility around the deferral and/or use of Health Canada funds would have allowed CBS to reserve financial resources for the anticipated period of higher activity around the implementation of the Call to Action.

Timing issues in the response of P/Ts to the Call to Action may have other implications for efficiency and resource use, such as potential duplication of effort between CBS' implementation of the national system recommendations and recent and ongoing OTDT initiatives within individual provinces.

Outside of this timing issue, key informant interviews indicated that the contribution funding to CBS is an efficient instrument for achieving national system design. CBS and Health Canada interviewees considered the CBS OTDT Program to be a good investment as it has the potential to generate a breakthrough national OTDT performance (e.g., reduce transplant waitlist times, increase domestic production of tissue products and help mitigate transplant tourism) with a relatively small investment. CBS interviewees also reported a high level of delivery/success in all three pillars of activity funded by Health Canada, considering the available level of funding. However, some external interviewees had reservations about the way CBS allocated its resources across activities, as they considered that particular components they viewed as priorities (e.g., leading practices, P/T-funded registries) could have been delivered in a more timely way if CBS had devoted less time/effort to the development of the national strategy under the current Contribution Agreement.

CBS interviewees, in particular, saw significant "value added" for the OTDT Program in becoming part of CBS. By benefiting from the existing capacity, experience, and national operational authority of this organization, the OTDT Program can achieve greater efficiencies and pursue an enhanced mandate. The CBS governance, corporate services and financial and performance policies also benefited the OTDT program. In other words, the OTDT Program could depend on these services and policies rather than having to develop their own. Meanwhile, CBS was reported to have benefited from the expertise of OTDT staff in community engagement activities, including organizing workshops.

CBS R&D Program

Overall, CBS used the grant funding efficiently to deliver the R&D Program and to produce a significant level of outputs over the period subject to this evaluation. In particular, the R&D Program was seen to have achieved efficiencies as it maintained its level of output and responded to expectations to increase its development/applied research efforts, although the funding levels for the R&D Program have remained largely unchanged since 2000, while research costs have increased steadily over this period (mainly due to inflation).

CBS representatives pointed to several practices used to ensure efficiencies, such as monitoring the performance and expenditures of CBS-supported projects to ensure efficient spending of the Program's resources. The allocation of available resources across activities of the R&D Program was generally considered appropriate by interviewees and was also found to be adequate in a recent independent performance review of the Program.^{Footnote 87} Since this review, there has been a continued shift toward supporting more applied/operational research, specifically to support projects carried out in the netCAD centre. This centre conducts research to troubleshoot issues encountered in CBS' everyday operations, among other activities. This transition was generally well received by most key informants across various groups. However, it was not unanimous. Some suggested the need for a greater focus on research that supports regulatory decision-making to ensure its use/uptake. Other key informants had mixed views on the ideal balance between basic and developmental research.

As discussed earlier, the need for research capacity support in blood R&D is ongoing, whereas the expectation for science-based decision-making for blood operations is increasing. The R&D Program primarily relies on fixed federal and P/T funding and its reserve of deferred federal funding is diminishing, while research costs continue to increase. This suggests that either alternative funding sources or increased internal allocations would need to be secured to maintain and enhance its R&D activities in coming years. As an example, CBS recently created the Research & Education function to support knowledge mobilization and application of research in the Canadian health care system in response to identified needs. This function was added to CBS R&D activities ($336,000 spent during FY 2011-12 and $506,000 planned for FY 2012-13) without a corresponding increase in federal or P/T funding, or without securing additional sources/revenues, such as, from foundations or not-for-profit organizations.

Finally, performance reporting has also improved over the period subject to this evaluation in a way that supports more efficient program management and delivery. Since, 2009-10, the research activity reports submitted to Health Canada are more detailed. This was found to have benefitted the work of other groups within Health Canada, as well as by the P/Ts consulted. The annual reporting is considered adequate within the Health Canada Grant Agreement, but CBS and Health Canada have developed a more comprehensive performance measurement framework to report on activities, outputs and outcomes under the new Contribution Agreement. Importantly, the provision of more complete financial information (including both Health Canada and other sources of funding) is also being discussed. This would help capture and assess the planned and actual expenditures of all sources of funding for each activity.

Question: Were the CBS Grant and Contribution Programs managed in the least costly way?

CBS OTDT Program

Overall, the evidence indicates that CBS manages and uses its contribution resources appropriately to ensure the economy and efficiency of the OTDT program. CBS also obtained additional revenue from P/Ts for two years (2009-10 and 2010-11) to help support the development of leading practices, performance metrics, and the Call to Action. This P/T allocation (nearly $3 million) corresponds to over 16% of the total Health Canada funding ($17.9 million) under the current Contribution Agreement.

Key informants within CBS stressed that the consultative process used to design a new national OTDT system was based on identifying and leveraging existing knowledge, capacity and expertise from members of the OTDT communities. This included the input of stakeholder groups and committee members, who largely offered their expertise on a volunteer (unpaid) basis. The notion of building on existing strengths is also apparent throughout the Call to Action report, in which there are mentions of pockets of expertise, existing practices, and the need to build on these by focusing on specific problems and performance gaps.^{Footnote 88}

There was a general perception among internal and external interviewees that CBS manages and uses its contribution resources in an economical and efficient manner. However, it should be noted that several interviewees from Health Canada and external to CBS indicated they had limited knowledge on how CBS uses its resources to limit costs and so were not often in a position to cite specific examples of efficiencies.

Nonetheless, CBS and some external key informants identified several efficiencies and cost saving practices. In particular, the OTDT Program has leveraged expertise within CBS (e.g., legal, financial, IT) and implemented policies to limit travel costs (e.g., advance planning to take advantage of cheaper rates, use of strict reimbursement policies). Importantly, as pointed out by many key informants and confirmed by program documents, CBS takes advantage of existing meetings, conferences or events^{Footnote 89} that bring OTDT stakeholders from different levels/regions together, thus saving on both organization and travel costs. This practice not only helped CBS conserve resources (as opposed to organizing additional meetings with these groups) but also likely helped attract more attendees who were interested in attending both the CBS meeting and the concurrent event.

However, several interviewees across stakeholder groups expressed concerns around potential duplication of efforts and unnecessary costs that could arise if the Call to Action is implemented by CBS as proposed. Indeed, some of the recommendations in this plan could overlap with existing or emerging OTDT systems implemented in some P/Ts while the Call to Action was being developed and reviewed. Interviewees from CBS and other groups therefore saw this as a timing issue, especially given the length of time taken by P/Ts to conduct their jurisdictional analysis and prepare their response (at least 18 months since the Call to Action was tabled with P/Ts).

CBS R&D Program

While there was limited documentation on CBS' R&D management practices, stakeholders reported lean management within the organization. CBS representatives in particular reported constant evolution and improvements of the Program toward more efficient program management and cost savings in the past due to the 2010 performance review. Several interviewees, including those outside CBS, pointed to the modest budget of the program given the costs of conducting research.

Interviewees generally agreed that the Program has been using its resources efficiently by allocating its resources on topics that focus on the safety, supply and efficiency of the Canadian blood system. To ensure that CBS' funds are allocated to high-quality research, all grant and fellowship applications under this Program are peer-reviewed. Moreover, the R&D Program has introduced priority areas of research, which are determined and reviewed annually by the Scientific Research Advisory Committee of independent experts to help ensure that the CBS-supported research is focused on areas applicable to CBS operations.

Importantly, interview and documented evidence shows that the adoption of research findings within CBS has resulted in increased efficiencies in CBS operations. These resulted (or are expected to result) in improved practices that demonstrate greater efficiency and savings. However, Health Canada noted that some CBS submissions in the last few years should have included more comprehensive science-based evidence. Health Canada thus expects that CBS' submissions for operational change be supported to a greater extent by R&D Program outputs.

The evaluation evidence also confirms that the Program has successfully leveraged additional resources through partnerships and coordinates with Héma-Québec to avoid duplication of effort on similar operational issues. However, given that precise data on leveraged funding, in-kind contributions, and infrastructure was not available, the evaluation was not able to measure the degree of leveraging. According to interviewees and the performance review,^{Footnote 90} key partnerships that contribute to CBS' leveraging success are those with CIHR and other research organizations. In particular, the CBS-CIHR partnership resulted in more high-quality research being funded in areas of interest to CBS because, in response to proposals submitted through this partnership, CIHR was the paying partner (from its own Operating Grants budget) for grants totalling more than $4 million over the evaluation period (2008-09 to 2012-13).^{Footnote 91} Further efficiencies were achieved by transferring the peer-review process of partnered grant applications to CIHR, instead of CBS conducting this process with its own budget.^{Footnote 92}

Interviews and documents also found that cost savings were associated with the R&D Program's research hub model. In this model, several CBS researchers (i.e., CBS salaried staff) and CBS adjunct scientists are located in academic and research organizations across the country. This constitutes an important leverage of Health Canada investments with the home organization (these were the Centre for Blood Research (Vancouver, BC); the McMaster University Transfusion Research Program (Hamilton, ON); and the Ottawa Hospital Research Institute (Ottawa, ON)) to promote a critical mass of expertise. The hub model also supports greater capacity development and training, and contributes to collaboration and high-quality research more generally.

Question: Are there alternative approaches which would be more economical and/or effective given the available resources and funding mechanisms for the CBS Grant and Contribution Programs?

CBS OTDT Program

There was consensus among stakeholders that Health Canada should continue to fund the program via a Contribution Agreement. In addition, all lines of evidence showed that CBS is the most appropriate organization to deliver the OTDT Program. Interviewees across all stakeholder groups (Health Canada, CBS and external) indicated that, as a trusted organization with potential corporate synergies, CBS is well positioned to continue the improvement of national coordination of the OTDT system. As such, their suggestions for alternative approaches focused on adjustments or clarifications regarding the future role and responsibilities of parties involved in the Program, as well as opportunities to work more closely with other groups, particularly national organizations and P/T governments.

Given the uncertainty around the direction of P/Ts on recommendations in the Call to Action, stakeholder perspectives differed around the future direction of CBS. CBS interviewees indicated that the organization/Program was on the right track, but recognized that they had some challenges ahead, such as working with F/P/T governments to obtain a clearer and more long-term mandate on OTDT. CBS also expects to develop new corporate structures and infrastructure to support its future role/mandate once the P/T decision on the Call to Action is known. Health Canada representatives identified a need for CBS to collaborate more closely with other organizations such as Héma-Québec, CIHI or CIHR to ensure that CBS leverages existing expertise and resources. Similarly, external interviewees and survey respondents tended to see a greater need for the organization to align more closely with P/T needs and priorities.

Finally, Health Canada plans to continue building and maintaining internal linkages among groups within the Department (e.g., federal-provincial-territorial relations, policy priorities and analysis) and share knowledge generated by the Program within the Health Portfolio. Meanwhile, CBS stressed that Health Canada's continued support, in terms of both funding and leadership, was crucial to the current and future success of the program.

CBS R&D Program

As the national blood operator, CBS is well positioned to continue its role in building blood-related R&D capacity, as well as increasing awareness and uptake of new blood products, technologies and blood-related processes among health care professionals. Although this view was shared by almost all stakeholders consulted, several provided suggestions on potential adjustments through which the Program could improve its resource use and effectiveness. These are discussed below.

The planned transition of the current funding arrangement from a grant to a contribution was the most discussed alternative by both Health Canada and CBS interviewees. As mentioned in Section 4.1.3, the impetus for the transition comes from Health Canada, whose representatives have been working with CBS to help inform this transition. Indeed, this change will impact the relationship between the two organizations, as well as resource requirements for more detailed and more frequent reporting by CBS and more in-depth review by Health Canada. Both organizations generally view this transition as a positive change, as they expect it to result in a closer collaborative relationship and improved program management. More specifically, the Department would obtain a more comprehensive and detailed view of the management and performance of the R&D Program as a whole (including elements funded by the P/Ts) to help steward the program's areas of focus (e.g., by helping the program stay on track and by providing input on potential research topics/areas that the Program could pursue with federal funding).

CBS staff indicated that they are constantly improving the Program in line with the most recent developments and needs of the field. They are also actively pursuing avenues to enhance the impact of its R&D activities and/or obtain additional resources from other sources. External stakeholders suggested CBS could enhance collaborations with national and international partners. For example, by leveraging to a greater extent the basic research expertise in academia and via international collaborations, CBS could free some of the R&D Program's limited resources for more operational research. A detailed analysis of the scientific areas of expertise of CBS-funded scientists and that of other Canadian blood research peers in Canadian and international universities would allow CBS to position its research and identify potential mutually beneficial collaborations in this regard.

CBS and Héma-Québec both participated in a 2010 survey of over a dozen countries with active blood R&D programs, which revealed some alternative funding models or ways of delivering the program.^{Footnote 93} For instance, while most countries support transfusion science programs through government funding or from the operational budget of blood centres, the Dutch organization Sanguin does not receive any direct government funding for research and instead funds its R&D program from sales of blood and plasma products, charity contributions and other sources. Combined sources of funding are also common. For example, in the UK, the R&D program is run from a combination of (i) a levy on the blood price to cover research, reviews and clinical trials, (ii) funding from the Department of Health, and (iii) external funding won by researchers from research councils, the European Union, charities, and industry.

The examples highlighted above demonstrate that there are a variety of models that can be used to fund blood R & D. Not all of these models are applicable to CBS.  Given the static state of blood R & D funding, CBS could begin to consider the various models that may lead to greater funding opportunities and a more sustainable level of funding.

5. Conclusions

5.1 Relevance Conclusions

Both components of the CBS Grant and Contribution Programs continue to address a demonstrable need and remain responsive to the needs of Canadians.

The range of activities and priority areas pursued by the CBS OTDT Program under the Contribution Agreement, are appropriate and well aligned with the ongoing need to improve the performance of the Canadian OTDT system. This includes improving health outcomes, as well as improving access, fairness and transparency for all patients on organ waitlists. Further improvements for the integration and coordination of the national OTDT system continue to be necessary.

Regarding the CBS R&D Program, although the Canadian blood system is now widely recognized as safe, there continues to be a need for scientific and evidence-based information to maintain and increase the safety, supply and efficiency of the Canadian blood system. This includes ongoing efforts to stay abreast of technological developments and to address the inherent risks associated with blood transfusions because of known and emerging pathogens. The R&D Program does so by advancing innovation and maintaining Canadian capacity in transfusion science and medicine. As such, the Program was generally found to be well aligned with related operational, regulatory, knowledge, and capacity needs.

Different groups of stakeholders agreed that CBS was the organization best positioned to respond to the needs associated with both the OTDT and R&D Programs. However, some challenges were noted in terms of next steps. The OTDT Program is currently in a transition phase, as it awaits direction from P/Ts on the Call to Action recommendations. The future role and mandate of the Program relating to the implementation of this strategy will largely depend on the decision of P/Ts on these recommendations. Meanwhile, the R&D Program is under increasing pressure from regulatory stakeholders to focus to a greater extent on addressing operational and regulatory needs. With the renewal of both programs (one year CA for OTDT and five years for R & D), there is an opportunity for Health Canada to clarify its expectations with regard to the long-term outcomes of each program.

Both components of the CBS Grant and Contribution Programs are aligned with government priorities.

Both programs are aligned with the Government of Canada's priority and related work to protect the health and safety of Canadians, as well as Health Canada's strategic outcome to support access to safe and effective health products and protect Canadians from health risks. The OTDT Program also aligns with Government of Canada and Health Canada health care system priorities, such as working with P/Ts to ensure the sustainability and accountability of the system, and ensuring the system responds to the needs of Canadians.

Both components of the CBS Grant and Contribution Programs are aligned with federal roles and responsibilities.

The CBS Grant and Contribution Programs' activities fall within the federal role, including Health Canada's mandate, regulatory role, and jurisdictional responsibilities for the regulation of biological health products. They align with the federal role in ensuring that all Canadians are provided fair and equitable access to health care, in collaboration with the provinces and territories, in accordance with the five principles of the Canada Health Act.

The nature of Health Canada's role for the OTDT Program (other than providing funding and oversight of the Contribution Agreement) was subject to differing expectations. Stakeholders across Health Canada, CBS and P/Ts had different views on the appropriate level of direct involvement and provision of "national leadership" from Health Canada on this file, which could also change if CBS plays a larger role in OTDT governance and service delivery (depending on which recommendations of the Call to Action are eventually implemented).

On the other hand, closer involvement and knowledge sharing of Health Canada with CBS' R&D Program was generally welcomed by representatives from both parties. In addition, greater opportunities for discussions of research results, agendas and needs that involve both key blood R&D researchers and users (i.e., regulators and blood operators, including Héma-Québec) were identified as being beneficial to the Canadian blood system in order to help inform research priorities and identify collaboration opportunities.

5.2 Performance Conclusions

Both components of the CBS Grant and Contribution Programs have produced expected outputs and are making progress toward their expected immediate and intermediate outcomes. Health Canada's management of the OTDT Program has been effective, while that of the CBS R&D Program has improved significantly in recent years.

The OTDT Program, through its extensive consultation and relationship-building activities, has raised awareness and understanding of the OTDT Program and priorities, developed improved system performance metrics, and supported an enhanced accreditation framework (immediate outcomes). Increased coordination and integration of OTDT activities was observed. Similarly, progress was observed toward intermediate outcomes, including the adoption of new/modified knowledge, policies, standards and regulations, as well as adoption of improved health-care practices and procedures. Many stakeholders stressed that, as CBS continues to build on past work and successes to date, there remains strong potential for greater achievement of intermediate and long-term outcomes. However, the extent to which these outcomes will be achieved is largely dependent on the direction given by P/Ts about the recommendations in the Call to Action.

The CBS R&D Program has generated numerous outputs in the important areas of basic and applied research that support the mission of the CBS. The Program has shown a good level of achievement across most of its immediate and intermediate outcomes. In particular, the Program has played a key role in building and maintaining research capacity in transfusion science and medicine. Moreover, several examples of application and adoption of R&D results within and outside CBS were identified (e.g., optimization of buffy coat method, selective testing for pathogens) that contributed to a greater effectiveness and efficiency of the Canadian blood system. However, the findings suggest there is further room for improvement in knowledge exchange and collaboration with research users to facilitate greater adoption and application of knowledge or innovation results in policy and practice. CBS has already begun to focus its efforts on these areas, but it is too early to assess the success of these efforts. Future improvements to the R&D Program could include more sustained exchange between Health Canada and CBS on R&D issues, including other key stakeholders in blood R&D. This was seen as a means to achieve even greater benefits from the work of the R&D Program from the perspective of research users. For example, Health Canada representatives identified a growing interest in and need for better awareness of CBS R&D activities, which could be addressed through more regular exchange between their regulatory scientists and CBS.

Health Canada's management of the CBS OTDT has been effective over the period under evaluation, while that of the CBS R&D Program has improved significantly in recent years. The outputs of these management activities have contributed to enhanced accountability and more focused program delivery for the OTDT Program and, more recently, for the R&D Program. The management activities have also helped enhance knowledge sharing and strengthened relationships within Health Canada, with other government partners in the Health Portfolio and with the CBS around issues associated with these two Programs.

Economy and Efficiency

Both components of the CBS Grant and Contribution Programs have demonstrated economy and efficiency in their operations and management. Alternative funding models have been identified that may help CBS identify additional funding sources to support its R&D activities and meet growing expectations of its stakeholders.

The inputs to manage both Programs by Health Canada were estimated at $300,000 for the entire period under review (2008-09 to 2012-13), or less than 1% of the total funding allocated for the Grant and Contribution Programs.

The CBS OTDT Program allocated the $3.58 million per year provided by Health Canada to conduct the activities outlined in the Contribution Agreement. These activities were also supported by other sources; particularly an additional $3 million from the P/Ts. National system design (i.e., development of the Call to Action) represented more than half of the OTDT expenditures from Health Canada funding. Interviewees were generally satisfied with the way CBS allocated the federal contribution across its activities and initiatives. The OTDT Program was also viewed as providing value for money given its potential to generate a breakthrough in national OTDT performance with a relatively modest budget, especially when compared to what other countries (e.g., Australia, UK) have spent on similar efforts. The CBS OTDT Program is efficient, having built extensively on past work, and successfully leveraged the capacity and expertise within the OTDT communities. Several cost saving practices were instituted by CBS, e.g., implemented policies to limit travel costs, piggybacked with existing meetings and events, used volunteers.

However, the length of time taken by P/Ts to respond to the Call to Action has implications for efficiency and resource use, which suggests a need to develop a flexible work plan for CBS during the current transition period. This plan could include a "building block approach" to implement next steps. Given that CBS is considered the most appropriate organization to deliver the OTDT Program, suggestions for alternative approaches focused on adjustments or clarifications on the future role and responsibilities of CBS and opportunities to work more closely with other interested parties.

The CBS R&D Program operated with a budget of $10 million per year, of which $5 million was provided by the federal government via the Grant Agreement and the other $5 million was allocated by P/Ts (excluding Quebec). Deferred funding carried over from previous fiscal years was also used, such that R&D expenses have represented $11 to $12 million in total from 2008-09 to 2012-13.

Overall, key informants indicated that CBS used the Grant funding efficiently to deliver the R&D Program and allocated its available resources adequately across activities. However, an unambiguous assessment of the costs of the CBS R&D Program by activity was not possible given lack of information on the use of P/T funds and other revenues. However, R&D funding from Health Canada and P/Ts has remained largely unchanged since 2000, while research costs have increased steadily over this period. Alternative funding models have been identified that may help CBS explore additional funding sources to support its R&D activities and meet growing expectations of its stakeholders (e.g., to increase development or applied research efforts).

The R&D Program has successfully leveraged additional resources through its partnerships with CIHR, universities and other research organizations. It has also implemented various measures to promote efficiencies and save costs (e.g., avoiding duplication with Héma-Québec, monitoring of research costs, focusing on priority research areas). The adoption of research findings has resulted in some increased efficiencies in CBS operations. Suggestions to improve the economy and effectiveness of the Program included greater collaboration with Héma-Québec and enhanced dialogue with Health Canada.

6. Recommendations

Two recommendations have been identified for the CBS Grant and Contribution Programs.

Recommendation 1 (OTDT and R&D):

The two CBS programs are experiencing a time of transition. The OTDT Program has developed the Call to Action (April 2011) and is awaiting a response from P/Ts. Many of the intended outcomes for the OTDT Program rely on the P/Ts to respond in full or in part to the Call to Action. Therefore, determining appropriate program objectives, taking into account possible responses to the Call to Action, will be necessary.

Health Canada has decided to strengthen its oversight by moving from a grant to a contribution agreement. Therefore, this would be an opportune time for Health Canada to refine and clarify its role as well as the program objectives.

Health Canada should determine the departmental results to be achieved with the two CBS Programs going forward and, based on this, clearly define the federal role and program objectives.

Recommendation 2 (R&D):

The current strategic direction for the CBS R&D Program is to increase knowledge mobilization activities, mainly through its Research & Education function, and to conduct more applied/operational research. The impact of these efforts should lead to an increase in the use of the R&D results.

Health Canada should look for opportunities to facilitate CBS' shift toward research and development which is focussed on operational and regulatory issues.

Annex A: Program Description

I. Program Profile

Organ and Tissue Donation and Transplantation (OTDT) Program

In 2001, the Canadian Council for Donation and Transplantation (CCDT) was established to help with the coordination of federal/provincial/territorial (F/P/T) activities associated with organ donation and transplantation. A 2006 summative evaluation determined that the CCDT was successful in addressing its objectives and highlighted that continued F/P/T effort was required to implement a coordinated strategy to improve OTDT in Canada.^{Footnote 94}

In December 2007, the Safety of Human Cells, Tissues and Organs for Transplantation Regulations (CTO Regulations) came into effect.^{Footnote 95} The CTO Regulations address safety in the processing and handling of these products, and standardize the screening and testing of potential donors to improve the protection of the health and safety of Canadian transplant recipients. The CTO Regulations incorporate by reference parts of the National Standards of Canada General Requirements for Cells, Tissues, and Organs for Transplantation and Assisted Reproduction. In addition, provinces and territories have each passed a Human Tissue Gift Act (or equivalent) regulating OTDT in their jurisdiction.^{Footnote 96}

In light of this complex legislative framework, and recognizing the need for improvement to the OTDT system in Canada, F/P/T governments decided in 2008 to transfer the CCDT's functions to CBS.

For the OTDT Program, CBS signed a Contribution Agreement with Health Canada in 2008, representing $17.9 million for a five-year period ($3.58 million per year).^{Footnote 97} Under this funding agreement, CBS's work concerning the OTDT program has been conducted in three pillars of activity. These pillars, listed below, are aligned with the program's primary objectives:^{Footnote 98Footnote 99}

1. Strategic Plan Development and Implementation;
2. Support Leading Practices / Public Awareness and Education; and
3. System Performance Improvement.

Concurrently, the provinces and territories (P/Ts) signed a Letter of Intent with CBS providing matching amount (an additional $3.58 million per year) for the development and operation of three national patient registries. These activities are funded exclusively through the P/Ts (excluding Quebec).^{Footnote 100} Although Health Canada funding was used to support some policy and coordination work associated with the registries, the registries are not covered by this evaluation, which focuses on the three pillars funded by Health Canada.

CBS has been mandated with the development and implementation of a national coordinated strategic plan on organ and tissue donation and transplantation (i.e., strategic plan development),^{Footnote 101} which would lead to improved performance of the OTDT system. In this context and throughout this evaluation, the term "improved performance" refers to a range of changes such as increased organ donation rates, reduced transplant waitlist times, increased fairness and transparency in the OTDT system,^{Footnote 102} and improved health outcomes for Canadians. In the first three years of the OTDT Program, the process to develop this plan included broad stakeholder consultations to bridge the diversity of the stakeholder groups (e.g., public, expert, government, international and other).^{Footnote 103} Drawing from these stakeholder consultations, CBS and its expert committees prepared a comprehensive guiding document on the implementation options and costing of the national OTDT system - the Call to Action.^{Footnote 104} The document (publically released in June 2012) provides 25 recommendations in total, 12 for Organ System implementation and 13 for Tissue System implementation. At the time this evaluation report was prepared, these recommendations were being considered by P/T governments.

CBS also engages in activities that support leading clinical practices and public awareness and education. For example, this includes the development and evaluation of clinical practice guidelines and recommendations, organ sharing algorithms, and supporting knowledge transfer to health practitioners. In practice, these activities translate into the organization of consensus conferences, collaborative sessions and other workshops. In addition, CBS is responsible for developing and disseminating education materials on OTDT for the Canadian public.

CBS is also mandated to facilitate and support system performance improvement through the development of performance metrics and accreditation standards. Since 2008, CBS has worked with Accreditation Canada on the development of a subset of standards and performance measurement indicators that will be incorporated into the general accreditation standards for all Canadian hospitals with a donation program. In addition, CBS, in consultation with a broad range of national and international stakeholders, has been developing guidelines and recommendations for improved OTDT measures and definitions. Notably, draft measures and targets have been included in the Call to Action.

Research and Development (R&D) Program

Since 2000-01, Health Canada has provided a grant of $5 million per year to support the R&D program, with approximately the same amount provided by P/Ts annually. With that funding, the CBS R&D Program seeks to drive continuous improvement and knowledge creation through research and development work into blood safety, alternatives to transfusions, and blood substitutes. Indeed, the R&D Program aims to advance Canadian innovation in transfusion^{Footnote 105} primarily through discovery research, training of highly qualified researchers, and development projects. These activities support the objectives to enhance CBS's products and services and to innovate, invent and discover information and products that will improve transfusion in Canada.^{Footnote 106}

One of the goals outlined in the original CBS strategic plan for the R&D program was to achieve national and international leadership in transfusion science R&D through the establishment of a national research network.^{Footnote 107} To this end, the program now encompasses several laboratories based in universities and research institutes across Canada. This expertise is concentrated in five research centers, or "hubs", each of which specializes in a particular area of transfusion science within CBS priorities: Vancouver - Blood Product Processing and Storage; Edmonton - Stem Cells, Nanotechnology and Cryobiology; Hamilton - Clinical Research; Toronto - Transfusion Immunology; Ottawa - Infectious Diseases.^{Footnote 108} CBS staff scientists (whose salary is paid by CBS) and CBS adjunct scientists (based at universities and research centres) and their teams also collaborate with a range of external partners.

CBS's work concerning the R&D program has been conducted across five functions, described in more detail below, which are aligned with the program's primary objectives:

• Research;
• Development;
• Intellectual Property & Business Development;
• Quality Monitoring Program; and
• Research and Education (as of 2011-12).

Through this network, the Research Activities & Support delivered by the Research function provides leadership and support for scientific research related to blood transfusion. This including the provision of operational grants and personnel funding awards in relevant areas for scientists (within and outside CBS), postdoctoral fellows and graduate students; note that some grants are delivered in partnership with the Canadian Institutes of Health Research (CIHR).^{Footnote 109} Applications to these programs undergo rigorous peer-review to ensure a high level of quality and scientific merit of supported research.

Meanwhile, the Development Activities & Support delivered by the Development function provides primary scientific support to the operational arms of CBS and helps with the development of research innovations. Much of this work is carried out by the Vancouver Network Centre for Applied Development (netCAD) facility, which is supported by functional sites in Ottawa and Edmonton.^{Footnote 110}

Within Operational Support activities, CBS also has a modest Intellectual Property & Business Development function that works on the protection of intellectual property arising from CBS-supported innovative research through patents.^{Footnote 111} Meanwhile, the Quality Monitoring Program examine the effects of clinical blood preservation and storage through regular product testing to provide CBS researchers (e.g., within netCAD) and operations with data to assist with product and process validation and troubleshooting.^{Footnote 112} For example, by helping monitor shipment temperature and product quality across manufacturing sites, this function allows CBS to select shipping routes to reduce product discards, and to inform modifications to blood manufacturing processes.^{Footnote 113} Finally, Research and Educationis a new function (as of 2011-12) that has been established to enhance knowledge mobilization of CBS knowledge, products and tools, including by collaborating with key public and private organizations involved in the Canadian health care system. For example, these efforts would seek to develop learning opportunities and educational resources, transfer knowledge to influence policies and practices, as well as to support technology transfer and commercialization.^{Footnote 114}

A central issue in Canada is that blood and blood components have been traditionally regulated as drugs by various provisions in the Food and Drugs Act. Following the Krever recommendations, new Blood Regulations^{Footnote 115} were developed by Health Canada's Health Products and Food Branch, with significant stakeholder consultation and guidance from an Expert Advisory Committee on Blood Regulation. At the time of this report, the Regulations were going through a last round of revision with the expectation of final publication in 2013, coming into force one year later.^{Footnote 116}

II. Program Logic Model

Source: Compiled by Science-Metrix from program documentation and consultations.

The integrated logic model for the evaluation of the CBS Grant and Contribution Programs is presented in Figure 1. This logic model was developed based on a review of available documentation and consultations with Health Canada representatives responsible for oversight of both programs. Although the logic model contains both programs, their activities, outputs and most outcomes are presented in parallel rather than combined because the consultations clearly indicated that, despite both being housed in CBS, the OTDT and R&D programs are operated completely separately.

Importantly, this integrated logic model was specifically designed for this evaluation and has not been used by CBS or Health Canada in the planning, management, delivery or oversight of either program. It also includes the activities conducted by Health Canada's SPB with regard to the CBS Grant and Contribution Programs.

The logic model uses a top-down diagrammatic structure to represent the program's composition and process flow. The organizational processes of the programs are presented in six main categories: resources (1.0), activities (2.0), outputs (3.0) and targeted short-term (4.0), medium-term (5.0) and long-term (6.0) outcomes. Major external factors and influences (0.0) on the design and delivery of activities as well as on the expected outcomes are also included in this logic model to provide additional context useful to the understanding of the program theory.

Generally, the time frame for short-term outcomes is one to three years, four to seven years for intermediate outcomes and eight or more years for long-term outcomes. The time frame for expected outcomes differs somewhat for each of the programs as the R&D Program has been operating since 2000, while the OTDT Program in its current form was established within CBS in 2008. The mandate of the OTDT Program and its objectives under the current Contribution Agreement with Health Canada are therefore different in the current evaluation period than in the pre-2008 period under the CCDT. In addition, the CBS R&D Program has undergone some important changes since 2000, including an enhanced focus on development activities, especially through the establishment (2003) and expansion (2008) of the netCAD centre.^{Footnote 117} As such, over the five-year period encompassed by the evaluation (2008-09 to 2012-13), the focus will be on the immediate (4.1 to 4.8) and intermediate outcomes (5.1 to 5.3). Findings on long-term outcomes stemming from pre-2008 activities are also noted periodically because of their importance from Health Canada's perspective. However, as these outcomes were not the focus of the data collection instruments (especially the survey and the document review) given the stated evaluation scope (2008-09 to 2012-13), findings on long-term outcomes are somewhat limited.

The top row of the logic model (boxes 0.1 to 0.4) contains elements that highlight the complex regulatory and multi-jurisdictional environment in which both programs operate. It is important to note here that Quebec has its own independent blood operator, Héma-Québec. From a legislative and regulatory standpoint (0.1), Health Canada, as the federal regulator for the blood system, is responsible for administering the Food and Drugs Act, which includes the regulations relevant to the blood supply system. The federal Minister of Health is also responsible for maintaining an effective system for the surveillance of blood-borne pathogens.^{Footnote 118} Meanwhile, Canada's P/T Health Ministers are responsible and accountable for the national blood supply program, including ensuring the overall integrity of the blood supply system. CBS is the blood system operator and therefore works across all these jurisdictions (except Quebec) in compliance with various F/P/T regulations, which involves multiple stakeholders in each jurisdiction (0.2). Given the tainted blood tragedy that led to the submission of the Krever's commission report in 1997, the Canadian blood system is under high public scrutiny and stakeholder expectations around its safety and effectiveness remain high (0.3). Finally, CBS and Héma-Québec, as the two Canadian blood operators, collaborate on many issues (0.4).

The inputs and resources (boxes 1.1 to 1.10) have been described in previous sections. Briefly, Health Canada provides oversight and management of the Grant and Contribution Programs (1.1) associated with the funding it provides for the OTDT and R&D Programs (1.2). Note that parallel P/T management and financial (1.3 and 1.4) inputs are also included given their contribution to both these Programs. Boxes 1.5 to 1.10 (except for 1.9) relate to the financial resources, human resources, corporate support/oversight within CBS that provides for these two programs. In addition, CBS develops and maintains relationships, collaborations and partnerships with other organizations and stakeholders across Canada, who contribute to the achievement of its mandate (1.9).

The CBS OTDT Program consists of four major initiatives and their corresponding activities. Three of them are covered by the Health Canada contribution: strategic plan development (2.1), system performance improvement (2.2), and supporting leading practices and public education (2.3). The fourth initiative - the development and implementation of three national registries - is not presented in the logic model because it is funded by P/Ts (except for Quebec) and is therefore not covered in this evaluation. Under the strategic plan development, CBS has been mandated to design a plan for a national, coordinated OTDT system (2.1.1). The system performance improvements initiative involves the development of new performance metrics (2.2.1) and collaboration with Accreditation Canada on the refinement and development of accreditation standards (2.2.2). Several types of activities also support leading practices and public education initiatives, including:

• knowledge transfer (KT) and collaborative and expert sessions to develop consensus and promote OTDT leading practices and guidelines (2.3.1);
• prepared initiatives and materials (e.g., presentations, documents, training, Organ & Tissue Donation website, other communication tools) to help educate professionals and the public and increase their awareness on OTDT practices (2.3.2).
• scans and studies on international practices (2.3.3) to inform leading practice development.

These activities lead to four categories of outputs. Through the consultations for the national OTDT system design and the range of activities related to leading practices, awareness and education, the OTDT program produces knowledge tools and products (3.1) that are shared with stakeholders through various dissemination mechanisms (3.3). The collaboration with Accreditation Canada and with other stakeholders around performance measurement results, accreditation standards and system performance metrics and their definitions (3.2). All these activities, in particular those around national OTDT system design, also help identify barriers and/or opportunities and strategies on how to address them (3.4).

It is expected that the information-sharing process and mechanisms through which CBS shares new knowledge products and tools will lead to an immediate outcome of improved awareness and understanding of the OTDT Program and its priorities by OTDT stakeholders and communities (4.1). Work on national system design and leading practices is expected to contribute to an increase in the integration and coordination of OTDT-related activities across Canada (4.2). This partially depends on the implementation of recommendations in the Call to Action (i.e., the proposed plan for a national system) by the P/Ts. Indeed, P/Ts (and not CBS or Health Canada) have the direct responsibility for health system delivery in their jurisdictions. Meanwhile, the improved performance metrics and accreditation standards provide the OTDT communities with new/enhanced ways to strengthen and measure the performance of the Canadian OTDT system (4.3).

Improved awareness and improved coordination are expected to contribute to increased adoption of knowledge, policies and health care practices by practitioners and policy makers across Canada (intermediate outcome 5.1). The existence of new/enhanced standardized performance metrics and accreditation standards is expected to result in their adoption and increased use (5.2). Both of these intermediate outcomes are expected to contribute in the longer term to improved access, fairness and transparency for patients in need of organ or tissue transplantation (6.1), which will further increase public confidence in the Canadian OTDT system (6.2). The ultimate outcome of the OTDT Program is improved health outcomes for patients. Note that this is also one of Health Canada's strategic outcomes.

The activities of the R&D Program can be broadly divided into three groups: 1) Research (2.4), 2) Development (2.5) and 3) all other activities that support CBS operations including Intellectual Property program, Quality Management, and the newly launched Research & Education function (2.6). Note that, in practice, these initiatives and activities tend to cross/blend to some extent across these boundaries. For example, some CBS scientists are tasked to work on both research and development questions, with each type of activity informing the other. In addition, new products or processes developed might involve intellectual-property activities (e.g., filing of patents) and their dissemination/commercialization could be supported through the Research & Education function.

These activities produce several types of knowledge dissemination products (3.5), most notably peer-reviewed publications, reports, guidelines, educational materials, and national and international conference presentations. Development activities in particular also result in new tools, technologies, blood processing/storing/transporting processes and product lines (3.6). The Program is delivered through a hub model, by several groups in facilities across Canada, and CBS has developed partnerships with several other institutions (e.g., universities, research institutes, CIHR). As such, collaborations and partnerships are also considered to be a principal output of the R&D Program (3.7).

The outputs produced by the R&D Program are expected to improve the awareness and understanding of blood system-related knowledge products and tools (4.5). Administration of research grants and awards, as well as a focus on collaboration and creating partnerships contribute to increased collaboration, capacity in blood science and medicine, and enhanced learning opportunities for the blood community (4.6). Both 4.5 and 4.6 will also stimulate increased application of evidence-based information and products in the blood operations and health care system (4.7), as well as increased adoption of new technologies, products, clinical procedures and processes (4.8).

The intermediate outcome of the R&D Program builds on all four immediate outcomes, which are expected to result in changes in policy and practice through a broader adoption of knowledge and/or innovations (5.3). In the long-term, the R&D Program is expected to support the safety, quality and supply of blood and blood products (6.3), which ultimately creates a safe, effective and efficient blood system that is responsive to the current and future needs of Canadians.

The activities of Health Canada relevant to both programs are described in Section 0. Program management and oversight activities are depicted in the logic model as

priority-setting, progress monitoring and financial accountability

(2.4). The Department further engages in some

knowledge brokering

between CBS and intended/potential users within the Health Portfolio (2.4.1) and seeks to

build or strengthen relationships

through outreach with key partners and stakeholders within and outside the government related to these programs (2.4.2). The risk assessments and the five-year evaluation (3.5) are some of the main Health Canada outputs that feed information into the management of both programs. Health Canada's knowledge brokering and outreach activities lead to new or enhanced

relationships and partnerships

, as well

as events or products that provide and/or enhance knowledge transfer opportunities

(3.6).