Evaluation of the Blood Safety Contribution Program at the Public Health Agency of Canada 2008–09 to 2013–14

1.0 Evaluation Purpose

The objective of the evaluation is to assess the relevance and performance of the Blood Safety Contribution Program (BSCP) over the period 2008-09 to 2013-14. The evaluation was  conducted to meet the requirements of the Financial Administration Act and to provide the Deputy Head and senior management with a comprehensive and reliable evidence base to support decisions regarding the program.

2.0 Program Description  

2.1 Program Context  

Blood safety was recognized as a major issue in Canada following the contamination of the Canadian blood supply with the Human Immunodeficiency Virus (HIV) and the Hepatitis C virus in the late 1970s through the 1980s. In the final report of the Commission of Inquiry into the Blood System in Canada (Krever Inquiry) issued in November 1997, Justice Krever emphasized the importance of blood surveillance for the safety of blood in Canada. The federal government's response included a series of initiatives to support and strengthen the safety of Canada's blood system, including the BSCP.

2.2 Program Profile  

The BSCP program is intended to support the development and/or enhancement of provincial and territorial systems to monitor adverse events associated with the transfusion of blood, blood products and cell, organ and tissue transplantation. BSCP funding can also be used for targeted research to identify and assess risks associated with the use of blood, blood products, cells, organs and tissues. The recipients of BSCP funding include provincial and territorial governments, transfusion and/or transplantation centres and agencies and/or groups designated by provincial and territorial Ministries of Health to undertake surveillance for blood/tissue/organ associated adverse events, and Canadian not-for profit organizations that support transfusion adverse event surveillance activities in the provinces and territories.

The beneficiaries of the program include funding recipients who benefit from improved  transfusion and transplantation knowledge and increased capacity to collect and analyze data. In the longer term, public health practitioners are expected to benefit from improved knowledge and faster diagnosis of blood, blood product, and cell, tissue and organ transplant-related infectious diseases. Federal and provincial health authorities will also benefit from improved availability of evidence to inform health policy development, public health planning and program development and decisions on actions to respond to outbreaks or other urgent public health events. Ultimately, Canadians are expected to benefit from higher quality blood, blood products, and cell, tissue and organ transplant products as well as improved practices leading to fewer adverse events and improved patient safety.

The program includes the following components:

• Transfusion Transmitted Injuries Surveillance System (TTISS): TTISS is a national voluntary surveillance system for capturing moderate and serious adverse events related to the transfusion of both blood/blood components and plasma derivatives from data provided by hospitals, provinces/territories, and blood manufacturers. It tracks twenty-four diseases and infections. Currently TTISS includes all provinces and territories with hospital participation, representing 83% of total red blood cell transfusion activity in Canada.  Hospitals report adverse transfusion events (ATEs) to provincial and territorial (P/T) surveillance offices using a standard format. P/T surveillance offices subsequently submit the data to the Public Health Agency on a quarterly basis. 
• Transfusion Errors Surveillance System (TESS): TESS is a non-punitive, anonymous, web-based error reporting systems to capture actual events (that may or may not have resulted in adverse consequences) as well as “near-miss” events (where an error is caught before a product is administered to a patient) in the transfusion chain for blood, blood components and plasma derivatives. TESS is a pilot project involving 15 hospitals in four provinces. 
• Cells, Tissues and Organs Surveillance System (CTOSS): CTOSS is a pilot project involving five transplantation sites within Canada initially focussing on the surveillance of recipients of tissue allografts^{Footnote 1} and tissue products. Once fully developed, the system will collect data on moderate and severe adverse events, eventually expanding to include serious errors and near misses, for the purpose of monitoring trends in known and emerging risks and reducing the transmission of infectious diseases due to transplantation.
• Data linkages and communications: Data linkages and communications with provincial Public Health Information Networks are intended to allow the identification of delayed adverse events detected outside the hospital reporting system and thus not captured by TTISS. The focus of these activities is at the provincial/territorial-level. Provinces and territories are encouraged to build data linkages with other health system-related data bases in an effort to further the understanding of transfusion and transplantation safety.  
• Surveillance of high risk populations and emerging pathogens:  Surveillance of high risk populations (individuals receiving large numbers of transfusions) has decreased in importance with advances in screening of blood donors and transmissible disease testing and as the surveillance systems have matured. The main at-risk population is hemophiliacs. The Association of Hemophilia Clinic Directors of Canada (AHCDC) was provided with contribution funding for the Blood Borne Pathogens Surveillance Project. The AHCDC used its funding to obtain blood samples from individuals with a diagnosed hereditary bleeding disorder, purchase equipment and provide blood sample storage. Funding to AHCDC lasted 10 years and sunsetted in March 2011. 

• The BSCP relies on surveillance data submitted by the provinces and territories and collected using funding provided under contribution agreements.^{Footnote 2} BSCP funding is provided for projects related to the design, development and operation of P/T adverse event reporting systems, the development of data linkages, the dissemination of public health information and surveillance of high-risk populations and emerging pathogens. Other activities eligible for funding include awareness building, regular monitoring of transfusion and transplantation recipients at hospitals, collection of information using standardized reporting forms, data analysis and reporting to the Public Health Agency, and may also include the funding of a regional/provincial coordinator to perform these activities as well as targeted research to identify and define risks associated with the use of blood products, tissues and organs. 

  Under each contribution agreement, data is submitted to the Public Health Agency on a quarterly basis after an initial 6-month period.  In addition to submitting data, funding recipients also provide progress reports on activities supported by Public Health Agency funding.^{Footnote 3}  Upon receipt of P/T submissions, program staff assess and validate the data and reconcile it with data in other reporting systems, notably mandatory reporting to the Canadian Blood Services^{Footnote 4}, in order to ensure completeness. Once the data have been validated, program staff analyze the data and discuss the results with the provinces at meetings, conferences or via webinars. The final step in the process is to produce a formal report which is sent to the P/Ts for review and comment before it is released to the public.

2.3 Program Logic Model and Narrative  

The long-term expected outcome for the BSCP is improved safety for transfusion and transplantation patients. The logic model, shown in Appendix 1, shows how activities, outputs and immediate and intermediate outcomes are expected to lead to the achievement of this long term outcome. This logic model was collaboratively developed by the Evaluation Directorate and the Program for the evaluation by tailoring the Public Health Agency's standardized Surveillance Program Logic Model ^{Endnote i} to the BSCP. The activities, outputs, and outcomes (immediate, intermediate and long-term) are described below.

The core BSCP activities include:

• soliciting proposals and preparing contribution agreements to support P/T transfusion and transplantation adverse event surveillance activities;
• assessing, validating and reconciling data received from the P/Ts;
• analysing and interpreting the data; and
• disseminating surveillance information through meetings, conferences, webinars and formal reports.

As a result of these activities, the BSCP produces surveillance products including:

• surveillance data;
• surveillance reports;
• guidance documents; and
• presentations for meetings and conferences.

The immediate outcome of the program is timely and useful surveillance evidence.  This information is intended for health system stakeholders responsible for monitoring, preventing and controlling transfusion and transplantation-related adverse events.  The national-level data is used by provinces and territories to benchmark the rates of key transfusion and transplantation adverse events. The Public Health Agency uses the data to monitor and report on transfusion and transplantation adverse event trends.

In addition to helping stakeholders reduce the likelihood of  adverse events, these stakeholders are also expected to use surveillance evidence to inform the public about health risks and threats as well as for planning and programming and in developing guidelines, policy, regulation and legislation (intermediate outcome).

Finally, evidence-based planning, programming, guideline development, policy, regulation and legislation and a well-informed practitioner community and public is expected to improve the safety of those patients undergoing transfusions and transplantations (long-term outcome).

2.4 Program Alignment and Resources

The Program is part of the Public Health Program Alignment Architecture (PAA): 1.2.1.2 Infectious and Communicable Diseases. The program's financial data for the years 2008-09 through 2013-14 are presented below.

 

Over the period of 2008-2013, the Initiative spent a total of $11.82 million in grants and contribution funding, $1.19 million in operations and maintenance (O&M) funding, and $2.43 million in salaries.^{Footnote 5}

3.0 Evaluation Description

3.1 Evaluation Scope, Approach and Design

The scope of the evaluation covers the period 2008-09 to 2013-14 and included the three surveillance systems (TTISS, TESS and CTOSS) managed by the Program, as well as activities  for the data linkages and surveillance of high risk populations components. 

The evaluation issues were aligned with the Treasury Board of Canada's Policy on Evaluation (2009) and considered the five core issues under the two themes of relevance and performance. Corresponding to each of the core issues, specific questions were developed based on program considerations and these guided the evaluation process (see Appendix 2).

An outcome-based evaluation approach was used for the conduct of the evaluation to assess the progress made towards the achievement of the expected outcomes, whether there were any unintended consequences and what lessons were learned.

A non-experimental design was used based on an evaluation framework document, which detailed the evaluation strategy for this program and provided consistency in the collection of data to support the evaluation. Data for the evaluation was collected using various methods:

• literature review;
• document review;
• key informant interviews; and
• comparison of surveillance system information available from other countries.

More specific detail on the data collection and analysis methods are detailed in Appendix 2. Data were analyzed by triangulating information gathered from the different methods listed above. The use of multiple lines of evidence and triangulation were intended to increase the reliability and credibility of the evaluation findings and conclusions.

3.2 Limitations and Mitigation Strategies

Most evaluations face constraints that may have implications for the validity and reliability of evaluation findings and conclusions. The following table outlines the data limitations encountered during the implementation of the selected methods for this evaluation. Also noted are the mitigation strategies put in place to ensure that the evaluation findings can be used with confidence to guide program planning and decision making.

The evaluation did not encounter any significant methodological issues. The Program provided administrative data in a timely manner and cooperated fully with the evaluation team. All stakeholders, except one (nongovernmental organization), agreed to participate in interviews. 

4.0 Findings  

4.1 Relevance: Issue #1 – Continued Need for the Program  

FINDING #1: The number of Canadians undergoing transfusions and transplantations is growing and the relative risks from the procedures are shifting from infectious to non-infectious.  As a result, there is a need for surveillance in Canada of transfusion and transplantation adverse events, but the needs appear to go beyond infectious risk and include non-infectious risks (e.g., injuries and errors).

Currently, over 800 hospitals transfuse over 1 million units^{Footnote 6} of blood every year, which includes various products such as red cells, plasma and platelets.^{Endnote ii} The number of blood units transfused in Canada has increased from 714,000 in 2002 to 1,184,000 in 2011 (see Table 3).  While many steps have been taken to minimize the risk from infections and other injuries, it is never zero because these are biological products taken from living donors.^{Endnote iii}  In addition, there are other risks that are inherent in the transfusions and transplantation procedures (e.g., errors, anaphylaxis, allergic reactions and injuries). Collecting information on these adverse events is necessary to understand, mitigate and manage the risk inherent in the transfusion and transplantation procedure. 

Data from the Transfusion Transmitted Injuries and Surveillance System (TTISS) suggest that rates of adverse reactions are  less than 1% of all blood transfusions that took place between 2002 – 2010.  While the numbers cited above appear low, in reality they are likely larger due to under-reporting. Under-reporting is a common issue for surveillance systems.^{Endnote iv,Endnote v}

Approximately 2,000 organs are transplanted and over 90,000 tissue allografts are distributed for transplantation and implantation.^{Endnote vi} For cells, tissues, and organs (CTOs), the percentage of reported adverse events is 0.1%. Program documents state that blood transfusion and transplantation will always pose a risk because of new and emerging blood pathogens not detectable with current methods, rapid changes in blood technology and new blood products that may harm Canadians.  As a result, there is a continued need for transfusion and transplantation surveillance and monitoring.  However, the surveillance needs are not static.

Changing Environment

Originally, the need for surveillance of transfusion-related adverse events was prompted by the blood tragedy in the 1970s – 1980s.  It is estimated that approximately one thousand individuals who received blood transfusions between the late-1970s and 1980s were infected with HIV, and another 30,000 were infected with hepatitis C.^{Endnote vii} As a result, the Canadian blood safety program, initially focused on screening for pathogens and preventing infections, was established. In addition, it was recommended to implement a transfusion surveillance system (Krever Report).

Since the blood tragedy and subsequent Krever Report, the haemovigilance landscape has expanded from infectious disease surveillance, to include monitoring for transfusion and transplantation injuries and errors.  In fact, these non-infectious risks now outweigh infectious risks.^{Footnote viii,Endnote ix} This is a result of blood and transplantation donor testing, screening and deferral policies.^{Endnote x} Many countries including Canada are developing or have developed surveillance systems to monitor injuries like transfusion-associated circulatory overload (TACO) and transfusion related acute lung injury (TRALI). While transfusion and transplantation errors have always posed a risk, they received less attention.^{Endnote xi} Errors are defined as any deviation from established policies and standard operating procedures.^{Endnote xii} The most common of these are inappropriate orders for blood and lack of accurate blood sample labelling and handling.   The shift from infectious to non-infectious risks means that haemvigilance systems must adjust appropriately.  Despite these observations, newly identified pathogens continue to emerge and challenge the sensitivity and specificity of screening tests (e.g., Chagas disease, Creutzfeldt–Jakob disease, new mutant strains of HIV that are undetectable by current means).  Rapid changes in blood banking science/technology, and development of new biologics such as ‘artificial' blood and recombinant blood products also pose health risks. As a result, there is a continued need to monitor emerging pathogens and the errors and injuries that occur as a result of transfusions and transplantations. Given the current context, there is a continued need for the blood surveillance program to monitor a wider range of risks than was identified in the Krever Report.

4.2 Relevance: Issue #2 – Alignment with Government Priorities  

FINDING #2: There are a number of blood and patient-related priorities that the Government is supporting, although blood safety surveillance is no longer mentioned specifically.

While transfusion and transplantation surveillance is not explicitly mentioned as a priority for the Government of Canada, it is encompassed in the broader priority of patient safety.  In the 2013 Speech from the Throne, the Government committed to patient safety through regulatory modernization regarding reporting of adverse drug reactions. This particular announcement was focused on product safety and oversight rather than on the surveillance of transfusion and transplantation-related surveillance, trend monitoring and transfusion and transplantation-related adverse event reporting. However, blood products are increasingly processed into different therapeutic products that are ‘drug like', which would fit into this overall area. Moreover, in 2013, the Minister of Health announced $38.1M over 5 years for Canadian Patient Safety Institute and stated that “we have a role to play when it comes to patient safety”.^{Endnote xiii}

Health Canada continues to fund a number of different blood-related initiatives including Canadian Blood Services for research and development ($25M to CBS, 2008-2009 to 2012-2013); developing a pan-Canadian Organ and Tissue Donation and Transplantation System and creating a separate regulatory framework for blood. Health Canada signed a contribution agreement with CBS for $17.9M (2008-2009 to 2012-2013) to develop a national organ and tissue donation and transplantation system. Moreover, the Health Care Policy Contribution Program of Health Canada funds the Canadian Medication Incident Reporting and Prevention System (CMIRPS) $1.6M annually to conduct activities designed to reduce harm caused by preventable adverse drug events.

These Government of Canada funding decisions demonstrate its ongoing interest in health product and patient safety.

4.3 Relevance: Issue #3 – Alignment with Federal Roles and Responsibilities  

FINDING #3: BSCP activities align with the Public Health Agency's legislative and program authorities, as well as the Public Health Agency's Surveillance Strategic Plan. BSCP is solely responsible for the coordination, analysis and timely dissemination of national surveillance data on adverse transfusion and transplantation events. Other partners in the transfusion and transplantation system monitor blood and blood products and CTOs for safety, quality and utilization and do not engage in the same activities as the Public Health Agency.

The Public Health Agency of Canada Act established the Public Health Agency of Canada for the purpose of assisting the Minister of Health in exercising or performing the Minister's powers, duties and functions in relation to public health. The powers, duties and functions are set out in the Department of Health Act, and those relevant to this program include: the protection of the people of Canada against risks to health and the spreading of diseases; investigation and research into public health, including the monitoring of diseases; the collection, analysis, interpretation, publication and distribution of information relating to public health; and cooperation with provincial authorities with a view to the coordination of efforts made or proposed for preserving and improving public health. As a result, there is a role for the Public Health Agency to engage in transfusion and transplantation surveillance activities.

Beyond the Act, the Public Health Agency of Canada's Surveillance Strategic Plan provides the Public Health Agency with a further strategic vision for conducting surveillance. The vision outlined in the Surveillance Strategic Plan is to become a credible source of current public health intelligence, derived from surveillance and other sources of public health information, a key influencer of public health policy, decision making and action at the federal-level and in other jurisdictions, that aims to improve the health of Canadians.^{Endnote xiv} Further, this document establishes the Public Health Agency as the national point of focus in collecting surveillance data and playing a coordination role in gathering information from provinces, territories, other government departments and non-government organization data so as to enable national-level data analysis.

There are also program authorities for the specific role of transfusion and transplantation surveillance. Prior to the creation of the Public Health Agency, Health Canada first received program authorities in 1996 to conduct surveillance of pathogens transmitted through blood.  Program authorities describe the purpose of blood surveillance as detecting/collecting data, investigating/researching/reporting/analyzing trends and responding to blood-borne infectious agents/threats in a nationally standardized fashion. These efforts were intended to be collaborative with provinces and territories and other blood stakeholders. Subsequent authorities were received in 1998, including the authority to develop a surveillance system for transfusion-related errors.  This focussed on the collection of data on the root causes of errors related to transfusions in hospitals with the goal of developing strategies to decrease transfusion related risks. Authorities were last received in 2004-05, where the Public Health Agency was specifically authorized to develop a pilot surveillance system for pathogens that pose a risk to Canadians receiving cells, tissues and organ transplants.

Interviewed stakeholders agree that there continues to be a role for the Public Health Agency in blood safety surveillance. Key stakeholder informants stated that there is a leadership role for the Government of Canada, and the Public Health Agency of Canada in particular, to mobilize, facilitate and coordinate a Canada-wide blood transfusion surveillance system network. Moreover, many comparable countries have blood safety surveillance systems (e.g., United Kingdom, France, Australia, New Zealand). At the international level, Canada has a role in participating in international haemovigilance efforts and to provide blood surveillance data (adverse event) to report and respond to infectious diseases and other adverse events that may be of potential international concern. Internationally, BSCP works with international organizations like the World Health Organization to standardize coding, monitoring and reporting and also alerting its global partners to new and emerging pathogens.

There are other stakeholders who have  roles and responsibilities to uphold in the transfusion and transplantation system in terms of regulatory oversight, product safety and utilization.  Figure 1 illustrates the key players in the Canadian transfusion and transplantation system. Their roles and responsibilities will be discussed in the next section.

Figure 1: Key players in Transfusion and Transplantation System

Other Transfusion and Transplantation Stakeholders

Health Canada is responsible for post-market safety of blood and blood products. In particular, Health Canada collects data on adverse events from plasma-based blood products, cells, tissues and organs, recombinant and fractionated blood products, and fresh blood components. These products are considered therapeutic products and Health Canada is responsible for monitoring their safety, effectiveness, and quality. In tracking adverse events, Health Canada uses active surveillance to mitigate risk from blood and blood products and CTOs.  The Department engages in signal detection and where a product is causing an adverse event the Department takes corrective action (e.g., advisories, recalls, warnings).

The two blood operators in Canada, CBS and HQ, also engage in surveillance activities (e.g., monitoring and tracking blood safety, quality, and utilization. Reactions relating to the quality or deficiency of the blood or blood product are monitored and reported directly from hospitals to CBS and HQ, who then report to Health Canada). CBS and HQ report any deaths and serious adverse events due to blood or blood products to Health Canada.  However, the blood operators do not have to report any reactions reported to them by hospitals that fall outside of this requirement (e.g., increases of acute hemolytic reactions with intravenous immunoglobulin, IVIG, not causing death but causing drop in hemoglobin and requiring additional transfusions).

At the local level, hospitals are required to collect information on transfusion adverse events if they are to keep their Canadian Standards Association (CSA) accreditation. They must be able to demonstrate that they record all transfusion adverse events and errors in recipient records to the blood product manufacturer or blood supplier (CBS/HQ).  However, these events and errors also encompass non-transfusion events (e.g., falls, medication errors). Reporting of serious adverse events and deaths to Health Canada is mandatory.

Therefore, other stakeholders in the transfusion and transplantation system monitor transfusion and transplantation errors, i.e. Health Canada, Canadian Blood Services (CBS), Hema-Québéc (HQ).  These stakeholders focus on the  safety, quality and utilization of blood, blood products, CTOs and utilization. The Public Health Agency, however, is the only stakeholder who collects and reports on national-level transfusion and transplantation adverse events as they relate to patient safety trends.

Overall, the BSCP is aligned with the roles and responsibilities of the Public Health Agency in terms of its program authorities and corporate documents, such as the Surveillance Strategic Plan. 

4.4 Performance: Issue #4 – Achievement of Expected Outcomes (Effectiveness)

4.4.1 To what extent have the immediate outcomes been achieved?

Blood system stakeholders need timely access to national-level transfusion and transplantation data to understand adverse event trends. National-level surveillance data allows provinces and territories to benchmark their own adverse event rates to the national rates. Benchmarking assists in determining if the provincial/territorial adverse events fall within or outside the statistically normal range for Canada. 

Immediate outcome: Availability of timely and useful surveillance evidence for health system stakeholders.

FINDING #4: Timely release of adverse blood transfusion events data is needed for benchmarking, trend analysis and as a tool to improve transfusion practices. Despite timely data submission by provinces and territories, all lines of evidence indicated that the Public Health Agency's surveillance data reporting is not timely.

National surveillance systems like those found in BSCP contribute to the understanding of transfusion and transplantation safety and practices. Monitoring and assessing transfusion and transplantation for infections, injuries, reactions and errors are needed to help anticipate, prevent and respond to transfusion-related safety risks. As a result, there is a need for timely transfusion and transplantation surveillance data by the provinces and territories to assist hospitals and health practitioners manage adverse events.  A review of other countries demonstrates that an annual cycle with a time lag of one year is the norm for reporting national-level adverse transfusion events. Key informants also stated this was an appropriate reporting cycle. Timely and accessible data is also needed because it provides transparency and information on documented risks, and ensures that any issues for improvement are heard by transfusion practitioners.^{Endnote xv}  

Transfusion and transplantation surveillance is shared between the federal, provincial and territorial governments and local health authorities.  The provinces and territories have the primary responsibility of detecting, monitoring and reporting transfusion and transplantation adverse health events.  The Agency relies on the collaboration of stakeholders to provide high quality, timely data.  In turn, stakeholders rely on the Agency to provide timely release of statistical and analytical reports which contain national-level transfusion-related data.  The Agency's goal is to be a consistent source of credible and current public health intelligence.^{Endnote xvi}

All lines of evidence support the finding that the Public Health Agency is not releasing national-level transfusion adverse event data in a timely manner: 

Internal document review: Appendix 3 provides the publication dates of the Public Health Agency's surveillance data. In 2008, the Public Health Agency disseminated 2000-2005 TTISS data with a lag-time of three years. The Public Health Agency is currently in the process of releasing 2006-2012 TTISS data, although this report will contain data that will be two to eight years old when available publically. The most recent TESS data covers the years 2005-2007 and was released in 2012. This is a lag time of five years, although currently seven years out of date.   

TTISS stakeholder survey: The majority of respondents (72%) participating in the  TTISS stakeholder satisfaction survey (2011) did not feel that national-level information was available in a timely fashion. Many commented on the lack of timely information and the impact on activities. Commentary from respondents included the need for timely information to determine if transfusion practices were having an impact on patient safety, with timely interpreted as six months to a year after provincial and territorial submission. 

Key informant interviews conducted for this evaluation with provincial and territorial representatives were unanimous in commenting on the lack of timely reporting by the Public Health Agency. Essentially, all stakeholders stated they are waiting for the national-level data reports. Their commentary repeated concerns raised in the 2011 survey mentioned above.

International Comparison:  Appendix 3 outlines the availability of national-level transfusion and transplantation surveillance data in other countries (Australia, France, New Zealand, United Kingdom and the United States^{Footnote 8}). These countries released their blood safety surveillance reports annually.  The data in the reports tend to be for the previous year which represents a one year lag-time. Annual reporting with one year data and reporting lag-times appear to be the norm in the broader transfusion surveillance community.

While not publically posting surveillance reports, there was evidence that the Public Health Agency has presented surveillance data at annual meetings of the National Working Group as well as at various conferences. Some provincial key informants stated that, while the information in a presentation format is interesting, it was not as useful as a published report. Stakeholders stressed that the Public Health Agency report (national-level data) has more legitimacy than provincial and territorial reports in the transfusion community and can be used to develop, inform and change transfusion practices. Provincial stakeholders also emphasized that tracking and reporting these injuries can be used to convince transfusion practitioners of the other potential dangers of transfusion, in addition to those stemming from pathogens.

Based on the evidence presented above, the national-level transfusion and transplantation reporting is not timely. This does not mean that some provinces were unable to use their own provincial data or attempt to consult with other provinces to understand transfusion and transplantation safety issues. The evaluation found evidence of larger provinces stratifying their data by regional health authorities or regions and benchmarking it with the provincial-level adverse event rates.  Smaller provinces could not engage in this type of analysis because they do not have the populations necessary to generate meaningful analysis. Some smaller provinces resorted to contacting other provinces that were regionally similar to discuss trends in the absence of larger intra-provincial or national-level data. This however, was not a common practice.

4.4.2 To what extent have the intermediate outcomes been achieved?  

Transfusion and transplantation surveillance data, rolled up at the national-level, should be used by provincial and territorial health authorities to assist in changing transfusion and transplantation practices. Potentially, anomalous transfusion/transplantation rates identified in the report could point to the need to better manage a specific event which could lead to: surveillance enhancement, awareness campaigns, development of guidelines, new training for transfusion practitioners and new transfusion/transplantation practices. Ultimately, the patients receiving a transfusion or transplantation benefit from surveillance information.

Intermediate outcome: Health system stakeholders utilize surveillance evidence to inform the public on health risks and threats, to inform planning, programming, and policy which contributes to regulatory and legislative development.

FINDING #5: There was little direct evidence to demonstrate the use and uptake of national-level surveillance findings because the program has not released a surveillance report since 2006 for TTISS, and TESS and CTOSS have been in a pilot phase for 8 years. 

It was not possible to conduct a meaningful assessment of the utilization of the Public Health Agency's blood safety surveillance, considering the limited timely information that was available. TESS and CTOSS have been in the pilot phase for 8 years and thus have limited provincial and territorial participation to make the data useful.  However, anecdotal TTISS-related information was available on the kinds of surveillance-related uptake activities that did take place by various partners. Utilization and uptake of surveillance information by blood system stakeholders varies depending on where they reside in the blood system. This section describes and assesses the utilization of transfusion surveillance data at different levels:

• hospital;
• provincial/ territorial;
• national (Health Canada [the regulator]);
• CBS/ Hema-Québéc (the blood operators); and
• blood researchers. 

Hospital-level

Utilization of transfusion surveillance data at this level took the form of knowledge uptake by health practitioners and use of transfusion-related guidelines. In order to report adverse transfusion events, transfusion practitioners had to be educated on what to report. Many key informants pointed out that the training on transfusion medicine in nursing and medical schools was limited in time and scope. As a result, they felt that most nurses and doctors were not sufficiently aware of the many types of adverse events that could result from a transfusion.  An unintended impact of setting up and maintaining TTISS and TESS was the increased awareness and utilization of knowledge by transfusion practitioners to recognize and report the signs and symptoms and manage adverse events. Some hospitals created apps and lanyard cards, that held information on identifying signs and symptoms and how to manage the adverse events. While the data did not impact transfusion safety at this level, the training necessary to feed into the surveillance system increased transfusion-related adverse event awareness and knowledge utilization by transfusion practitioners.

Provinces/territories^{Footnote 9}

Provincial key informants stated that they have drawn from and used their provincial and territorial data but they have also pointed out that they have needed the national aggregate data to determine whether they fall within or outside of national trends. Some provinces, like British Columbia and Quebec, have proceeded with their own reporting outside of TTISS rather than wait for the Public Health Agency surveillance reports.  Ontario has begun working on a proof of concept surveillance system called Transfusion Registry for Utilization Surveillance and Tracking (TRUST).

Health Canada (the regulator)

The Public Health Agency, through the TTISS surveillance system, shares information with Health Canada to reconcile adverse reporting.  In 2012, the Public Health Agency provided aggregate data to Health Canada concerning TACO following intravenous immunoglobulin (IVIG).  The Public Health Agency and Health also engage in reconciling adverse event data.  These efforts found that a number of serious reports were submitted to TTISS and not to Health Canada.  Through these data reconciliation activities, Health Canada was able to have a more complete set of data to monitor product safety. The evaluation found that BSCP surveillance data were not used to inform the development of Health Canada policy, regulations or legislation for the time period under study. 

CBS/Hema-Québéc (the blood operators)

Both blood operators have epidemiology and surveillance departments. Key informants from these organizations stated that they share surveillance data with each other but do not compile national-level data. The blood operators report that they too would like to have national-level surveillance data in order to identify adverse event trends.  However, they commented that current information is not available for them to use. 

Blood researchers/academics

Two blood researchers were interviewed for this evaluation and both stated that transfusion and transplantation surveillance was not important for their research. They have only used the available surveillance data for teaching purposes rather than for research. A review of the literature found a small number of published articles that used TTISS and TESS data for the period under study which could be due to the lack of timely reports.

4.4.3 To what extent has the longer term outcome been achieved?

Longer term outcome: Improved Patient Safety

FINDING #6: The evaluation was not able to determine if patient safety had improved as a result of the BSCP's activities. 

Theoretically, transfusion surveillance can be used to identify safety issues and to better manage transfusion practices.  Empirically, the evaluation was unable to demonstrate how national-level data contributed to improved patient safety for the period under observation.  Lack of timely reporting is the most likely explanation for this.

There are some anecdotal examples of improved patient safety, as provided by key informants for this evaluation.  Quebec provided examples of how their provincial-level surveillance data help them identify risks from bacterial contamination of blood platelets and take action to reduce the risk by 10 times.  Incidences of TRALI were also identified and action was taken to reduce TRALI by 50%.  TACO was addressed because of analysis coming from provincial-level surveillance data.  These kinds of patient safety outcomes were not seen elsewhere in Canada.  Key informants from smaller provinces and territories stated that they needed the national-level data to better understand transfusion injury trends and take the necessary actions to mitigate or manage them.  Under the current circumstances not all provinces and territories can benefit from their surveillance data in the same way.

Ditomasso et al(2012) point out that the TTISS mainly captures and reports information on severe reactions but the breadth and depth of the data being collected is not sufficient to address clinical outcomes.  Specifically, they note that  “More comprehensive data on the frequency of signs and symptoms associated with different types [types of reactions] could facilitate the development of electronic algorithms that could assist in developing a differential diagnosis and identifying different types of reactions”.^{Endnote xvii}

To pursue improved patient safety in this area, Ontario is piloting the TRUST which consists of three of the four McMaster hospitals, contains 50 or more demographic and clinical data variables such as diagnoses, procedures, length of stay, ICU status, labs and discharge status.^{Endnote xviii}  The TRUST database also collects information on blood and blood product inventory and blood and blood product distribution. As a result, this type of surveillance system can be used by transfusion practitioners to understand how often certain signs and symptoms (e.g., fever, shortness of breath, purpura) are associated with a transfusion and what blood products are associated with particular signs and symptoms. TRUST demonstrates how surveillance data can be linked to other databases, such as discharge and laboratory records to improve the safety of transfusion practices and patients.  The TRUST provides an example of the direction in which haemovigilance is going and the limitations of TTISS in impacting patient safety.

Currently, the success of BSCP lies not at a national-level but at the provincial-level. Success is unequally shared between the provinces and territories. Key informants stated that larger provinces have more robust datasets to work with than smaller provinces and therefore can use their data to improve health outcomes for patients undergoing transfusions. The TRUST project points to the possibility that patient safety may be better served by capturing a broader spectrum of reactions beyond those that are severe and linking those reactions to databases that capture patient records.

4.5 Performance: Issue #5 – Demonstration of Economy and Efficiency

FINDING #7: Some efforts were made by the Public Health Agency to operate the Blood Safety Contribution Program more efficiently and economically.  However, expenditures were made without timely outputs. In addition, the Public Health Agency underspent its 5 year budget in this area by $1.032M (6.6%).

The Treasury Board of Canada's Policy on Evaluation (2009) and guidance document, Assessing Program Resource Utilization When Evaluating Federal Programs (2013), defines the demonstration of economy and efficiency as an assessment of resource utilization in relation to the production of outputs and progress toward expected outcomes.  Assessing economy and efficiency of the BSCP was conducted through the comparison of planned and actual spending by the Program and also through the examination of measures that were taken to find program savings.

Functional Inefficiencies

The BSCP was unable to produce significant outputs which demonstrated that it is not functioning efficiently.  In 2008, the Public Health Agency released its TTISS report for 2004-2005 data and in 2012 released the 2005-2007 TESS report.  The 2006-2012 TTISS report is currently awaiting approval for release. Concerns raised about the lack of progress on key program outputs has been raised previously in both internal reports (the 2009 evaluation of the BSCP) as well as externally in the 2008 Parliamentary Report Canada's Blood Supply Ten Years After the Krever Commission^{Endnote xix}.  However,  the lack of timely reporting remains a problem.

Concerns about efficiency are also based on the extended pilot stages of both CTOSS and TESS. Some stakeholders questioned the efficiency of BSCP because TESS and CTOSS have limited participant sites which may diminish their value to produce national-level or representative data. Decisions on the program's products and future directions do not appear to have been taken as demonstrated by a lack of outputs over several years as well delayed decisions on the status of CTOSS and TESS.

Financial efficiencies

The Program underspent a total of $1.032M or 6.6% of $15.44M for this 5 year time period.  The program consistently underspent its planned budget, ranging from $10K to $650K annually (see Financial Table 4). Significant underspending in 2010-2011 occurred because Quebec was not able to sign its contribution agreement for >$500K^{Footnote 10}. 

One way that the Public Health Agency tried to operate the BSCP more efficiently was through the elimination of non-core activities like face-to-face meetings and replacing them with WebEx teleconference calls. For years, these meetings brought surveillance stakeholders from all regions of Canada to one location, which rotated between regions. The mechanism for working group meetings associated with the individual surveillance systems was not outlined in the original program authorities and the respective Terms of Reference. The importance of collaboration for the development, conduct and improvements to the three surveillance systems was identified in these documents but mainly in regard to surveillance information.

The following table outlines when the meeting formats were changed and the concomitant savings these changes represented. Overall, the financial cost savings that this represents is .5% for fiscal year 2013-2014.

Social Capital

The in-person National Working Group (NWG - TTISS) meetings were seen, by the majority of blood stakeholders interviewed for this evaluation, as a good platform for mobilizing those in the transfusion and transplantation system to formally and informally address blood and CTO safety and utilization issues. Key informants stated that the working group provided those in the transfusion and transplantation community with a network and access to resources and information that they believed would otherwise not have occurred.

This mobilization can be perceived as an increase of social capital for transfusion and transplantation stakeholders.  Interviews with key informants found that stakeholders from diverse backgrounds were brought together (i.e. regulators, blood operators, blood researchers and provincial and territorial health ministries) as a result of these face-to-face meetings. These exchanges took place formally in scheduled meetings and presentations as well as informally in hallway exchanges, between presentations and after the daily meeting sessions. Many stakeholders stated that WebEx teleconference calls do not provide them with the same quality of interaction to share and address transfusion and transplantation issues with their network.  As a result, bridging social capital has been reduced.

Observations on the Adequacy and Use of Performance Measurement Data  

The Public Health Agency has no service standards or performance measurement system in place for its BSCP.  The BSCP is not a complicated program that needs numerous performance measures to manage and report on its activities. Its primary activities are

1. data collection,
2. coordination, analysis and interpretation of data and
3. dissemination of data. 

The provinces and territories provide data to the Public Health Agency on a quarterly basis. The Public Health Agency reviews the data for completeness and validity.  Once this is done a brief analysis is prepared and the report is written.  What would assist the program in the timely delivery of surveillance reports is a set service standard for each primary activity and an established release cycle (e.g., annually) for the dissemination of reports and in a format that is useful and easy to produce. 

5.0 Conclusions

5.1 Relevance Conclusions

There is a continued need for transfusion and transplantation surveillance activities to monitor emerging blood pathogens, rapid changes in technology and new blood products. This is especially relevant for blood transfusion errors. Adverse events resulting from blood transfusions and transplantations continue to occur and the consequences, although rare, can have a serious impact on Canadians.

Since the original program authorities that were received after the Krever Commission in 1996, other organizations such as CBS/HQ and Health Canada have established a role in monitoring blood safety activities where corrective action has taken place based on product safety, quality and utilization. However, even within this current environment, there is a role for the Public Health Agency to collect and report on national-level transfusion and transplantation adverse events as they relate to patient safety trends. Currently, there are no other organizations conducting national surveillance like this in Canada.

Overall, the BSCP appears to align with the roles and responsibilities of the Public Health Agency in terms of its program authorities and corporate documents, such as the Surveillance Strategic Plan. Key informant stakeholders continue to look to the federal government to ensure that national-level surveillance data is available to all stakeholders who manage the safety of transfusion and transplantation practices. Stakeholders are concerned that the Public Health Agency is losing interest in haemovigilance and that the Public Health Agency has no clear vision for the current Blood Safety Contribution Program. 

5.2 Performance Conclusions

Key outputs have not been produced in a timely manner, such as the release of national-level surveillance reports and movement of TESS and CTOSS from pilots to fully operating surveillance systems as originally outlined in program authorities. The surveillance data that the provinces and territories provide to the Public Health Agency has value.  However, its value lies in its timely availability for provincial and territorial use.  Other countries are able to publish transfusion surveillance data annually and with a one year lag time. Demonstrations of use were also seen at the provincial level but these efforts were limited because national-level data were not available.

These findings point to the fact that the BSCP is not achieving the vision as articulated in the Public Health Agency's Surveillance Strategic Plan. The BSCP should be a credible source of current public health intelligence, derived from surveillance and other sources of public health information, a key influencer of public health policy, decision making and action at the federal-level and in other jurisdictions, that aims to improve the health of Canadians.

6.0 Recommendations

Recommendation 1

Develop a haemovigilance action plan which takes into consideration the current blood safety environment:

1. a risk assessment of the transfusion and transplantation surveillance environment;
2. a clear determination of stakeholder/user needs; and
3. the role of key players in monitoring transfusion and transplantation safety and where the Public Health Agency should concentrate their effort.

The environment has changed since the original program authorities, although there is a continued need for transfusion and transplantation surveillance activities. It is time for the Public Health Agency to review its haemovigilance activities in light of these current changes, including those organizations who now have a role in the blood safety system.  This will help the Public Health Agency assess where to focus efforts based on current risks in the environment. This action plan should also take into consideration corporate strategic direction for surveillance.

Recommendation 2

Develop and publicize service standards for the timing of the release of national-level transfusion and transplantation surveillance reporting.

To ensure that national-level data is available to stakeholders when needed, the Public Health Agency should develop standards for publication of surveillance information to stakeholders and the broader blood safety community.  This would enhance predictability and help ensure that information is available to influence transfusion practices at the local-, provincial- and national-levels. The program could also consider the reporting format that best meets stakeholder needs. The following are potential formats: short and full reports, shelf tables, pivot tables, or specific studies that are published in refereed journals.  The type of format chosen will likely influence the level of effort needed and therefore the amount of time necessary for the production and posting of the transfusion and transplantation results.

Recommendation 3

Review and clarify the goals and objectives of the National Working Group.

The TTISS National Working Group was set up to collaborate on the development, conduct and improvements to the three surveillance systems.  However, it appears as though the National Working Group also provided those in the transfusion and transplantation community with a network and access to resources and information that they believed would otherwise not have occurred. Lately, this ability to exchange knowledge on transfusion practices has been limited. To maximise knowledge exchange within the blood safety community, the goals and objectives of the working group should be reviewed and clarified.