ARCHIVED - Summative Evaluation of the Blood Safety Contribution Program - Final Report

III. Summative Evaluation

Methods

Methods for the evaluation included a literature review, review of program documents, surveys of key program stakeholders, key informant interviews, and an expert panel. Detailed technical reports associated with each data source were produced as part of the evaluation process.

Literature review

The literature review provided information to support the design of the BSCP Evaluation Framework, enabled understanding of blood surveillance systems in different countries, and contributed information to answering several of the evaluation questions.

The search for literature included a review of Canadian and international resources. It is our understanding that both France and Québec have been leaders in haemovigilance, so we paid close attention to reports that came out of both of these jurisdictions. Resources were selected for inclusion based on relevance to the project, date of publication, and credibility. A full list of resources used in the literature review is provided at Appendix A.

Document review

The document review enabled an understanding of the BSCP, supported the design of the BSCP evaluation framework, and contributed information to answering most of the evaluation questions.

The document review included program documents provided by BSCP staff, including (but not limited to) program planning documents, the program’s integrated RMAF/RBAF, program reports, presentations and abstracts, a sample of contribution recipient proposals and reports, terms of reference and meeting minutes for various committees, and contribution agreement templates. A full list of documents included in the review is provided at Appendix B.

Survey of key stakeholders

An on-line survey was conducted in August, 2008 with a variety of individuals who are familiar with the BSCP to obtain information about implementation and outcomes of the program.

Program staff, funding recipients (both current and previous) and working group/committee members were invited to participate in the online survey. The 45 individuals invited to participate included representatives from different types of organizations, from all provinces and territories, and from all five of the BSCP components. Respondents included representatives from all five components, and all of the provinces and territories except for Nunavut. In total, 37 individuals completed the survey (a response rate of 82%).

Interviews

Semi-structured telephone interviews were conducted in September, 2008, with a selection of key informants who are familiar with the BSCP, to obtain more in-depth feedback about implementation and outcomes of the program.

Key informants included PHAC program managers and staff, the Director General, a small number of Working Group/Party members, and a selection of funding recipients from each of the Provinces and Territories. These participants were chosen in order to represent a variety of viewpoints and to provide information relevant to the evaluation questions that could not be obtained otherwise. A total of 27 key informants were invited to participate in interviews, and 26 interviews were completed.

Expert Panel

An online expert panel was conducted to both inform and contextualize the evaluation findings and recommendations. Experts were asked to provide input on issues such as best practices and lessons learned from implementing surveillance systems, the appropriateness of current surveillance activities in Canada, and future directions for blood safety. A series of five questions were posted, one each day, on a secure online bulletin board for panellists to respond to. Invitations were extended to nine identified international experts and four agreed to participate.