ARCHIVED - Evaluation of The Prion Diseases Program

 

2.0 Evaluation Methodology

2.1 Evaluation Framework

Orientation, Rationale and Policy Context

The objective of this evaluation is to assess the success and relevance of the Prion Disease Program, the effectiveness of the programs’ design and delivery and to make recommendations if necessary. The evaluation will cover the ten years that the program has been operating.

The list of evaluation issues was developed based on the TB Evaluation Policy and the issues that the Agency would like investigated. These are summarized below and the detailed performance indicators and data sources are outlined in Annex B.

Relevance

  • Is there a clear and relevant vision and are there clear objectives for activities under PDP?
  • How has the mandate changed since program inception and what internal and external factors have contributed to the identified changes?
  • Does the initiative continue to be consistent with the PHAC mandate and government-wide priorities?
  • Is there a continued need for the program? Do current client needs and risk environment indicate a continued need?

Success

  • Are activities being implemented as planned and producing the expected outputs?
  • Have the expected outcomes and reach been achieved through program activities?
  • Have there been any unexpected outcomes?

Design and Delivery

  • Are there appropriate management and decision-making structures in place to meet the objectives?
  • Are there effective partnerships / linkages to external programs in place and do they support the delivery of the program?
  • Has there been an assessment and strategic use of lessons learned?
  • Are the resources adequate to achieve the expected outputs and outcomes?
  • Are there alternative means of achieving these same program objectives that might be more efficient or effective?

2.2 Data Collection through Multiple Lines of Evidence

Data collection consisted of the following elements:

  • Document review including policy documents, program documentation, web sites, previous evaluations and audits, Reference Centre Review Reports, MOUs and agreements, and planning and reporting documents, and performance reports/statistical reports;
  • Interviews with PHAC staff (See below for a list of the groups of interviewees and Annex E for the interview guides);
  • Interviews with key external stakeholders;
  • Survey of physicians and healthcare professionals; and
  • A Steering Committee Focus Group meeting.

Table 1: Interviews by Group

Interviews by Group Number
Internal 
PDP Staff 10
PHAC Management 3
CJDSS Investigators 2
External 
HC 3
PrioNet 1
Canadian Food Inspection Agency 1
University of Alberta 1
Physicians and Health Practitioners 11
Total 32

2.3 Data Quality and Limitations of the Evaluation

Overall, data quality is generally considered to be very good since 2005, however there is substantially less information available for the early years of the program. Planning and reporting documentation was very limited prior to 2005, resulting in a reliance on verbal reports and recollection for the activities and outputs in the early years. For the years that the PDP was split between Ottawa and Winnipeg, the only financial data that was retrieved pertains to the expenditures for the Winnipeg operations.

The health care professionals survey, while providing valuable information, was not conducted in a random manner and provides a very small sample.

Page details

Date modified: