ARCHIVED - Evaluation of The Prion Diseases Program
In 2008, the Public Health Agency of Canada (PHAC) elected to conduct an evaluation of the Prion Diseases Program (PDP). The objective of this evaluation was to evaluate the overall effectiveness and relevance of the prion program and to chart implementation recommendations for the road ahead. The evaluation focused on the ten years that the program has been operating.
This program was originally established in 1998 by the Laboratory Centre for Disease Control (Health Canada). The PDP concentrates on the infectious agents and host responses of Transmissible Spongiform Encephalopathies (TSEs). TSEs comprise a number of lethal transmissible neurodegenerative conditions affecting both humans (Creutzfeldt-Jakob Disease (CJD)) and animals (bovine spongiform encephalopathy, BSE, in cattle; scrapie in sheep and goats; and chronic wasting disease, Chronic Wasting Disease (CWD), in deer and elk).
The PDP is structured around three key activities:
- Creutzfeldt-Jakob Disease Surveillance System (CJDSS);
- CJD Reference Services and Applied Research; and
- Pathobiological Research.
The three PDP activities operated as two sub-projects (CJDSS in Ottawa; the latter two activities in Winnipeg) until 2005 when management was integrated in Winnipeg, with CJDSS continuing to operate out of Ottawa. The program has contracts with three nurses across Canada who conduct family interview and medical chart reviews, as well as with the University of Ottawa for neuropathology services, and with a neuropathologist who conducts the neuropathology analysis and makes the final diagnoses and formal case classifications for reporting purposes. Reference services and research are conducted primarily at the National Microbiology Laboratory (NML) in Winnipeg.
The management of the program initiated this evaluation and ensured that it considered all three activity areas individually, including those activities which are contracted out and the interaction among them.
The evaluation was conducted between the spring and fall of 2008. It covered the ten year period from 1998 to 2008. This represents a time of significant change and growth within the PDP.
This evaluation is the first full assessment of the Prion Disease Program. An evaluation framework had not been developed for this Program so the first step was the development of a Logic Model and Evaluation Framework. The assessment was conducted following Treasury Board guidelines and used multiple lines of evidence, including:
- document review;
- review of web publications;
- staff and management interviews;
- external stakeholder interviews;
- health care professionals survey; and
- steering committee focus group.
The evaluation provides conclusions on three areas of interest: relevance, success, and program design and delivery. Five sets of recommendations are presented to address the most significant issues found.
Overall, the PDP is successful in implementing its core activities and achieving its outcomes. It is relevant to the PHAC mandate, and there is a continued need for the program. Improvements in management processes would further enhance the effectiveness of the program.
There is a clear, documented mission statement and specific objectives for the PDP program since 2005-2006. A vision for the CJDSS component is developing, while an overall vision statement for the PDP is not evident. There is limited awareness of the formal mission and objectives by PDP managers and staff.
The mandate of the program has shifted over time in response to significant changes in the environment, such as disease factors, scientific developments and organizational restructuring.
The initiative is consistent with PHAC mandate. Research activities also appear consistent with the PHAC Strategic Plan however; research priorities are not well articulated among the PDP areas.
There is a continued need for core PHAC involvement in human prion diseases based on public health need for disease surveillance, the rarity of the disease, the uncertain risk environment, the specialized nature of the technical requirements and expertise, and the need to maintain and develop skills and expertise at the federal public health level.
Broadly, the activities are being implemented and are producing the expected outputs. CJDSS is successfully conducting case and event investigations, maintaining the database and publishing basic statistics, although there is limited additional analysis, research or publication of the surveillance data. Research Ethics Board approvals have increased significantly. Health care professionals in contact with the CJDSS are very satisfied with the services provided, with the exception of turnaround times for neuropathology results. Reference Services are consistently providing the testing analysis and have shown process improvements in terms of ISO certification for specific tests and improved turnaround times. Neuropathology analysis takes an extended time to complete. Some planned improvement activities, while in progress are not being completed in a timely manner, possibly related to resource constraints. Research has increased the number of collaborations both nationally and internationally leading to a greater access to funding from a wide range of agencies. The unit has developed research capacity since its inception in 1998, and has demonstrated an increase in the number of refereed papers, conference presentations, posters and abstracts, and can be considered to have moderate research publication productivity.
The PDP has broadly met its outcomes as set out in a range of funding and reporting documents generated since its inception in 1998. Some objectives based on expected diagnostic tests were over-ambitious and so have not been met. The CJDSS is considered to be successful with respect to providing effective surveillance and public health intervention. It is finding the expected number of cases of CJD annually in Canada and using internationally accepted methods of surveillance. The laboratory reference services are also successful, and appear to be fulfilling the required functions with high, quality work and ISO Accreditation in place. The success to date in improving awareness of CJD is thought to be somewhat less, with resource and capacity constraints limiting the success in this area. Overall, the research activities of the PDP are very successful, considered to be of high quality and are contributing to the understanding of prion diseases. There are opportunities to more clearly articulate the research priorities of the PDP.
There were no significant unexpected outcomes.
Design and Delivery
The PDP benefits from a very dedicated, professional staff. Recent improvements are noted in planning and reporting and staffing levels and stability. The organizational structure is quite flat, with centralised decision making and financial management. The PDP would benefit from improved decision-making structure, enhanced internal communication, and increased clarity of roles and responsibilities. Poor communication and conflict has limited the overall success of the program in the past.
The PDP has a wide range of relationships with partners, funders and collaborators, with somewhat uneven levels of effectiveness. The CJDSS relationships with hospitals and their research ethics boards appear to be good which supports delivery of the surveillance activities. Provincial/Territorial relationships have been identified by the PDP as in need of strengthening in the areas of sharing of information for public health purposes, and steps toward improvement in this area have been initiated. Relations with the EuroCJD have historically been good though concerns of a decreasing level of participation in recent years were raised. Interactions with other federal government departments is generally good, however participation in the TSE Interdepartmental Science Committee has decreased. The PDP Research Scientists have been very successful in developing collaborations to access grants from a wide range of funding agencies. However, the PDP is seen by some as isolated from other prion related organizations, including PrioNet, APRI and EuroCJD.
The PDP actively participates in a number of audits, reviews and monitoring activities. Implementation of recommendations is occurring in some cases, though capacity appears to be limiting full implementation.
Human, financial and laboratory resources have generally been adequate in the past to meet core commitments. Staffing levels, though improved, are challenged by ongoing turnover, lack of a medical specialist and limited availability of the neuropathologist. Other pressures exist including sunsetting of research funding and the loss of prion dedicated animal facilities.
While other delivery models exist, the Canadian system provides effective surveillance and reference services in response to the distinct needs presented by Canadian governments, health care capacity, population density and geography. Other organizations in Canada are conducting prion-related research; however, there are benefits to continuing to conduct research within PHAC, with a need to ensure strong collaborations with the other institutions.
The following recommendations are based on the conclusions above.
Clarify research priorities and secure funding
- Clarify and communicate research priorities, ensuring alignment to PDP and PHAC mission and objectives;
- Secure long term funding to provide security; and
- Establish a mechanism to ensure alignment of all NML prion researchers.
Review, prioritize and implement identified program improvements in line with available resources
- Review recommendations from various reviews, audits and commitments made in annual plans (e.g. Reference Centre Advisory Subcommittee Annual Report/Business Plan for National Centres Evaluation, EuroCJD Audit, and PDP Annual Report/Business Plans)
- Prioritise program improvements to respond to the various recommendations strategically based on resources and program priorities; and
- Implement these prioritised program improvements.
Review organizational structure
- Consider identifying managers from existing pool of staff and delegating responsibilities; and
- Clarify roles and responsibilities within the CJDSS nurse consultant group and with the methods development technician.
Improve internal and external communications
- Institute regular PDP team meetings;
- Distribute key planning documents and status reports amongst staff;
- Communicate surveillance publications (success); and
- Communicate more widely the current mandate and vision (future directions) for PDP internally and externally.
Review approach to partnerships and improve effectiveness of key relationships
- Identify key partners required to meet program priorities;
- Target and improve key partnerships to realize these priorities; and
- Encourage other staff to participate according to their corresponding areas of expertise.
Report a problem or mistake on this page
- Date modified: