Section 1: Evaluation of food-borne enteric illness prevention, detection and response activities at the Public Health Agency – Introduction

1. Introduction

1.1 Purpose and scope of evaluation

The purpose of the evaluation is to assess the relevance and performance of the prevention, detection and response to food-borne enteric illness activities at the Public Health Agency. Senior management requested this evaluation to support program planning and decision making. Evaluative information was also needed to inform the Public Health Agency’s contributions to the Government of Canada’s response on the implementation of the recommendations made by the Independent Investigator of the 2008 Listeriosis outbreak, Sheila Weatherill.

The evaluation addresses the relevance of these activities, including the continued need for this program and its alignment with federal government roles and priorities. It also addresses the performance of the Public Health Agency in achieving expected outcomes and demonstrating efficiency and economy. The time period examined is between April 2006 and December 2011.

The following questions guided the development of the evaluation’s data collection instruments and the collection of data. All of these questions were covered by the evaluation. Key findings for each question were then synthesized into broader conclusions.

Relevance

• What is the magnitude of food-borne enteric illness in Canada?
• Has this changed over time? Has the Public Health Agency adapted to these changing needs? What are the gaps that still need to be addressed?
• What should the Public Health Agency’s role be in preventing, detecting and responding to food-borne enteric illness? How does it align with the role of the provinces and territories? 
• How does the Public Health Agency’s role align with other government departments? Is the Agency doing what others should or could be doing?
• Does the Public Health Agency’s current role align with its own priorities and those of the federal government? What are the gaps? Is there any duplication?
• Does the Public Health Agency have a long-term strategic approach to preventing, detecting and responding to food-borne enteric illness? 

Performance

• Can the Public Health Agency improve its food-borne enteric illness activities to better serve Canadians?
  • What evidence exists on the impact of food-borne enteric illness? To what extent is the Public Health Agency and other government departments using this evidence to revise policies and programs?
  • To what extent is there a coordinated assessment of laboratory and epidemiological evidence, as well as good internal and external coordination of the use of evidence?
  • What activities are undertaken to address public knowledge and awareness of the prevention of food-borne enteric illness?
• Is the Public Health Agency likely to face the same challenges encountered in previous food-borne enteric illness outbreaks (e.g. Listeriosis)? If so, how can the Public Health Agency overcome these challenges?
• To what extent is performance measurement in place?
• Is there variance between planned and actual expenditures? If so, what are the implications? To what extent have these funds been appropriately targeted?

The scope of this evaluation does not include food security, allergens, chemicals, and water-borne and zoonotic diseases, nor does it include other food-borne non-enteric illnesses.

This report describes what the Public Health Agency does to prevent, detect and respond to food-borne enteric illness. It assesses links, outlines challenges and opportunities, and culminates in key findings and conclusions which in turn lead to four recommendations.

1.2 Methodology

Approach and design

This evaluation was conducted by Evaluation Services, a group internal to the Public Health Agency but not involved in the program areas responsible for the administration of food-borne enteric illness activities.

A goals-based model was used to examine food-borne enteric illness activities across the Public Health Agency. The evaluation also examined whether the Public Health Agency has an effective system in place to respond to food-borne enteric illness outbreaks. 

As the other purpose of the evaluation was to understand any gaps and challenges in food-borne enteric illness activities at the Agency; a process-based model was also used. In addition, while detection and response activities were reviewed after the 2008 Listeriosis outbreak, senior management requested an assessment of improvements made in the Public Health Agency following the review.

This evaluation incorporates multiple lines of evidence and a combination of qualitative and quantitative measures to ensure a balanced analysis of relevance and performance of food-borne enteric illness prevention, detection and response activities.

Methods

This evaluation used multiple lines of evidence, including a review of literature and relevant documents, as well as key informant interviews. To determine the role of public health in the prevention, detection and response to food-borne enteric illness, the evaluators looked at activities conducted in the United States, through the Centers for Disease Control and Prevention (Appendix A). They also reviewed the lessons learned from the E. coli O157 outbreak in Germany (Appendix B) and identified progress made since the 2008 Listeriosis outbreak (Appendix C).

The evaluation matrix, which outlines evaluation questions and lines of evidence, can be found in Appendix D. Evaluators conducted the interviews using structured interview guides that followed the evaluation questions identified in the evaluation framework. Evaluation tools, such as interview guides, are available upon request.

1. Literature review

Evaluators conducted a review of literature to explore the magnitude of food-borne enteric illness in Canada and whether the incidence/prevalence has changed over time. The purpose of this review was to obtain information about whether there was a need for the Public Health Agency’s activities in preventing, detecting and responding to food-borne enteric illness.

The 56 documents reviewed included literature (including unpublished literature) that program staff suggested, along with documents identified through Internet searches. As well, database searches were conducted in Medline, Embase, Global Health and EconLit for English-only articles published after 2005. The search focused on the prevention, detection, response, and control of food-borne enteric illness at the national and international levels.

2. File and document review 

Evaluation Services reviewed the files and documents available on food-borne enteric illness prevention, detection and response activities at the Public Health Agency. Files and documents were initially identified by program staff, as well as staff from other government departments, and through a search of relevant websites. This review addressed the evaluation issues related to relevance and performance. In total, evaluators reviewed 115 documents, not including websites.

The types of documents reviewed included:

• program authorities, acts, protocols and agreements
• annual and progress reports
• exercise and training reports
• strategic plans
• external evaluation and internal/external audit reports
• lessons learned reviews
• briefing notes and decks
• financial data
• public opinion research.

3. Interviews with key Public Health Agency senior managers and staff, senior managers and staff in other federal government departments, and external experts in the area of food safety

The interviews addressed particular evaluation issues related to relevance and performance, including roles and responsibilities, the description of food-borne enteric illness activities at the Public Health Agency and their performance. The interviews also identified areas for improvement and existing gaps.

In addition to the perspectives of a broad cross-section of Public Health Agency program management and staff, the evaluation considered the perspectives of some of the major stakeholders, in particular, a selection of other government department staff.

Purposive sampling was based on a snowball approach. Following an initial scoping interview with key program contacts, additional names, both internal and external to the Public Health Agency, were suggested for an interview. From September to December 2011, 47 one-hour (on average) interviews were conducted. These interviews were audio-recorded. The majority were transcribed (verbatim) and used for analysis. Where transcription was not possible (due to time constraints), notes from the interviews were used for analysis.

The breakdown of interviewees by type of key informant is as follows.

4. Case studies

Three case studies were conducted to gather additional details on: a country’s approach to food-borne enteric illness; a significant international food-borne outbreak; and progress made following Canada’s most serious food-borne outbreak. Case studies examined evaluation issues related to relevance and performance.

First, the evaluators conducted a comparative review of the United States Centers for Disease Control and Prevention’s approach to preventing, detecting and responding to food-borne enteric illness. It also included interviews with senior managers from the Centers for Disease Control and Prevention.

The second case study looked at whether the Germany E. coli Outbreak could occur in Canada. In terms of data collection, key informants from within the Public Health Agency were asked to provide insights and key documents were reviewed.

Finally, a case study looked into progress made by the Public Health Agency, following the 2008 Listeriosis outbreak, to address the recommendations made by the Independent Investigator, Sheila Weatherill. In addition to a file and document review, five interviews were conducted with program area leads. These interviewees have not been included in the table above.

Limitations

Most evaluations are confronted with constraints that may have implications for the validity and reliability of evaluation findings, conclusions and recommendations. This section discusses the limitations with respect to the design and methods for this particular evaluation. Also discussed are the mitigation strategies put in place by the evaluation team to ensure that the evaluation findings can be used with confidence to guide program planning and decision making.