ARCHIVED - Evaluation Framework of the Canadian Food Inspection Agency's Food Safety Activities Related to Fresh Fruit and Vegetables

Abstract

This evaluation framework is a plan for evaluating CFIA's food safety activities related to fresh fruit and vegetables. The two key elements of this framework are the logic model of CFIA's food safety activities related to fresh fruit and vegetables and the evaluation matrix. The logic model is a diagram which depicts the main inputs, activities, outputs and outcomes related to the safety of fresh fruit and vegetables available in Canada. The evaluation matrix tabulates evaluation questions, as well as their indicators and proposed data analysis and methodologies using multiple lines of evidence to answer these evaluation questions. The evaluation questions are organized under four broad categories: rationale, design, delivery, and outcomes. The implementation of this plan in a future evaluation will determine the CFIA's effectiveness in achieving its intended goals and objectives related to the safety of fresh fruit and vegetables.

Bureau of Food Safety Assessment, Health Canada
July 2007

ISBN: 0-662-46243-9
Cat. H164-46/2007E
HC Pub. No. 4299

Table Of Contents

Copyright

"Our mission is to help the people of Canada maintain and improve their health"
Health Canada

Contents may not be reproduced for commercial purposes, but any other reproduction, with acknowledgements, is encouraged.

Également offert en français sous le titre :

"Cadre d'évaluation des activités de l'Agence canadienne d'inspection des aliments en matière de salubrité des fruits et légumes frais".

For further information or to obtain additional copies, also in alternate formats, please contact:

Ashwani Wadhera, Director
Bureau of Food Safety Assessment
Sir Frederick Banting Building
4 East - 2204D
Ottawa, Ontario K1A 0L2

Telephone: (613) 954-2996
FAX: (613) 954-0149
E-mail: BFSA_BESA@hc-sc.gc.ca

© Her Majesty the Queen in Right of Canada, represented by the Minister of Health, 2007

ISBN: 0-662-46243-9
Cat. H164-46/2007E
HC Pub. No. 4299

Executive Summary

The Health Canada's (HC's) Bureau of Food Safety Assessment (BFSA) uses an evaluation approach in accordance with Treasury Board of Canada Secretariat's Evaluation Policy to evaluate the effectiveness of the Canadian Food Inspection Agency's (CFIA) activities related to food safety. BFSA also provides feedback to HC's Food Directorate with respect to developing policies, standards, and guidelines.

This evaluation framework is a plan for evaluating CFIA's food safety activities related to fresh fruit and vegetables (FFV). The implementation of this plan in a future evaluation will determine the CFIA's effectiveness in achieving its intended goals and objectives related to the safety of fresh fruit and vegetables. The time of implementation of the evaluation will be determined by the joint HC-CFIA Food Safety Advisory Committee at its next planning session.

Under the Canada Agricultural Products Act, the Fresh Fruit and Vegetable Regulations objective is to provide fresh fruit and vegetables that are safe, wholesome, properly graded, packaged and labelled. The federal regulatory control of FFV falls under CFIA"s Fresh Fruit and Vegetables Program (FFVP) in the Agri-Food Division reporting to the Food Safety Directorate. The FFVP through product inspections, aims to contribute to the safety of fresh fruit and vegetables produced domestically, imported into Canada and/or traded inter-provincially.

The FFVP is supported by internal partners such as the Food Safety Investigations Program (FSIP) in the Food Safety Division (formerly the Bureau of Food Safety and Consumer Protection), Food Safety Directorate which enforces the food safety provisions of the Food and Drugs Act and Regulations through activities such as establishment assessments and inspections, food safety investigations, and risk-management practices. The FSIP focuses on imported foods and those domestic foods that are not manufactured or produced under other federal inspection programs such as the FFVP. The FFVP is also supported by the Food Microbiology and Chemistry Evaluation Division (FMCED), which works closely with FFVP on risk assessment issues and trend analysis, the design and development of sampling plans and selection of analytical methodologies.

The two key elements of this framework are the logic model of CFIA's food safety activities related to fresh fruit and vegetables and the evaluation matrix. The logic model is a diagram which depicts the main inputs, activities, outputs and outcomes (in both ongoing and aspired terms) related to the safety of fresh fruit and vegetables available in Canada. The evaluation matrix tabulates evaluation questions, their indicators as well as proposed data analysis and methodologies using multiple lines of evidence to answer these evaluation questions. The evaluation questions are organized under four broad categories: rationale, design, delivery, and outcomes. As the FFVP intends to go through various changes in the future (stronger food safety focus), this evaluation places emphasis on program design and its delivery along with the potential impacts for the future re-design. This framework can also assist the CFIA in the development of performance measures for FFVP and to further promote the Program's science-based and risk-management practices.

This evaluation framework consists of a proposed evaluation strategy agreed to by the joint Health Canada and Canadian Food Inspection Agency Food Safety Assessment and Advisory Committees.

1.0 Introduction

1. There is an increasing demand by the Canadian public for a continuous supply of fresh fruit and vegetables (FFV) in order to fulfill Canadians' dietary needs and nutritional goals. Traditionally, the public perceived that the food safety risks (with the exception of pesticides, irradiation and fruits and vegetables derived from genetically modified plants) associated with FFV were low whereas reality indicates a major concern is with microbial contamination of produce as indicated in section 1.4 below 1 2 3. As well, developments in globalized trade with a wide variety of imported fresh produce, new process technologies, and new products (fresh-cut and prepackaged salads) have brought new identified hazards and risks.

2. Consequently, CFIA's FFV Program (FFVP) (as depicted in the FFVP logic model (section 3.2)) intends to re-design its program and activities related to domestic and imported fresh fruit and vegetables. The re-design objective is to improve overall coverage, encompassing all FFV related food safety hazards while introducing new activities and controls to reduce, mitigate, and eliminate food safety risks associated with FFV across the food continuum.

1 Views of the Public & Stakeholders on Health Products and Food: A report to Assist Policy Development in HPFB (Paul MacNeil Consulting, 2001).
2 Proposal for the Re-design of the FFV Food Safety Program, CFIA. 2005
3 "Foodborne Outbreaks in Canada Linked to Produce" A.M. Sewell & J.M. Farber, Journal of Food Protection, 2001

1.1 Evaluation Scope

3. Section 11(4) of the Canadian Food Inspection Agency Act provides the Minister of Health with the responsibility of assessing the effectiveness of CFIA's food safety activities in accordance with the Treasury Board of Canada Secretariat's Evaluation Policy. Health Canada's Bureau of Food Safety Assessment (BFSA) conducts these assessments with the objectives of providing advice and guidance to CFIA on its food safety activities, and providing feedback to HC to assist in carrying out its roles of developing food safety and nutrition policies and standards.

4. In September 2003, the joint HC-CFIA Food Safety Assessment Advisory Committee (referred to as the Advisory Committee in this document) comprising senior representatives from HC and CFIA identified domestic fresh fruit and vegetables as an evaluation priority for the fiscal year 2004-2005 (see Appendix A).

In March 2005, due to the proposed re-design of the FFVP and after negotiations with the Advisory Committee, it was decided to develop an evaluation framework and logic model for the FFVP, that included the proposed FFVP re-design and import activities. The full FFVP evaluation would be conducted at a later date, to be agreed upon by the Advisory Committee. In addition, any other CFIA activities related to FFV, such as the sprouts project under the CFIA's Food Safety Investigation Program (FSIP), may be considered in the future evaluation.

1.2 Evaluation Approach

5. In collaboration with the FFVP, BFSA developed an evaluation framework which is essentially a guidance document for assessing the effectiveness of FFVP's food safety activities in a future evaluation. The evaluation framework includes a logic model of both the current and proposed FFVP food safety activities. The logic model is a visual diagram that outlines activities and outputs of the program as well as intended outcomes. This logic model was used to formulate evaluation questions concerning program rationale/ relevance, design, delivery, and outcomes. As part of the evaluation framework, these evaluation questions are presented in a matrix that identifies the information required to address them (indicators), data sources, as well as methodologies for collecting and analysing the information. These evaluation questions are the basis on which the effectiveness of CFIA's food safety activities related to FFV will be assessed in the future evaluation. The evaluation framework also outlines some evaluation challenges that need to be examined such as data availability, multiple jurisdictions, food continuum, and provides an evaluation strategy for the future assessment of the FFVP.

6. In order to understand the FFVP and identify key food safety activities for the development of a logic model, the assessment team interviewed Agency staff at National Headquarters, and in the Ontario and Quebec Area offices which account for 36% and 15% of farm cash receipts for the agricultural sectors in Canada respectively 4. The team also met with industry stakeholders 5 to obtain a better understanding of the FFV community and its concerns within Canada. Interviews were also conducted with key partners within CFIA involved in FFV activities such as the FSIP which were involved in a special project between the FFVP, Provinces and industry stakeholders to mitigate recent food borne illnesses linked to sprouts.

4 A Profile of the Horticultural Sector and Associated Economic Contribution JRG Consulting Group, October 2005
5 Stakeholders include: consumers, wholesalers, retailers, producers, manufacturers, distribution centres and importers and their respective associations.

1.3 Fresh Fruit and Vegetables Industry and Consumption in Canada

7. The fresh fruit and vegetable industry in Canada comprises approximately 25, 000 domestic farms, some 80-100 minimal processors 6 , 800-1000 packers and re-packers, and 1000 wholesalers, food services and importers involved in interprovincial and international trade (see Appendix B). 7

8. The CFIA's proposed re-design of the fresh fruit and vegetable's food safety program directly and indirectly targets packers, minimal processors, wholesalers, importers and food services in a manner that influences the primary production sector to implement food safety provisions. This re-design would result in new CFIA activities for the middle segment of the industry and subsequently promote their implementation of controls such as:

  • Good Agricultural Practices (GAPs);
  • On Farm Food Safety Programs (OFFS);
  • Memoranda of Understanding (MOUs); and
  • and Good Importing Practices (GIPs).

9. The "Canada's Food Guide to Healthy Eating8" indicates that fresh fruit and vegetables can contribute significantly to the health and well-being of Canadians and recommends that these foods constitute a major component of their diets. In addition, there is a growing array of fresh-cut products, prepackaged salads, processed products, and imported produce available in the marketplace because of the increased globalization of the food supply. Imports account for 88% of the total volume of fresh fruit and 48% of the fresh vegetables, and originate from about 150 countries with the majority coming from United States, Mexico, the Caribbean, and other Central and South American countries. 9

10. According to Statistics Canada, fresh fruit and vegetables are the highest consumed commodities in the Canadian diet in terms of kg per person. Consumption patterns have increased from 34.64 to 37.59 kg of fresh fruit and 62.58 to 74.78 kg of fresh vegetables per person, per year (1976-2004) 10. Though traditional fruit and vegetables are still the main choice, tropical and foreign produce is becoming increasingly popular. Moreover, new vegetables are continuously being incorporated into the Canadian diet.

6 Minimally processed vegetables consist of raw vegetables that have been peeled, sliced, chopped or shredded prior to being packaged for sale in Canada.
7 Update to the draft Proposal for the Re-design of the FFV Food Safety Program, CFIA. 2006.
8 Canada's Food Guide to Healthy Eating
http://www.hc-sc.gc.ca/fn-an/food-guide-aliment/index-eng.php, Health Canada, 2007-09-05
9 Draft Proposal for the Re-design of the FFV Food Safety Program, CFIA. 2005.
10 Food Statistics, 2004 Table 5, http://www.statcan.ca/english/freepub/21-020-XIE/21-020-XIE2004002.pdf, Statistics Canada, 2007-09-05

1.4 Food Safety Concerns Related to Domestic and Imported Fresh Fruit and Vegetables

11. Raw fresh fruit and vegetable commodities may contain safety hazards and if not controlled, could result in foodborne illness. A "hazard" as defined by the Codex Alimentarius Commission is a biological, chemical or a physical agent in or condition of food with the potential to cause an adverse health effect.

12. Over the past decade, agricultural practices have changed and new commodities from countries with diverse food safety infrastructures and controls have arrived, challenging the FFVP's current food safety activities used to assess potential health hazards. Many microbiological pathogens are ubiquitous in the environment and their presence on fresh fruit and vegetables are unavoidable. The control of organisms must occur throughout the food continuum including in the processing/packaging (fresh cut) environment, by preventing conditions favourable to their growth. There is increased evidence that microbiological hazards including bacteria, viruses and protozoans such as Escherichia coli O157:H7, Listeria monocytogenes, Shigella, and Cyclosporeae cayetanensis and Hepatitis A, have been implicated in FFV-related foodborne outbreaks in the past two decades. 11 In addition, concern exists that certain fruit and vegetables are more prone to contamination based on production practices, physical characteristics, and extensive human handling. 12

13. Chemical contaminants associated with fresh fruit and vegetables including pesticides, environmental contaminants, metals, and natural toxins may have undesirable short-term or long-term effects on human health, such as contributing to chronic diseases (e.g., pesticide residues), and causing acute illnesses, including mortality (e.g., some natural toxins).

14. A CFIA-commissioned report 13 ranked the FFVP as a high priority for a re-design review to ensure adequate coverage of potential chemical hazards, such as pesticides, heavy metals, environmental contaminants, safe protective waxes and coatings, food additives (sulfites) and natural toxins.

Other report highlights comment on the FFVP's capacity to manage the concerns related to genetically modified fruit and vegetables, irradiation, nutritional labelling and oral allergy syndrome.

15. According to the CFIA's Annual Performance Report of 2004-2005, the compliance rates for FFV chemical residue testing conducted by the Food Microbiology and Chemical Evaluation Division (FMCED) were 97.6%, 98.9% and 99.8% in 2002-03, 2003-04, and 2004-05 respectively. The compliance rates for FFV microbiological testing conducted by FMCED were 93.7%, 99.8% and 99.2% in 2002-03, 2003-04, and 2004-05 respectively. Work plans typically call for approximately 10,500 chemical residue samples (2,500 domestic and 8,000 import produce) and 700 microbial samples (400 domestic and 300 import produce) per year. 14

11 "Foodborne Outbreaks in Canada Linked to Produce" A.M. Sewell & J.M. Farber, Journal of Food Protection, 2001
12 Results of the Ontario Fruit and Vegetable Microbial Baseline Study, Ontario Ministry of Agriculture, Food and Rural Affairs, 2005
13 CFIA Resource Review Program Re-design Needs and Priority, KPMG, 2002
14 FFV Food Safety Program Presentation, CFIA, 2006

1.5 Food-borne Illness Data

16. Between 1991 and 2000, Canada had 15 microbial foodborne outbreaks associated with fresh fruit and vegetables, resulting in 1,364 illnesses and one death. Thirteen of these outbreaks occurred between 1995 and 2000 (no data available for 2001-2004). In 2005-2006 there were four fresh fruit and vegetable related microbial outbreaks resulting in 900 illnesses respectively in Canada. Fresh produce implicated in foodborne illnesses include berries, cantaloupes, field tomatoes, green onions, fresh herbs, sprouts, and fresh-cut fruit and vegetables.

17. It is difficult to verify the link between FFV and foodborne illness due to limits in data collection and analysis both within CFIA and their stakeholders. Foodborne disease is greatly under reported for a variety of reasons such as: the ill individual does not seek medical help; the correct specimens may not be collected or discarded, and the appropriate tests performed; and the results may not be reported in a manner resulting in the cases being reported at the provincial or national level. As well, although CFIA and provincial health officials carry out investigations, the source of contamination and underlying causes may not always be identified. Increased overall consumption of FFV, along with a corresponding rise in the ready-to-eat raw fresh-cut produce, an increasingly diversified supply, and an ageing population may also contribute to an increase in foodborne illnesses related to FFV contamination.

2.0 Program Profile

2.1 CFIA Governance Structure

18. The CFIA is headed by a President, who reports to the Minister of Agriculture and Agri-Food. Executive Committee members from CFIA's Branches and Directorates are accountable for developing and delivering the Agency's policies, programming, and administrative functions. CFIA's Programs Branch consists of 14 Programs in animal health, plant protection and food safety, each responsible for program design and management. The FFVP, as one of the 14 Programs, falls within the Agri-Food Division of the Food Safety Directorate. The FFVP collaborates with other CFIA partners such as the Office of Food Safety and Recall (OFSR), the Food Microbiology and Chemical Evaluation Division (FMCED), and the Food Safety Division's, Food Safety Investigation Program (FSIP) on food safety activities such as those related to sprouts. The Operations Branch at the national and area levels is responsible for the delivery of the 14 Programs, primarily focussing on monitoring, compliance and enforcement activities. The CFIA's Science Branch conducts research, develops methods and provides scientific advice and analysis in support of all 14 Agency Programs. Scientists in CFIA's laboratories also analyse samples collected each year by CFIA inspectors.

2.2 FFV Program Description

19. The CFIA's FFVP includes within its scope: fresh fruit, fresh vegetables, fresh herbs, sprouts, fresh mushrooms, as well as fresh-cut fruit and vegetables ready-to-eat or not. Ready-to-eat (RTE) fresh fruit and vegetables are defined as those that have been washed and peeled, sliced, chopped or shredded prior to being packaged for sale and are intended to be consumed raw and not for further processing or cooking, e.g., shredded bagged lettuce, coleslaw, baby carrots, fresh-cut melons or fruit salad. Genetically modified organisms (GMO) are also included in the scope of the FFVP.

20. The Program excludes processed fresh fruit and vegetables in respect of a food product that is, canned, cooked, frozen, concentrated, pickled or otherwise prepared to assure preservation of the food product in transport, distribution and storage, but does not include the final cooking or preparation of a food product for use as a meal or part of a meal such as may be done by restaurants, hospitals, food centres, catering establishments, central kitchens or similar establishments where food products are prepared for consumption rather than for extended preservation.15

21. The FFVP activities primarily focus on national food safety monitoring where samples are taken and analysed for chemical and microbiological hazards associated with domestic and imported FFV commodities. Other FFVP food safety activities are reactive in nature and include investigation and follow-up on product violations, foodborne illness outbreaks, and consumer complaints. These activities are managed jointly by Programs Branch and Operations Branch under the FFVP.

22. The CFIA has the authority to establish regular inspection programs for the industry involved in interprovincial and international trade of FFV. Currently, no establishment inspection program exists with dedicated full time equivalents (FTEs).

23. Other FFVP activities include: collaborative initiatives with national stakeholders such as the On Farm Food Safety Programs (OFFSPs); development of qualitative risk assessments and risk management policies for specific import commodities (e.g., Guatemalan raspberries) and domestic sprouts (e.g., Code of Practice for the Hygienic Production of Sprouted Seeds); and development of consumer food safety fact sheets.

24. The FSIP enforces the safety and nutritional quality provisions of the Food and Drugs Act and Regulations (FDAR) via establishment inspections, food safety investigations (including consumer and trade complaint investigations), and risk management activities. FFVP and FSIP collaborate and investigate specific activities such as management of food safety hazards. If required, OFSR conducts FFV recalls. Science committees were established to ensure a risk-based approach to setting program priorities to risks identified on a horizontal basis across the Agency. The priorities and recommendations developed by the science committees are used to develop specific strategies and projects to manage food safety risks. According to the risk analysis report developed by the Joint CFIA-HC Science Committee(CFIA/HC Food Safety Science Committee Report, Microbiology, November 23-25,2005), fresh produce and sprouts are medium priorities based on their potential contamination with Shigella and Salmonella, respectively.

15 Definition taken from the Processed Products Regulations pursuant to the Canada Agricultural Products Act.

3.0 Logic Model

3.1 Narrative for the Logic Model

25. As noted previously, a logic model is a diagram that depicts the relationships amongst the main elements of a program, thus portraying the intended causal chain of events linking activities, outputs, and related outcomes. As the FFVP may be re-designed, the logic model depicts both current and proposed program elements, the latter identified in shaded boxes. Both current and proposed elements were used to formulate evaluation questions which can be found in section 4.0 Evaluation Matrix.

26. This narrative explains the activities, outputs and the intended outcomes for the FFVP and should be read in conjunction with the diagram of the FFVP logic model found in section 3.2. Both the logic model diagram and the narrative were developed in collaboration with CFIA's Audit, Evaluation and Risk Oversight (AERO) as well as FFV Program and Operations staff. For the purpose of this evaluation framework, the FFV logic model focuses on food safety elements for both domestic and imported fresh fruit and vegetables.

3.1.1 Inputs

Legislation

27. Fresh fruit and vegetables are covered by the following legislation:

  • Canadian Food Inspection Agency Act (CFIA Act);
  • Canada Agricultural Product Act and FFV Regulations (CAP Act & FFV Regulations);
  • Consumer Packaging and Labelling Act and Regulations (CPL Act & Regulations); and
  • Food and Drugs Act and Regulations (FDAR).

28. These Acts and Regulations give CFIA inspectors the authorities to inspect fresh fruit and vegetables and collect samples of domestic and imported produce. However, the Acts do not regulate primary production (growing and harvesting) of agricultural products as these generally fall under provincial and territorial jurisdictions. The FFV Regulations contain food safety provisions for post harvest activities and require that produce meets all pertinent provisions of the FDAR. As well, the Consumer Packaging and Labelling Act contain food safety provisions for the labelling of fresh fruit and vegetables. The Food and Drugs Act and Regulations applies to the importation and manufacture of all food sold in Canada.

Food Safety Policies Related to Fresh Fruit and Vegetables

29. Health Canada (HC), through its Food Directorate, has the statutory responsibility to establish policies and standards relating to the safety and nutritional quality of food sold in Canada. HC's Pest Management Regulatory Agency (PMRA) establishes Maximum Residue Limits (MRLs) for pesticides under the FDA. Health Canada also provides CFIA with health risk assessments (HRAs), recommendations on specific food safety issues, and letters of no objection for the usage of non-food products in the food industry.

CFIA is responsible for enforcing the food safety policies and standards that Health Canada sets, by establishing guidelines and codes of practice in collaboration with Health Canada, to reduce the risks associated with fresh fruit and vegetables (see section 3.1.2 for further information).

Partnerships with Provinces or with Foreign Governments

30. Partnership arrangements with provincial governments refers to activities to be conducted with respect to domestic fresh fruit and vegetables, for example sharing of sampling activities between CFIA and le ministère de l'Agriculture, des Pêcheries et de l'Alimentation du Québec (MAPAQ). Another example of an established partnership arrangement between Health Canada, CFIA and the provinces includes the development of the Foodborne Illness Outbreak Response Protocol (FIORP).

31. With respect to the FFV food safety activities, relationships with foreign governments, for example the United States Food and Drug Administration (USFDA) and the Servicio Nacional de Sanidad, Inocuidad y Calidad Agroalimentaria (SENASICA) are maintained and enhanced through ad hoc collaboration on emerging issues. Other collaborative initiatives include a certification program, Import Requirements for Mexican Cantaloupes, and the Import Policy for Guatemalan Fresh Raspberries and Blackberries.

3.1.2 Activities and Outputs

Develop/Review Inspection Materials

32. Inspection materials include but are not limited to food safety standards, inspection manuals and procedures, inspection and import directives, inspection work plans, and food safety training modules. Manuals and procedures are developed and updated by Programs Branch in consultation with Operations Branch (Areas) as well as key stakeholders and partners. For example, the Traceback Investigation Guidelines for Fresh Fruit and Vegetables was developed by Headquarters (HQ) Program in October 2001 and sent for review to CFIA Areas and Industry Associations in 2003. The FFVP in consultation with the Food Safety Division and Health Canada developed a Code of Practice for the Hygienic Production of Sprouted Seeds to assist sprout manufacturers in minimizing the risk associated with the consumption of sprouted seeds and beans. The FFVP also developed a Code of Practice for Minimally Processed Ready-to-eat Vegetables and a Manual on Fresh Fruit and Vegetables Sampling for Laboratory Testing.

33. The FFVP annual work plans (number of samples to collect and type of analysis required) are developed in HQ by the Food Microbiology and Chemical Evaluation Division (FMCED) in collaboration with the FFVP and the Lab Services Division (Science Branch) and then refined and organized by Areas and Regions. These work plans mostly consist of monitoring sampling plans but in the future FFVP re-design would include establishment (domestic and import) inspection plans. The FFVP work plans are developed for both domestic and imported products and are based on risk profiles and data such as compliance rates, non-compliant establishments, and availability of commodities in the marketplace provided by the Areas.

34. The FFVP makes recommendations on their training needs to the Human Resources National Training Unit (NTU) which provides funding for the delivery of the training across all Areas. For example, the Food Safety Division's, Food Safety Investigations Program (FSIP) worked in collaboration with the FFVP to develop and deliver a National Training Initiative (NTI) on Introductions to the Inspection of Sprouts Establishments. In addition, informal training is provided in the Areas/Regions via job shadowing and mentoring by senior inspectors.

35. Outputs for the activities above include standards, manuals, procedures, directives, work plans, and training modules.

Conduct Data (including Trend) Analysis

36. Data analyses are conducted using results from microbiological, chemical, and irradiation analyses, investigation reports, issue tracking sheets, international alerts, and food-borne illness reports for the purpose of revising work plans and also standards, manuals, procedures, and directives. These trend analyses are completed on an annual basis at the HQ level with input from the Areas/Regions. As a result of doing trend analyses, the FFVP developed import directives such as the Import Policy for Guatemalan Fresh Raspberries and Blackberries and the Import Requirements for Mexican Cantaloupes. In the future, results from establishment inspection reports would also be included to develop trend analysis.

37. The FSIP contributes to the roll-up analyses by providing results on sprouts sampling and analyses, and on establishment inspection activities.

38. The outputs of conducting trend analyses are revised work plans (including sampling plans) and also revised standards, manuals, procedures, and directives.

Conduct Product Sampling

39. In the four Areas (18 regions) across Canada, the FFVP operations inspectors collect samples of fresh fruit and vegetables in accordance with national sampling procedures and guidelines, in order to fulfill the monitoring sampling plans for microbiological analysis, chemical analysis, and irradiation analysis. Samples are taken at warehouses, wholesalers, and import establishments. The microbiological and irradiation analyses are conducted in CFIA laboratories while chemical analyses are done by a private laboratory. Currently, routine product monitoring is the prominent food safety activity of the FFVP. When a laboratory test reveals non-compliance, additional follow-up activities are initiated.

40. There are three levels of sampling activities for chemical and irradiation analyses in the FFVP - monitoring, surveillance and compliance based on historical produce data.

The monitoring phase is designed to gather data and provide information on the occurrence of chemical hazards and irradiated foods through random sampling of fresh fruit and vegetables. If the samples are found to be in violation, the product is put under the surveillance phase or follow up action is taken.

The surveillance (directed sampling) phase is conducted to confirm presumptive positive results and identify suspected problems. A specific commodity is targeted and samples from five shipments are collected and analysed. If all five samples are in compliance, the product is returned to the monitoring phase. If any one of the five samples is in violation, the product is placed on the compliance list.

The compliance phase is a regulatory control measure to prevent the sale and facilitate the removal from the marketplace of any product known to be contaminated and/or adulterated and/or in violation of standards. Shipments of the product on the compliance list are then tested at a recognized laboratory at the expense of the grower and shipper. The product will remain on the compliance list until five consecutive samples are in compliance. Then, the product is returned to the monitoring phase.

41. There are also three levels of sampling/inspection for microbiological contamination: monitoring, investigation and/or establishment inspection, and compliance.

42. The monitoring phase is designed to gather data and provide information on the occurrence of microbial hazards through random sampling of fresh fruit and vegetables. If the samples are found to be in violation, the product is put under the surveillance phase or follow up action is taken.

43. If a violation is found on an imported product, the FFVP will contact the foreign country's embassy to request that they carry out a follow up investigation in their country. In the case of an American or Mexican product, the FFVP will contact the USFDA or SENASICA directly.

44. If evidence leads the CFIA to believe that no action is taken in the country of origin or that the product still poses a food safety risk, an import policy is developed which may include a restriction on the imported product. Examples include the Import Policy for Guatemalan Fresh Raspberries and Blackberries and Import Requirements for Mexican Cantaloupes.

45. In the case of domestic produce an investigation/ establishment inspection is performed to determine the source of contamination and corrective measures are developed to minimize the risk.

46. The FSIP conducts sampling activities for sprouts to gather data on the occurrence of microbial hazards, confirm presumptive positive results, and remove unsatisfactory samples from the marketplace.

47. The outputs of conducting product sampling include product sampling reports and roll-ups as well as monitoring, surveillance and compliance lists. Such reports, roll-ups and lists are used to determine compliance rates and identify problematic producers or importers and high risk commodities. The data generated from these reports, roll-ups and compliance lists is used to conduct data/trend analysis and environmental scans, one of the FFVP's major activities. Compliance lists for fresh fruit and vegetables can be found on the CFIA website.

Conduct Establishment Inspections

48. Currently, FFV establishment inspections 16 take place only as part of investigations or through work specifications by the Food Safety Division. When a food safety risk is identified, the establishment must implement corrective measures to comply with CFIA's requirements. The FFVP does not require the industry to have HACCP systems in place or to be federally registered; however, some large retail and wholesale clients in the FFV industry require suppliers to have HACCP systems in place, as a condition of purchase.

49. Follow-up establishment inspections take place to verify Good Manufacturing Practices (GMPs) of the establishment. Following the inspection, effective corrective actions are requested by CFIA and sent to the establishment. CFIA does follow-up visits to ensure that the corrective actions are implemented.

50. According to the proposal to re-design the FFVP, food safety establishment inspections (domestic and importer verifications and audits) would be implemented. The proposed establishment inspection would consist of an in-depth evaluation of the establishment's food safety controls as well as its equipment and environmental conditions which would incorporate Good Manufacturing Practices (GMPs), Good Importing Practices (GIPs) elements and HACCP systems.

51. In the Areas, Food Safety Division inspectors conduct establishment inspections of production, packing and sorting establishments dealing with sprouts and pre-cut fresh ready-to-eat vegetables.

52. The outputs of this activity are establishment inspection reports and roll-ups. Such reports are used by FFVP to determine compliance rates and identify problematic importers, establishments and high risk commodities. Data generated from these reports and roll-ups are also used to conduct trend analysis and environmental scans, one of the FFVP's major activities. In the future, establishment compliance may also be used to determine inspection frequency.

Conduct Investigations and Non-compliance Actions

53. Food safety investigations are conducted as a result of product violations, food borne illness outbreaks, consumer complaints, and international advisories. Non-compliant test results or product violations (e.g., positive results for pathogens such as Salmonella spp and Listeria monocytogenes) lead to non-compliance activities such as follow-up sampling and testing, surveillance letters sent to establishments related to the non-compliant product, establishment inspections, and border lookouts. Follow-up inspections verify that corrective actions are implemented by the establishment and are effective. Repeated failure of an establishment to correct identified deficiencies can result in enhanced non-compliance actions including compliance letters and detentions until corrective actions are implemented.

54. International food safety issues, for example lettuce from the US found to be contaminated with perchlorate, also lead to food safety investigations. Likewise, a Hepatitis A outbreak in the US linked to Mexican green onions led to a food safety investigation by CFIA to ensure that similar products did not pose a health threat to Canadians.

55. When regulations or standards do not exist for a given food safety issue, (e.g., perchlorate in lettuce, cyclospora in raspberries) an investigation is conducted and FMCED's technical specialists may request health risk assessments (HRAs) from Health Canada. Further to HRAs, recalls of non-compliant products may be initiated by CFIA via the Office of Food Safety Recall (OFSR) or import directives may be issued.

56. The outputs of this activity include investigation and non-compliance follow-up reports, detentions, border alerts, HRA requests, import directives and recalls. Data generated from these reports are used to conduct trend analyses and environmental scans, one of the FFVP's major activities.

Develop and Provide Information and Advice for Consumers, Domestic Industry and Importers, Federal/Provincial/Territorial (F/P/T) Partners

57. The FFVP develops reference material for consumers, the domestic and import industry as well as for various government partners.

58. The FFVP at the HQ level, in collaboration with the Areas/Regions, stakeholders and partners such as Health Canada, develops fact sheets for dissemination to consumers. These include:

  • Food Safety Facts for Fresh Fruit and Vegetables;
  • Food Safety Measures for Cantaloupes;
  • Sprouts Health Risks;
  • Natural Toxins in Fresh Fruit and Vegetables;
  • Oral Allergy Syndrome;
  • Safety Tips for Black Widow Spiders in Grapes; and
  • Why Sparks Occur when Cooking Vegetables in the Microwave

all of which can be found on the CFIA website.

59. In collaboration with Areas/Regions, stakeholders and partners, the FFVP develops food safety reference material for the industry such as:

  • Code of Practice for Hygienic Production of Sprouted Seeds;
  • Code of Practice for Minimally Processed Ready-to-Eat Vegetables;
  • Submission Guidelines for the Acceptability of Novel Packaging Materials;
  • Traceback Investigation Guidelines for Fresh Fruit and Vegetables; and
  • Import Policy for Guatemalan Fresh Raspberries and Blackberries and the Import Requirements for Mexican Cantaloupes import directives.

60. In the re-designed FFVP, reference materials will be developed to assist industry in taking more responsibility for fresh fruit and vegetables' food safety.

61. The Agency also provides technical review feedback 17 to stakeholders on Good Importing Practices (GIPs) and On Farm Food Safety Programs (OFFSP). There are plans in the future to develop and provide information to better inform consumers, importers, and distributors on their responsibilities related to fresh fruit and vegetables.

62. Through establishment inspections, the Food Safety Division inspectors provide some training for the producers. In addition, Food Safety Division inspectors exchange information with their provincial counterparts (e.g., MAPAQ) in relation to sprouts.

63. In addition to publications such as fact sheets for consumers and reference material for industry (e.g., codes of practice), outputs can include GMPs and GIPs as well as information related to fresh fruit and vegetables provided through CFIA's Automated Import Reference System (AIRS).

16 Establishment includes packers and re-packers, minimal processors, wholesalers, importers and food services (see Appendix B)
17 Through Technical Review, CFIA's review team, comprised of commodity specialists, examine and assess the producer organization's on-farm food safety program to confirm that it is technically sound and adheres to Hazard Analysis Critical Control Points (HACCP) principles.

3.1.3 Reach

64. Although not included in the logic model graphic, reach is an important element of the model. Reach consists of the individuals and organizations affected by a policy, program or initiative, directly or indirectly. In the case of the FFVP, the reach includes partners and stakeholders such as Health Canada, provincial/territorial governments, CFIA's inspectors, industry associations, wholesalers, manufacturers, distributors, importers and Canadians who consume fresh fruit and vegetables.

3.1.4 Outcomes

65. The FFV outcomes are immediate (short-term), intermediate (longer-term) and ultimate societal (future) in nature. The immediate outcomes of the FFVP can be summarized as increasing the level of awareness and knowledge of food safety issues and best practices related to fresh fruit and vegetables by CFIA's Operations, industry, and domestic and import establishments as well as consumers. This increased level of knowledge and information will lead to the intermediate outcomes of better behaviour and practice related to the food safety of fresh fruit and vegetables.

3.1.5 Immediate Outcomes

66. The immediate outcomes are the intended results derived from key activities and outputs related to fresh fruit and vegetables, as stated in the logic model.

Inspectors have the necessary tools to carry out their activities

67. Inspection is the backbone of the FFVP system. To deliver inspection services, inspectors require technology to measure compliance of FFV with Acts/Regulations (including for example, effective performance measurement/management framework, timely data reports and analyses, decision-making capacity and tools). However, inspectors also require training (and associated tools); comprehensive, current standards (Acts/Regulations); manuals, procedures, directives, work plans; and appropriate resources (human, financial). Training provides inspectors the ability to perform their activities more efficiently and effectively.

Industry is aware of FFV food safety requirements

68. The development and update of key CFIA standards and manuals, monitoring product sampling, establishment inspections, surveillance and compliance lists, and other non-compliance actions are control measures expected to raise awareness amongst industry, of the fresh fruit and vegetables food safety requirements. The CFIA's intent is to develop a manual for the listing of requirements that the industry should be aware of.

Industry complies with requirements and adheres to good practices relevant to the Acts

69. Industry improved awareness subsequently leads to an industry that will assume more responsibilities toward the safety and compliance of fresh fruit and vegetables and one that adheres to food safety requirements and good practices relevant to the Acts described in the inputs section above. Industry compliance includes having a sound control system in place that includes important elements such as promoting compliance, identifying non-compliance and enforcing compliance.

Non-compliance enforcement activities occur to mitigate FFV food safety risk in the marketplace

70. When non-compliance actions do persist, enforcement activities are initiated. Enforcement activities include: warning letters, product detention, product removal from the marketplace and destruction of the product if necessary, issuance of a recall order when establishments do not cooperate, removal from Canada or disposal of the imported product, and prosecution(s). The FFVP has drafted an Enforcement Strategy which is currently under review and approbation by CFIA Enforcement and Investigation Services.

Consumers and partners have improved knowledge of fresh fruit and vegetables food safety

71. Establishment inspections, investigations and non-compliance actions contribute to the increase in food safety awareness of the industry. Also, information fact sheets for consumers, codes of practice, good manufacturing practices and good import practice for industry are expected to lead to industry awareness and compliance with FFVP's food safety requirements. Improved access to information for consumers and partners should improve knowledge of fresh fruit and vegetables food safety.

3.1.6 Intermediate Outcomes

72. All of the current and future activities (e.g., conduct product sampling ) conducted in FFVP, the outputs that they produce (e.g., product sampling reports and roll-ups) and the immediate outcomes (e.g., industry complies with requirements) that results are expected to lead to the following intermediate outcomes.

FFV Regulations reflect current international requirements and Canadian priorities

73. In Canada, the FFV regulatory approach is thorough and integrated such that the Canadian food safety system retains its current high standing on the world marketplace and that FFV stakeholders are actively engaged to provide safe FFV to the Canadian market.

74. With an enhanced compliance control system in place and better information sharing by FFVP on FFV food safety, Health Canada can develop and revise food safety standards, guidelines and policies related to fresh fruit and vegetables. Both organizations have the responsibility of ensuring the safety of food in Canada. Health Canada requires CFIA information related to FFV in order to develop food safety standards, policies and guidelines and in turn, CFIA will have access to these new and revised standards, policies and guidelines in a timely manner.

Strong stakeholders' confidence in the safety of FFV available in Canada

75. The totality of all of the above activities and outcomes related to fresh fruit and vegetables food safety will lead to consumers, industry and government being confident in the safety of FFV available in Canada. Overall, Canadians appear to be confident in the food safety system. Despite some food safety concerns, recent results indicate that this confidence has strengthened as a result of the way in which the Mad Cow Disease situation has been handled.18 Canadians were asked to name items that would make them feel better about the safety of food in Canada and suggestions included having more detailed food labels (34%), providing more information about food safety (10%), reducing/eliminating pesticide use (10%), and improving the regulatory system (8%)19.

Increased access to interprovincial and international FFV markets relative to pre-re-design of FFVP

76. The FFVP is seeking this outcome as part of its mandate to improve market access to Canadian FFV not related to food safety but rather quality grading. The rationale for this is that, increased industry awareness of food safety will lead to increased industry compliance which would allow more domestic and imported FFV products to be available for consumption. This in turn, would allow the public and industry to have increased access to diversified, nutritious and safe fresh fruit and vegetable products throughout the year.

Increased stakeholder implementation of FFV food safety practices relative to pre-re-designed FFVP

77. Increased industry awareness of and compliance with fresh fruit and vegetable requirements along with better information about FFV food safety provided to consumers and partners will lead consumers and industry to adopt effective FFV food safety practices. A recent study asked consumers about the food safety information available to them. Almost one quarter (23%) said they would like more information on food processing issues, such as food safety in general processing, and Escherichia coli and Salmonella spp. contamination. Supplying consumers' information on the processes and practices of establishments and individuals involved in food production would likely have a positive impact on their perceptions of food safety and quality, provided, of course, that the information demonstrates the positive actions taken to manage quality and safety risks.20

18 Consumer Perceptions of Food Quality and Safety. Agriculture & Agriculture Canada. Ipsos Reid Corporation, November 2004.
19 Public Security Monitor, Wave VIII, EKOS Research Associates, March 2005.
20 Consumer Perceptions of Food Quality and Safety. Prepared for Agriculture & Agriculture Canada. Ipsos Reid Corporation, November 2004.

3.1.7 Ultimate Outcome

78. All elements in the logic model (activities, outputs, immediate and intermediate outcomes) are expected to lead to the ultimate outcome: Protection for the people of Canada from preventable health risks related to fresh fruit and vegetables. Along with CFIA, there are several other organizations that contribute to this ultimate outcome, however, the FFVP can play a major role in achieving this outcome by enhancing its food safety activities and the resulting impacts.

3.2 Logic Model

Logic Model

4.0 Evaluation Matrix

79. The evaluation matrix outlines the key evaluation questions, along with indicators, data sources and methodologies to address the questions. The components of the evaluation matrix are defined as follows (refer to section 4.2 for table):

Evaluation Question: A question that needs to be answered in order to address the broader evaluation issue. The major purpose of the evaluation question is to provide data on the characteristics and merits of a program (see section 4.1 for more detail). The evaluation questions in the framework will explore CFIA's risk management strategies and food safety activities, and possible linkages with human health outcomes, as well as Health Canada's policies and standards.

Indicator Construct: Provides a description of one aspect of the evaluation issue or concept to be pursued, for example, outputs in a logic model.

Indicator Variable: A quantifiable indicator under an Indicator Construct, which is the means for obtaining concrete measurement information on the concept, issue, or indicator construct, of interest. These indicators provide a quantitative or qualitative measurement of the extent to which the program is achieving its outcomes. Indicator variables must be simple, measurable, accurate, reliable (over time), relevant (to the audience and to what is being addressed), and timely.

Data source: The source of the information used to address the evaluation question. In evaluation, several sources of data are used to answer the same question to ensure reliability.

Methodologies: The methods used to collect and analyse the information to answer the evaluation question (see section 4.3 for more detail).

4.1 Evaluation Questions

80. The FFVP logic model provides an outline of CFIA's activities and outputs of the program as well as the expected outcomes. This logic model was used to formulate the evaluation questions identified in four categories that are relevant to the assessment of domestic and imported fresh fruit and vegetables.

  • Rationale/Relevance: These questions focus on the relevance of the program or activities related to fresh fruit and vegetables, and their objectives related to CFIA's overall mandate. They ask whether the program or activities are still needed (e.g., to what extent do they continue to respond to federal and CFIA/ Health Canada priorities?). The questions also discuss the potential food safety risks and hazards which can affect human health.
  • Program Design: This focuses on how the design of the program supports its performance. Questions include food safety activities, outputs (things that are tangible) and outcomes (longer term developments), their ongoing management and design, and linkages with science based and risk management strategies.
  • Program Delivery: Questions consider food safety activities described in the design with a focus on the effectiveness of their implementation, (e.g., consistency of delivery with the Program design).
  • Outcomes: These questions address the intended outcomes and effects that are identified in the logic model, with a focus on health and safety components. The questions also explore the extent to which the program or activity is achieving its intended outcomes (how successful it is) and whether there are any unintended outcomes.

81. For the purposes of the evaluation questions in section 4.2:

  • FFVP staff includes: FFV inspectors, FFV Area Program Specialists, Inspection Managers, National Headquarters (HQ) Work plan Coordinator, National FFV Program Specialist, Import Chief.
  • Partners include: Area Import Coordinator, Import Service Centre representative, National Import Operations Manager, Food Safety Investigation Program, Office of Food Safety and Recall (OFSR), Food Microbiology and Chemical Evaluation Division (FMCED) and the Science Branch.
  • F/P/T partners include: Health Canada (HC), Canadian Border Services Agency (CBSA), Public Health Agency of Canada (PHAC) (National Microbiology Laboratory (NML)/ Laboratory for Foodborne Zoonoses (LFZ)), Pest Management Regulatory Agency (PMRA), Agriculture and Agri-Food Canada (AAFC), and Provincial/ Territorial counterparts.
  • Stakeholders include: consumers, wholesalers, retailers, producers, manufacturers, distribution centres and importers and their respective associations.

82. The following nine evaluation questions listed in the Evaluation Matrix Table that follows (section 4.2) are proposed as possible evaluation questions for a future FFVP evaluation. Before implementing the evaluation, the BFSA evaluation team will assess the currency of the following questions to identify which ones can best address the evaluation needs and may add others in response to issues that may have arisen after the completion of this framework.

4.2 Evaluation Matrix Table

Category Evaluation Question Indicator Construct Indicator Variable Data Source Methodologies
Relevance, rationale To what extent is there a legitimate and continuing need for the Fresh Fruit and Vegetables Program (FFVP) to conduct fresh fruit and vegetables food safety activities?
  • What is these role of the CFIA regarding food safety according to current legislation?
  • Are current FFV food safety standards and regulations appropriate to address FFV safety risk?

  • What are the current risks associated to FFV safety?
  • What are the emerging issues and trends associated to FFV safety?
  • Legitimate
  • The current FFV legislation is consistent with scientific evidence of health risk related to FFV
  • Legislation, with respect to fresh fruit and vegetables, food safety is appropriate, known and reflected in the mandate of the Program
  • Relevance/need of Federal presence in this capacity with current legislation
  • Sustained relevance of the Program with current needs

  • Scientific Basis
  • science and evidence based need for CFIA/FFVP to mitigate the health risks associated with FFVproducts exists
  • number of other countries using a similar science-based approach
  • Legislation
  • the Canadian food safety legislation provides CFIA the responsibility to implement an FFV food safety inspection program.All FFV related food safety legislation is known by document reviews and interviews, and reflected in the Program mandate.
  • % of CFIA personnel, partners and stakeholders satisfied with present FFV food safety legislation
  • % of CFIA personnel, partners and stakeholders satisfied those program address FFVfood safety hazards
  • Scientific Basis
  • number and type of health risks, emerging issues and trends associated with FFV, eg., reported incidences of outbreaks, illnesses and related deaths associated with consumption of FFV
  • number of scientific and stakeholder reviews on FFV food safety risks
  • number and type of risk-based prioritization methods for domestic and imported FFV commodities
  • number of international food safety system reviews undertaken to determine equivalency of similar science-based approach
  • Key Players
  • FFVP and partners staff (Headquarters, Areas, and Regions)
  • legal experts
  • Health Canada scientific experts
  • industry stakeholders (e.g., associations)
  • F/P/T partners
  • consumer stakeholders (e.g., associations)
  • academia

  • Documentation
  • documents that outline FFV food safety legislation, mandate, and FFV food safety risks.
  • other CFIA Program documents related to FFV, eg. FSIP
  • corporate documents
  • scientific literature
  • interdepartmental documents, eg., Health Risk Assessments (HRA)
  • CFIA informatics generated reports related to FFV, eg., Incident Management System (IMS)
  • Foodborne Illness Reports
  • projects specifications (i.e., sprouts)
  • FFV consumption statistics
  • foreign government FFV food safety strategies
  • Program document review and analysis
  • Scientific literature analysis
  • surveys
  • interviews
Design 1 To what extent does the FFV program structure (mandate, authorities, activities, roles and responsibilities) support achievement of its objectives of identifying and mitigating FFV risks?
  • Are the goals and objectives well known by all parties?
  • Do roles and responsibilities correspond to the authorities/mandate of the organizations involved and well known/reasonable?
  • Are their gaps/overlap among the partners/stakeholders?
  • How well are FFV food safety risks covered with regards to roles, responsibilities, gaps, overlap?
  • Goals and Objectives
  • FFVP and partner goals and objectives for domestic and import activities are understood, clear, and relevant

  • Roles and Responsibilities
  • clear and understood roles and responsibilities between FFVP (Program and Operation Branches) its partners including F/P/T governments and stakeholders to reduce gaps and duplication
  • Agreements
  • agreements between FFVP, partners and stakeholders exist and provide FFV food safety coverage and reduce gaps and duplication
  • level of FFV food safety contribution/collaboration from FFVP, its partners and stakeholders
  • Risk Mitigation
  • strategy to mitigate health hazards based on scientific risk assessments
  • evidence that FFVP design manages identified hazards/ risks associated with FFV to promote and enforce compliance
  • stakeholders and partners collaboration of risk mitigation strategies within their respective jurisdictions
  • Goals and Objectives
  • number and position of CFIA staff, partners and stakeholders that know their respective goals and objectives for FFVfood safety activities
  • extent to which Program's objectives identify which strategies and activities are necessary and sufficient to ensure FFV food safety
  • goals and objectives are reflected in the program activities and outcomes
  • Roles and Responsibilities
  • number and position of CFIA staff, partners and stakeholders that know their respective roles and responsibilities related to FFV food safety activities
  • % of partners and stakeholders satisfied that roles and responsibilities are clearly understood
  • descriptions of roles and responsibilities exist in program documentation and are clearly understood
  • Agreements
  • numbers and positions of CFIA staff, partners and stakeholders that know of agreements and coordinated strategies related to FFV food safety activities
  • number and type of partner and stakeholder agreements in the Programs' documents
  • number of consultations/ meetings to develop agreements with partners and stakeholders
  • improved participation of partners and stakeholders with agreements in place: number, frequency and type of input and feedback (includes ad hoc consultations)
  • number of existing gaps and duplications among all parties
  • level of FFVP, partners and stakeholders satisfaction
  • Risk Mitigation
  • number and types of science-based strategies developed based on risk assessments
  • FFVP design includes activities to mitigate FFV food safety hazards/risks
  • length of time to manage identified hazards/ risks
  • % of strategies to reduce incidences of FFV hazards that are adopted by industry
  • evidence of partners and stakeholders collaboration with risk mitigation tools and strategies
  • Key Players
  • FFVP and partners staff
  • Headquarters
  • Areas
  • evaluation and legal experts
  • HC experts
  • industry stakeholders (e.g., associations)
  • F/P/T partners
  • consumer stakeholders (e.g., associations)
  • academia
  • Documentation
  • documents that outline FFV food safety legislation, mandate, and FFV food safety risks.
  • corporate documents
  • FFV Program documents, e.g., work plans, Inspection Manual
  • other CFIA Program documents related to FFV, eg. FSIP
  • scientific literature
  • inter-departmental documents, eg., Foodborne Illness Response Protocol (FIORP), Memoranda of Understanding (MOUs), meeting minutes between FFVP, its partners and stakeholders
  • CFIA informatics generated reports related to FFV, eg., IMS
  • Foodborne Illness Reports
  • projects specifications (i.e., sprouts)
  • interviews
  • focus groups
  • surveys
  • document review and analysis
Design 2 To what extent is FFVP guided by a vision/strategic plan?
  • To what extent is there by-in from stakeholders and partners into FFVP's vision/strategic plan?
  • What tools, mechanisms and type of information/ data supports the strategic plan, design and management of FFV food safety activities? How are these updated?
  • Strategies Selected
  • extent to which the Program's design selects and incorporates clear, appropriate and effective strategies to address food safety risks
  • extent to which the Program has incorporated a sound transition strategy into new activities, i.e., quality to food safety focuses in establishment inspections
  • strategy to enhance overall industry responsibility exists
  • strategies to define the FFV industry profile (domestic establishments and importers)
  • Capacity to generate data
  • Capacity to input and produce internal food safety data
  • Capacity to capture feedback from partners and stakeholders
  • Enhance design and management
  • Evidence of appropriate design and management systems in place to enhance FFV program design, delivery and risk mitigation strategies related to FFV.
  • Systems identify Areas for improvement and incorporate food safety emerging issues, trend analyses, stakeholder and partner communication and performance measurement data to improve program design, management and delivery.
  • Strategies Selected
  • number and type of domestic and import control policies, guidelines and food safety standards (e.g., Maximum Residue Levels (MRL's), Colony Forming Unit (CFU), irradiation) are clear and useful to mitigate food safety risks.
  • number and type of establishment compliance food safety policies and guidelines are clear and useful to mitigate food safety risks.
  • national and Area work plan priorities are based on food safety risk
  • number and position of CFIA personnel, partners and stakeholders involved in the planning of selected strategies
  • % of industry that is aware of FFV risks and food safety requirements
  • number and quality of methods (e.g., environmental scans) used by the Program to define the FFV industry profile ensuring appropriate coverage
  • Capacity to Generate data
  • numbers and position of CFIA staff, partners and stakeholders that know the type of data they can access and generate
  • number, type and quality of food safety data and reports required for decision making
  • % of FFVP staff, partners and stakeholders generating reliable and valid food safety data related to FFV.
  • satisfaction by FFVP, partners and stakeholders that data gathering capacity meets design needs
  • % of staff trained on informatics systems (including a performance measurement framework)
  • informatics needs' assessment completed
  • number and type of informatics tools used for continuous improvement
  • number, frequency and type of input and feedback from partners and stakeholders
  • Enhance design and management
  • number and type of evaluations conducted to determine program consistency and appropriateness with design and delivery
  • number and type of evaluations conducted to determine reliable and consistent data entry throughout the Program
  • number and type of information systems to track, analyse and report on FFV food safety activities
  • information systems are designed to be compatible with each other for data management purposes
  • number of new and updated existing policies, procedures and standards for program design and management improvements
  • number and type of performance indicators developed and designed to enhance FFV food safety
  • established and interim performance measurement systems are in place and used to improve the program management, design and delivery
  • number and types of trend analyses done to enhance program delivery
  • Key Players
  • FFVP and partners staff
  • Headquarters
  • Areas
  • evaluation experts
  • HC experts
  • industry stakeholders (e.g., associations)
  • F/P/T partners
  • consumer stakeholders (e.g., associations)
  • academia
  • CFIA informatics
  • Documentation
  • documents that outline CFIA Program designs include risk mitigation strategies related to FFV; eg., Good Agricultural Practices (GAP), sprouts project specifications
  • corporate documents
  • FFV Program documents, e.g., work plans, Inspection Manual, HRAs
  • other CFIA Program documents related to FFV, eg. FSIP
  • trend analyses of inspection results
  • scientific literature
  • inter-departmental, partner and stakeholder documents and information exchange, eg., FRIOP, MOU, meeting minutes
  • CFIA informatics generated reports related to FFV, eg., IMS
  • Foodborne Illness Reports
  • cost-analysis of FFV food safety strategies
  • FFV Industry Profile
  • CFIA informatics generated reports related to FFV, eg., IMS, national and regional databases
  • interviews
  • focus groups
  • surveys
  • document review and analysis
Delivery How are the role and responsibilities of the Agency currently being operationalized and implemented?
  • What is CFIA staff capacity to apply relevant procedures, directives and standards as well as implement FFV food safety activities?
  • What are the mechanisms in place to improve program design and delivery of FFV food safety activities?
  • Delivery consistent with Design
  • training, sampling, industry and establishment inspections and follow-up/enforcement activities are conducted as designed to meet work plan requirements
  • evidence that adjustments to program delivery are made as needed and are effective as a result of performance measurement activities and trend analyses
  • evidence of collaboration between the FFVP, its F/P/T partners and stakeholders
  • Role and Responsibilities
  • adherence to roles and responsibilities by FFVP (Program and Operation Branches) and its partners to reduce gaps and duplication
  • Staff Capacity
  • training covers full range of FFV food safety activities in addition to support activities
  • Program Improvement
  • evidence indicating effective management of:
    - domestic and imported products
    - domestic establishments
    - importers,
    for FFVfood safety monitoring and enforcement activities including recalls, emergencies, incidents
  • Delivery consistent with Design
  • number and type of food safety training activities conducted versus planned
  • number and type of samples collected, establishments inspected and follow-up actions on non-compliant products and establishments
  • evidence of timely implementation of activities as designed
  • % CFIA staff conducting sampling and industry inspections as designed
  • % CFIA staff satisfied with sampling and industry inspections implemented work plans
  • % CFIA staff, partners and stakeholders satisfied that enforcement activities are taken and emerging issues are controlled in a timely manner.
  • consistent application and tracking of FFV food safety activities amongst Area/ Regions
  • number, frequency and type of input and feedback from partners and stakeholders
  • level of CFIA staff, partner and stakeholder satisfaction of FFV food safety activities, i.e., on collaboration and coverage of FFV food safety
  • Roles and Responsibilities
  • percentage of CFIA staff and partners that know and adhere to their respective roles and responsibilities related to FFV food safety activities
  • Staff Capacity
  • % of qualified/ certified CFIA staff that meet work description requirements related to FFV food safety activities
  • number and type of policies, guidelines, documents and courses related to food safety training
  • % increase in knowledge and retention of FFV food safety training
  • % staff satisfied with training activities, i.e., types and level of training
  • number, type and frequency of input and feedback from staff on training initiatives/ courses.
  • Program Improvement
  • Number and type of sample and establishment monitoring activities, compliance and enforcement actions
  • number and type of modifications to work plans and program adjustments to address food safety activities and emerging issues
  • food safety criteria to measure compliance have been identified
  • an integrated informatics system in place to collect, record and analyse data to confirm improvements in overall product and industry compliance trends
  • level of satisfaction for the Agency's effective management of recalls, emergencies and incidents amongst:
    - FFVP staff
    - F/P/T partners
    - industry
    - consumers
  • extent to which FFVP contributes and facilitates requirements from CFIA's OFSR and other partners
  • Key Players
  • FFVP and partners staff
  • Headquarters
  • Areas
  • consumer stakeholders
  • industry stakeholders
  • F/P/T partners
  • Health Canada experts
  • Documentation
  • FFV Program documents, e.g., work plans, Inspection Manual, HRAs, recalls
  • other CFIA Program documents related to FFV, eg. FSIP
  • trend analyses of inspection results
  • inter-departmental documents including information exchange, informatics generated reports related to FFV, eg., IMS national and regional databases
  • documents that outline CFIA Program designs include risk mitigation strategies related to FFV; eg., GAP, and sprouts project specifications
  • corporate documents
  • exchange of information with partners and stakeholders (e.g., ad hoc consultations and established agreements)
  • related FFV training documents and manuals
  • training / certification listings and dates
  • training needs assessment -linked with various work descriptions
  • evaluation reports at the end of training
  • staff performance evaluation reports
  • interviews
  • focus groups
  • surveys
  • document review and analysis
Outcome 1 How has the industry's level of awareness for FFV food safety requirements and standards changed?
  • What are the resulting impacts of the industry's awareness on FFV safety?
  • To what extent are these changes and impacts attributable to CFIA's FFV food safety activities?
  • Awareness
  • evidence that CFIA activities promote industry awareness of and voluntary compliance to FFV food safety requirements
  • Awareness
  • extent to which programs provide documented guidance including management of food safety emergencies and incidents to stakeholders
  • number and type of FFV food safety collaborations and consultations between CFIA and the industry, e.g., technical reviews, HRA's, presentations, joint training initiatives
  • number and type of establishments that implemented FFV food safety controls through voluntary implementation of:

  • - EOL (end-of-line/ finished product sampling) and adherence to established Maximum residual limits (MRLs), Colony forming unites (CFUs)
    - HACCP like systems
    - Good Importing Practices, Good Agricultural Practices, On-Farm Food Safety Program
  • Key Players
  • FFVP and partners staff
  • Headquarters
  • Areas
  • Health Canada experts
  • industry stakeholders (e.g., associations)
  • consumer stakeholders (e.g., associations)
  • F/P/T partners
  • academia
  • Documentation
  • FFVP program documents and other Programs documents related to FFV
  • FFV food safety standards, policy and program
  • scientific literature
  • Health Risk Assessments
  • partner agreements minutes and feedback
  • industry training initiatives, technical reviews and consultations
  • media reports and reviews
  • compliance and enforcement reports
  • reviews and evaluation reports
  • interviews
  • focus groups
  • surveys
  • document review and analysis
  • Baseline study
Outcome 2 To what extent does industry comply with FFVP requirements and good practices and to what extent can compliance be attributed to CFIA's FFV food safety activities?
  • What are the trends in FFV safety non compliance since the FFVP re-design?
  • Compliance Status
  • compliance data for FFV with food safety requirements is collected and analysed
  • extent to which FFV is implicated in recalls, incidents and foodborne illnesses

  • Compliance Status
  • food safety criteria to measure FFV compliance have been identified
  • food safety compliance data that are collected on an ongoing basis determine:

  • - product compliance rates
    - establishment compliance rates
  • recalls
  • foodborne illnesses

  • Key Players
  • FFVP and partners staff
  • Headquarters
  • Areas
  • evaluation experts
  • Health Canada experts
  • industry stakeholders (e.g., association)
  • F/P/T partners
  • consumer stakeholders (e.g., association)
  • academia
  • Documentation
  • legislation (FDA, CAPA, CPLA, FFVR)
  • performance measurement framework
  • FFV food safety policy and program
  • Science Committee Meeting minutes
  • Health Risk Assessments
  • recall information/ IMS database reports
  • stakeholder agreements and minutes
  • compliance and enforcement reports
  • interviews
  • focus groups
  • surveys
  • document review and analysis
Outcome 3 To what extent are stakeholders, including consumers, informed and confident in the safety of FFV available in Canada? How is this attributable to CFIA's FFV food safety activities?
  • How has information access related to FFV safety changed after FFVP re-design?
  • How have consumers and stakeholders confidence changed after the re-design of FFVP in FFV safety?
  • Stakeholder confidence
  • Stakeholders and consumer are confident in the safety of FFV available in Canada
  • Stakeholder and consumer trust that CFIA's FFVP effectively mitigates health risks related to FFV
  • Changes in stakeholder confidence are attributable to FFVP re-design
  • Evidence that any such change can be attributed to changes made to the FFVP
  • Stakeholder confidence
  • % FFV consumption per capita. Rates are maintained or increase from year to year
  • consumers are exposed to new and existing best food safety practices as they arise
  • % of consumers that are confident in the safety of FFV available in Canada
  • % of consumers changing FFV food safety practices
  • % of consumers that are optimistic or positive about the federal government's role in the safety of FFV available in Canada
  • % of consumers satisfied with the level of FFV food safety in Canada
  • Key Players
  • FFVP and partners staff

  • - Headquarters
    - Areas
  • industry stakeholders (e.g., associations)
  • F/P/T partners
  • consumer stakeholders (e.g., associations)
  • Health Canada experts
  • Media Reports
  • website news bulletins
  • television reports
  • radio announcements
  • Documentation
  • expert/scientific opinions
  • Public Health Agency of Canada annual reports (Foodborne Illness Reports)
  • Statistic Canada Reports
  • external Research Studies (e.g., EKOS, Ipsos Reid)
  • interviews
  • focus groups
  • surveys
  • document review and analysis
  • analysis of media reports
  • Baseline study
Outcome 4 To what extent are Canadian regulatory requirements integrated for FFV food safety activities?
  • Are their gaps/ overlaps in Federal and Provincial departments/agencies related to FFV food safety activities?
  • Are the Canadian requirements related to FFV food safety consistent with international requirements?
  • Evidence of consistency across federal agencies in FFV food health and safety regulatory requirements (e.g., HC-CFIA)
  • Demonstration of consistency of federal and international (e.g., Codex) requirements
  • Identification of gaps and overlaps between Federal departments and agencies related to FFV food safety activities
  • Identification of gaps and overlaps between Federal and Provincial jurisdiction
  • Analysis of CFIA, HC, Codex and other FFV related regulations among Canada's trading partners (including provinces)
  • % of stakeholders (Codex representatives, provincial government representatives) regarding congruence among regulatory requirements
  • Number/ Evidence of active mechanisms and processes to coordinate regulatory review and agreement across governments/agencies
  • number of technical reviews, working groups, presentations and linkages between CFIA, partners and stakeholders
  • Analysis of Canadian regulatory requirements (Federal and Provincial) to conclude on gaps and overlaps in relation to FFV safety
  • regulations (review of regulations)
  • interviews
  • files/program's data
  • interviews
  • surveys
  • focus groups
  • document review and analysis
Outcome 5 To what extent has the occurrence of preventable health incidents related to FFV changed?
  • Are FFV available in Canada safe and to what extent can this be attributed to the CFIA FFV food safety activities?
  • extent to which program's activities reduce exposure to risks associated with FFV and the number of foodborne illnesses related to FFV
  • identification of the number and type of food safety hazards associated with FFV to establish risk management strategies
  • decrease in chemical and microbial levels and food safety incidences in FFV
  • decrease in exposure too microbial, chemical, irradiation and extraneous matter contamination risks associated with FFV
  • number of domestic and imported FFV products implicated in foodborne illnesses
   

4.3 Evaluation Methodologies

83. The Evaluation Matrix Table outlines the evaluation questions, specific indicators, data sources and the methodologies that may be used to answer the evaluation questions. The matrix table provides a broad range of methodologies, both qualitative and quantitative in nature to be selected (based on information availability and feasibility) by the BFSA assessment team as part of a future evaluation of FFVP. Specific methodologies are chosen by the BFSA team in consultation with CFIA, as their priorities and available resources may influence the choice of methodologies to be used in the future evaluation. In collaboration with the FFVP, the BFSA team also used some of these evaluation methodologies in developing the FFVP logic model.

84. A range of methodologies could be undertaken as part of a future evaluation and may include:

  • Document Review: This includes the review of documents that would be useful in exploring the evaluation questions. These documents could include the FFV program manuals, program output and data records, performance measurement reports, training manuals, and other pertinent information from the Agency. It also includes legislation, the CFIA Import Policy, FFV Products policy, FFV inspection procedures and FFV guidelines and standards that apply to the FFVP's import activities, including CODEX documents. The assessment team may also review documents related to the management of food safety risks which could include sampling plans, guidelines, laboratory methods, and risk assessments.
  • Environmental Scans: This includes conducting scans on food safety hazards, emerging issues, and foodborne illnesses. Environmental scans can also obtain information from internet sites on the regulation of FFV imports.
  • Literature reviews: This includes reviewing academic and scientific papers pertinent to the assessment.
  • Interviews: The following groups have been identified for interviews: CFIA HQ's Managers, Area Managers, partners and stakeholders. The interviews at Headquarter levels are primarily with those staff who works in the FFV Program, the Import Sector, and the Office of Food Safety and Recall. It also includes consultation with HQ's Technical Specialists and the National Coordinator of Operations. In the Areas, interviews will be conducted with Import Service Centre Managers, Area Operations Coordinators, FFV Program Network Specialists, Regional Inspection Managers and Inspectors (including multi commodity inspectors with FFV responsibilities), and Program Network Directors and Chiefs. External to CFIA, interviews will be conducted with Health Canada, Agency partners with responsibilities for import control such as Canada Border Services Agency (CBSA) and Department of Foreign Affairs and International Trade, academia, FFV importers and industry including their associations, and possibly Provincial representatives.
  • Focus Groups: This involves interview techniques and open discussion with a selected small group of individuals. The small group can be selected from within one of the larger groups or can be in any number of combinations from the following: CFIA HQ Managers; FFV specialists and inspectors; and industry representatives.
  • Trend Analysis: This involves the assessment team's review of existing CFIA's trend analyses on such issues as the compliance rate of domestic and imported FFV, or the team conducting a trend analysis exercises to obtain information.
  • Case Studies: The assessment team may conduct some case studies in order to obtain the domestic and importers' perspectives of CFIA's FFVP and related activities and to determine their understanding of documented guidelines such as Good Manufacturing and Good Import Practices. Focussed questions will be posed to these stakeholders and their responses will be summarised.
  • Surveys: Industry and FFVP staff survey questions will be developed in collaboration with FFVP management and may include assistance of services from a recognized authority in survey design (e.g., from Health Canada or Statistics Canada). The method of survey delivery can be through mail or electronic means such as e-mail. Information gathered from the surveys will be used to answer questions related to program rationale / relevance, design, and impacts /success.
  • Review of Databases: This involves the examination of any database information that is available and would benefit the assessment team in answering the evaluation questions. Some of the databases that are important to the team include: the Incidence Management System database which contains incident and recall information and the Import Retrieval System. The team will also review information from the disease surveillance database that is maintained by the Public Health Agency of Canada.
  • Baseline Study: This involves the comparison of historical (longitudinal) data which has been collected before the re-design of the FFVP, with data collected after its re-design.

5.0 Evaluation Challenges

85. The following section outlines some challenges that may have an impact on the future evaluation of the FFVP. Many of these challenges relate to food safety data gaps and potential barriers or limitations in assessing the impacts or successes of the FFVP.

5.1 Food Continuum and Legislative Mandate of the Program

86. As shown in Appendix B, CFIA's roles and responsibilities primarily fall in the part of the FFV production continuum that deals with the FFV processing, import and distribution establishments. Although the pre- and post- processing steps are not under the jurisdiction of the CFIA, the Agency's desires to develop strategies to mitigate risks and reduce existing gaps in the FFV food safety continuum may result in a review and/or intervention of these areas.

Challenge:
Due to the multi jurisdictional nature of the FFV food continuum, the evaluation will require a thorough understanding of the dynamics between CFIA and its stakeholders.

5.2 Food Safety Risks

87. An evaluation of the FFVP will take into consideration both quantitative and qualitative data to reach food safety and evidence-based conclusions. Several factors impact on FFV food safety. The food safety risks vary between the FFV commodities, the growing environment and the domestic and imported production methods, all of which may require specific interventions to reduce the presence and level of hazards (e.g., on-farm food safety programs (OFFSP)).

Challenge:
Specific risk profiles for each FFV commodities and processing methods are currently incomplete and this is a challenge for the Program planning. Thus, the evaluation challenge would be in analysing the Program's rationale, logic, and successes in mitigating food safety risks.

5.3 FFVP Food Safety Data

88. Currently, the FFVP is mainly focussed on the collection of product quality inspection data based on Canadian and international trade requirements. Food safety data is limited to test results of sample analyses, complaint investigation reports, and ongoing specific food safety issues. In the re-design of FFVP, the proposed food safety activities include establishment inspections and increased collaboration between the industry and F/P/T partners. These changes would generate useful food safety data for evaluating the Program's effectiveness. In addition, the Program is establishing performance measurements as part of a CFIA wide performance framework implementation. This performance data will not only support food safety trend analyses and management decision-making, but will also provide useful evaluation information on the program's delivery and impacts.

Challenge:
Limited food safety data is currently available on chemical and microbiological testing. It is unlikely that an evaluation team could formulate useful food safety conclusions using the current level of data.

5.4 Capacity to Conclude on Industry Adherence to Food Safety Standards

89. Currently, the FFVP has limited documentation on the overall industry (processing establishments, distribution centres, wholesalers and importers) adherence to FFV food safety requirements. The Program's information is limited to non compliant products follow up investigations in domestic establishments. This results in data that does not represent the entire FFV industry.

Challenge:
It would be difficult to measure the impact of the Program without a complete and comprehensive set of food safety data that represents all of the FFV industry within CFIA's jurisdiction.

6.0 Evaluation Strategy

90. The following section proposes considerations for an eventual evaluation of the effectiveness of FFVP.

91. As FFVP is planning a re-design (more focus on food safety), BFSA recommends that an evaluation be conducted one to two years after the implementation of the re-design plan. The HC- CFIA Assessment Advisory Committee will review the feasibility of conducting the FFVP evaluation and the date of the evaluation in the next planning cycle. The evaluation's objective would include providing feedback to senior management about the impact of the re-design on the safety of FFVs available in Canada. Such evaluation could then utilize performance data collected by the Program and would focus among other things on comparing pre and post re-design food safety data and measure early outcomes of the re-designed FFV Program.

92. Findings and recommendations are then expected to be used to provide CFIA with evidence-based advice and guidance to help improve their food safety-related activities.

Appendix A - Terms of Reference

Purpose
1. The purpose of this terms of reference is to outline the work to be performed by the Bureau of Food Safety Assessment (BFSA), Health Canada, and to describe the scope and approach of the evaluation project of CFIA's food safety activities related to domestic and imported fresh fruit and vegetables.

Objective of Evaluation Project
2. The objective of this evaluation project is to develop an evaluation framework, which is to include logic models for both the current Fresh Fruit and Vegetable Program (FFVP) and the proposed / possible future FFVP food safety activities. Through the development of the evaluation framework this evaluation project will contribute to the re-design of the FFVP, a possible future evaluation of a re-designed FFVP, and provide advice and feedback to HC on its food safety policies and standards.

Background
3. In September 2003, the joint HC-CFIA Food Safety Assessment Advisory Committee identified domestic fresh fruit and vegetables as an evaluation priority for 2004 (see the original proposal in Appendix 1). BFSA began pre-evaluation work and environmental scanning in May 2004, paused during the summer due to mutual program resource pressures, and restarted work in October 2004.

4. In February 2005, CFIA requested that the Advisory Committee review the proposed evaluation due to an upcoming re-design of the FFVP. The Advisory Committee met on March 1, 2005 and decided that BFSA should adjust the evaluation project to take the re-design into account. The Advisory Committee also decided that the assessment should provide timely advice in the re-design of CFIA's domestic and imported fresh fruit and vegetable's food safety activities.

Program Description
5. Under the Canada Agricultural Products Act and the Consumer Packaging and Labelling Act, the FFVP ensures that fresh produce, which is imported, exported and marketed domestically under federal standards is safe, meets standards, and is packaged and labelled to avoid fraud and marketed in an orderly fashion.

6. The major activities undertaken by CFIA under this program include: sampling of fresh fruit and vegetables, inspecting of products for standardized container sizes, verifying of industry grading accuracy to established standards, and inspecting at points of destinations for defects in condition and appropriateness of the grade.

7. Outside of the FFVP, CFIA also conducts food safety activities related to domestic and imported fresh fruit and vegetables such as recalls, food safety investigations and enforcement actions, as provided under the Food and Drugs Act.

Scope of the Evaluation Project
8. The evaluation project will consider the relevance, design, implementation and food safety outcomes of the FFVP as well as all other CFIA activities related to domestic and imported fresh fruit and vegetables that are outside of the FFVP. Possible food safety outcomes for this project could include: effectiveness of processing and monitoring controls for microbial hazards (pathogens and viruses); effectiveness of processing and monitoring controls for chemical hazards (natural toxins, pesticides, food additives, heavy metals); and extent of CFIA's participation in on-farm food safety initiatives (as mentioned in Appendix 1).

9. Fresh fruit and vegetables for this evaluation project include: fresh fruit, fresh vegetables, fresh mushrooms, fresh herbs, sprouts and fresh-cut produce consumed in Canada.21 This includes irradiated and fumigated fruits and vegetables, and fruits and vegetables contained in modified atmosphere containers / packages. It does not include products which are processed (such as nuts) or where ingredients such as vinegar are used.

Methodology
10. The methodology for this project includes the following activities, which can be useful for program re-design and planning as well as future evaluation and performance measurement:

  • document and literature reviews and interviews with CFIA staff from Headquarters, Areas and relevant stakeholders (e.g., provinces, industry associations, etc.) as required;
  • the development of logic models (for current and proposed FFV food safety activities) in a collaborative workshop with HC and CFIA representatives;
  • the development of an evaluation framework, which will include evaluation questions and issues, and performance measurement criteria and indicators (see appendix 2);
  • the provision of advice, guidance and recommendations for FFVP and other food safety activities outside of FFVP that are related to fresh fruit and vegetables; and
  • drafting of the final evaluation framework.

Roles and Responsibilities

11. The evaluation project will be conducted by BFSA in consultation with CFIA and relevant stakeholders as required. A joint HC-CFIA Food Safety Assessment Committee for this project was established in the fall of 2004. The committee facilitates the process during the course of the evaluation project and provides appropriate representation and input (see Appendix 3 for list of committee members).

12. The specific activities of the CFIA - HC Food Safety Assessment Committee include:

  • reviewing the terms of reference (which includes the scope and approach of the evaluation project);
  • supporting the evaluation project during its duration by expediting the process, providing guidance and resolving outstanding issues encountered;
  • recommending the creation of working groups as needed;
  • reviewing the evaluation framework, including the evaluation questions and strategy, and determining the appropriate evaluation option if several are presented; and
  • recommending the approval of the logic model and evaluation framework to the Advisory Committee.

13. Given CFIA's re-design of the FFVP, the Advisory Committee will also review and approve the terms of reference for this evaluation project.

Time Frames and Deliverables
14. Work will commence upon acceptance of the terms of reference by the Assessment Committee and the Advisory Committee. An evaluation framework and logic models will be delivered in 2006, for approval by both committees and for publishing on Health Canada's website.

Meetings and Travel
15. BFSA will develop a Work plan for interviews and travel plans, and will coordinate with CFIA. These plans will allow CFIA to identify headquarters and regional area contacts and make appropriate arrangements well in advance.

Approved

Johanna N. Read
Co-chair Assessment Committee
Interim Director, Bureau of Food Safety Assessment
Health Canada

René Cardinal
Co-chair Assessment Committee
Acting National Manager, Fresh Products Section
Chief, Fresh Produce Safety
CFIA

Paul Mayers
Co-chair Advisory Committee
Director General
Food Directorate
Health Products and Food Branch
Health Canada

Tom Beaver
Co-chair Advisory Committee
Executive Director
Corporate Planning, Reporting and Accountability
CFIA

21 The project will include all CFIA activities related to fresh fruit and vegetables. These activities include both intra- and inter- provincial, which are covered under the provisions of the Food and Drugs Act.

Appendix B - CFIA-FFVP Jurisdiction Flowchart

Simplified flow of production, marketing and distribution of fresh fruit and vegetables from primary production to consumption in Canada and abroad, including Canadian jurisdiction over certain areas and activities under the F&D Act and the CAP Act.

Agricultural Inputs include fertilizers, manure, compost, pesticides, seed, soil, etc.
Primary Production also includes enclosed environments such as green house production, sprouts and mushrooms
____ Source: Fresh Fruit and Vegetable Food Safety Program, CFIA 2006

CFIA-FFVP Jurisdiction Flowchart

Appendix C - Fresh Fruit and Vegetables Assessment Committee Members

Health Canada

  • Ashwani Wadhera, Director, Bureau of Food Safety Assessment, FD, HPFB (co-chair)
  • Lucien Comeau, Project Lead, Bureau of Food Safety Assessment, FD, HPFB
  • Theresa Paolasini, Regulatory Policy Officer, Bureau of Food Policy Integration, FD, HPFB
  • Hélène Couture, Section Head, Policy and Method Development, Bureau of Microbial Hazards, FD, HPFB
  • Bernard Fortier, Scientific Evaluator and Program Officer, Bureau of Chemical Safety, FD, HPFB
  • Sheila Dubois; Bureau of Biostatistics and Computer Applications, FD, HPFB
  • Julie Daunais; Departmental Performance Measurement and Evaluation Directorate
  • Louise Croteau; Health Evaluation, Pest Management Regulatory Agency

Canadian Food Inspection Agency

  • Réne Cardinal; Acting National Manager Fresh Products Section, Fresh Fruit and Vegetable Program, Food of Plant Origin Division (co-chair)
  • Jean-Pierre Hanchay; Regional Director, Operations, Quebec Area
  • Bashir Manji; Acting Director, Food of Plant Origin Division, OVPP,
  • Jean Kamanzi; Director, Food Microbiology and Chemical Evaluation, Food Safety
  • Eli Niedert; National Manager, Chemical Residues, Food Microbiology and Chemical Evaluation, Food Safety
  • Michel Laurendeau, Director, Audit, Evaluation and Risk Oversight (AERO)
  • Ann Cooper, Evaluation Manager, Corporate Program Evaluation, AERO

Appendix D - Health Canada Evaluation Team Members

  • Lucien Comeau
  • France Lacroix
  • Yasmin Hasham
  • Jason Pelletier
  • Michael Andre

Appendix E - Glossary of Terms

CFIA Areas - with its headquarters in the National Capital Region, the CFIA is organized into four Operations Areas (Atlantic, Quebec, Ontario, and Western) that are subdivided into 18 regional offices, 185 field offices (including border points of entry), and 408 offices in non-government establishments, such as processing facilities.

CODEX Alimentarius Commission - a subsidiary body of the United Nations World Health Organization (WHO) and the Food & Agriculture Organization (FAO) of the United Nations.

Consumer Complaint - see definition for incident.

Effectiveness - the extent to which a program achieves its objectives. In the case of the FFVP, effectiveness includes the extent to which the Program's activities are contributing to the level of compliance of FFV in Canada.

Enforcement Actions - the FFVP provides a compliance regime which includes a range of enforcement activities that are employed when necessary to ensure compliance with federal regulations and standards. These include warning letters, product detentions and suspension. These activities may lead to product recalls or prosecutions, which are coordinated respectively by CFIA's Office of Food Safety and Recall (OFSR) and Enforcement and Investigations Services (EIS).

Food Emergency - is broadly defined as any situation involving or potentially involving food which may pose a health and safety concern to humans. Emergencies usually involve significant resources and require the coordination of a timely and/or extraordinary operational response.

Hazard - According to the CODEX Alimentarius Commission of WHO and FAO, a hazard is a biological, chemical or physical agent in, or condition of, food with the potential to cause an adverse health effect.

Incident - an issue that requires documentation and/or investigation into food safety, quality, labelling or fraud concerns within the CFIA's Programs' priorities and mandate. An incident can be generated by a variety of sources including consumer complaints.

Incident Management System (IMS) - CFIA's informatics system for tracking information regarding incidents which could include consumer complaints, emergencies and recalls.

Recall - action where an establishment removes from further sale or use, or corrects, a marketed product that contravenes legislation administered and/or enforced by CFIA.

Appendix F - List of Acronyms

AERO
Audit, Evaluation and Risk Office (formerly CPRA - Corporate Program Review and Audit)
AAFC
Agriculture and Agri-Food Canada
AIRS
Automated Import Reference System
BFSA
Bureau of Food Safety Assessment
CAPA
Canada Agricultural Product Act
CBSA
Canada Border Services Agency
CFIA
Canadian Food Inspection Agency
CFU
Colony Forming Unit
CPLA
Consumer Packaging and Labelling Act
EOL
end-of-line
FDAR
Food and Drugs Act and Regulations
FFV
Fresh Fruit and Vegetables
FFVR
Fresh Fruit and Vegetables Regulations
FFVP
Fresh Fruit and Vegetable Program
FIORP
Foodborne Illness Response Protocol
FMCED
Food Microbiology and Chemistry Evaluation Division
FSIP
Food Safety Investigation Program
FTE
Full Time Equivalent
GAPs
Good Agricultural Practices
GIPs
Good Importing Practices
GMO
Genetically Modified Organisms
GMPs
Good Manufacturing Practices
HACCP
Hazard Analysis Critical Control Points
HC
Health Canada
HRAs
Health Risk Assessments
HQ
Headquarters
IMS
Incident Management System
LFZ
Laboratory for Foodborne Zoonoses
LSTS
Laboratory Sample Tracking System
MAPAQ
ministère de l'Agriculture, des Pêcheries et de l'Alimentation du Québec
MOU
Memorandum of Understanding
MRLs
Maximum Residue Limits
NML
National Microbiology Laboratory
NTI
National Training Initiative
NTU
National Training Unit
OFFSPs
On Farm Food Safety Programs
OFSR
Office of Food Safety and Recall
OMAF
Ontario Minister of Agriculture and Food (Minister of Agriculture, Food and Rural Affairs)
OVPP
Office of the Vice President, Programs
PHAC
Public Health Agency of Canada
PMRA
Pest Management Regulatory Agency
RTE
Ready-To-Eat
SENASICA
Servicio Nacional de Sanidad, Inocuidad y Calidad Agroalimentaria
USFDA
United States Food and Drug Administration
Date Modified:
2007-07-10
Report a problem or mistake on this page
Please select all that apply:

Thank you for your help!

You will not receive a reply. For enquiries, contact us.

Date modified: