ARCHIVED - Evaluation of the C-EnterNet Program Final Report
The methodology for the C-EnterNet Program evaluation was developed during an evaluation planning phase completed before the evaluation. Evaluation planning was undertaken in consultation with the C-EnterNet team, and with input from the C-EnterNet Advisory Committee. The following section outlines the evaluation issues and questions, data collection methods, and methodological considerations for the evaluation.
The C-EnterNet Program evaluation examined issues relating to program relevance, design and delivery, success, and cost-effectiveness/alternatives. Table 4 provides a summary of the evaluation issues and questions addressed in the evaluation. Please see Annex A for the complete evaluation matrix, which also includes specific indicators and methodologies for each evaluation question.
The C-EnterNet evaluation included the use of multiple lines of evidence and complementary research methods to help ensure the reliability of information and data collected. The following data collection methods were used to gather data for the evaluation:
- key stakeholder interviews;
- a document review;
- a survey of C-EnterNet information recipients;
- a review of administrative and financial data;
- a survey of partners to examine costs of the program; and
- a focus group with staff of the ROWPH.
Each of these methods is described in more detail as follows.
A total of 25 interviews were completed for the evaluation. Interviews were undertaken with members of the C-EnterNet team, PHAC senior management, members of the ROWPH Steering Committee, members of the C-EnterNet Advisory Committee, and relevant experts. The number of interviews completed for each interview group is illustrated in Table 5.
|Interview Group||Number of Interviews Conducted|
|C-EnterNet team members||3|
|PHAC senior management||3|
|ROWPH Steering Committee members||4|
|C-EnterNet Advisory Committee members||12|
Different interview guides were used for each of the interview groups (see Annex B for a matrix of the interview questions used). Interviews were conducted both in-person (during a site visit to Guelph and Waterloo, as well as in the National Capital Region) and by telephone. Interviews were entered into a database for subsequent analysis.
The following types of documentation were reviewed during the evaluation:
- Corporate, accountability and political documents: including PHAC’s Performance Measurement Framework and Program Activity Architecture, Departmental Performance Reports and Reports on Plans and Priorities. Other documents included Speeches from the Throne and the Government of Canada’s Science & Technology Strategy, among others.
- Operational documents: including the C-EnterNet business case, business plan, and previous reviews of the program.
- Program outputs: including annual and accomplishment reports, newsletters, and information on activities related to source attribution completed by the program.
- Academic and “grey” literature: including information on the costs borne by Canadian society from enteric disease, trends in pathogens, and other background documentation identified during the planning phase of the evaluation.
Annex C contains a full list of documents reviewed.
During the review, findings from documents were entered into a matrix, organized by evaluation issue. The document review was primarily used to inform an assessment of the relevance and the design and delivery of the program. It also provided information related to program success and cost-effectiveness/alternatives.
The C-EnterNet Program maintains a distribution list of individuals who are sent C-EnterNet documents/outputs, including annual reports and newsletters. Using this distribution list, the evaluation team conducted an online survey of C-EnterNet information recipients.
A survey questionnaire was developed to address the evaluation questions identified in the evaluation matrix. GCS undertook a pre-test of the survey questionnaire, in both English and French, with seven respondents from a pre-determined group of individuals identified by the C-EnterNet team. Minor revisions to the survey questionnaire wording were undertaken based on the results of the pre-test.
Potential respondents were sent an e-mail invitation inviting them to participate in the online survey. An e-mail invitation was sent to 284 potential respondents. The online survey form remained active from June 29th to August 14th, 2009. During this time, GCS also sent out reminder e-mails at two intervals to those who had not completed the survey.
In total, 102 respondents completed the survey, representing a response rate of 35.9%. Based on these numbers, the overall margin of error for the survey is ±7.8%, 19 times out of 20. It should be noted that the number of survey responses did not allow for extensive analysis of the survey results by particular respondent groups (e.g., results from federal government respondents compared to provincial government respondents).
Two major types of administrative data were analyzed during the evaluation: program data related to program activities/outputs and program financial data.
Program activity/outputs data analysis included: an examination of the number and type of presentations, training and workshops completed by project team members; the number and nature of partnerships; and the types of information inquiries received by C-EnterNet.
Financial information from the C-EnterNet Program was analyzed to assess the total costs of the program, as well as the specific costs of different areas of the program, including administrative versus other operating costs, and costs related to each of the program components. Costing was undertaken according to the general approach outlined in the Treasury Board Guide to Costing.
Financial data was obtained from C-EnterNet management. As available data from fiscal year 2006-2007 was the most detailed, and little change in program budget was noted in recent years, 2006-2007 became the focus of the costing analysis.
A short survey of partners and other agencies with whom C-EnterNet collaborates was undertaken to collect data to estimate the value of leveraged and in-kind contributions of other agencies to the C-EnterNet Program. These data helped address the value for money provided by C-EnterNet through establishing an estimate of cost-savings realized by C-EnterNet through collaborative activities.
The evaluation team worked with C-EnterNet to finalize a list of 19 partners and other agencies that have provided in-kind support to the program for which they were not provided financial compensation. In-kind support was often provided by partners in return for access to C-EnterNet Program data.
Each of these organizations was contacted by email to ask for their participation in a short e-mail survey. To obtain maximum response to the survey, telephone follow-up was undertaken by the evaluation team with those who did not complete the survey. In total, GCS received nine responses. These responses do, however, represent those partners identified by the C-EnterNet team as providing the most significant portion of in-kind support to the program.
GCS moderated a focus group with seven ROWPH staff members involved in activities in support of the C-EnterNet Program. Participants included an epidemiologist, a planner and five public health inspectors. The focus group was used to examine whether, and in what ways, C-EnterNet has increased public health capacity, including what types of impacts the program has had on ROWPH staff and the region. A moderator’s guide was developed in advance of the focus group to ensure that the session addressed all relevant evaluation questions. The moderator’s guide can be found in Annex D.
As in all evaluations, there are methodological limitations and considerations that should be noted.
The impact of the C-EnterNet Program must be considered within the context of the pilot nature of the program and its recent implementation.
C-EnterNet’s program design envisions a series of surveillance sites located across Canada. Multiple sentinel sites are intended to allow for results to be extrapolated across all regions of Canada. At the time of the evaluation, the program included only one sentinel site. Consequently, the program could not reasonably be expected to fully achieve its expected outcomes on a pan-Canadian level. However, the need for additional sentinel sites was examined as part of the evaluation.
Similarly, an important purpose of the data collected by the C-EnterNet Program is to examine trends over time in both human cases of enteric disease, and in pathogens from retail food, animal and surface water sources. The program only released its first results of its surveillance activities in 2006, and has only produced annual reports for three years. In addition, some surveillance activities only commenced more recently, such as produce sampling. Trends over time are only now beginning to be identified.
Evaluation is descriptive, rather than comparative, in nature.
It was beyond the scope of the evaluation to empirically assess the program’s design, delivery, success, and cost-effectiveness in comparison to similar programs, in Canada or internationally. The evaluation approach is, therefore, descriptive, rather than comparative, in nature.
The extent of response bias in the survey of information recipients is not known.
It was not possible to undertake non-response bias analysis for the survey of C-EnterNet information recipients. As a result, the degree to which the responses of the survey represent those of the entire population of information recipients is not known.
Despite these considerations, GCS has designed the evaluation to use multiple lines of evidence (e.g., interviews, surveys, administrative data, documentation, and a focus group), thus strengthening the reliability and validity of the evaluation results.
- Date modified: