ARCHIVED - Integrated Strategy on Healthy Living and Chronic Diseases - Community Based Programming Functional Component

Methodology

2.1 Methodology/Approach

Four lines of evidence were used in this evaluation: a review of performance monitoring data collected by the 2006-07 funded projects using pilot Program Evaluation Reporting Tool (PERT) data; a document review; a review of the Early Outcome Evaluation of Diabetes Community-Based Projects Report, and interviews with three program representatives. As illustrated in Table 2, most lines of evidence include both qualitative and quantitative data sources.

The four lines of evidence used in this evaluation are further described below.

Review of Performance Monitoring Data for 2006-07 Funded Projects

The Project Evaluation and Reporting Tool (PERT) is a self-reporting questionnaire developed to help monitor and document the effectiveness of projects funded by PHAC, as well as to assess their early impacts.  It was piloted on diabetes-funded projects (except in Quebec)Footnote 1 in fiscal year 2006-07.  PERT contains questions related to processes undertaken, project outputs and outcomes. Funded projects completed the tool twice over the funding cycle (six months after project initiation and again upon completion) (Public Health Agency of Canada, 2007e; Public Health Agency of Canada, 2007d; Public Health Agency of Canada, 2008d).

Of 41 funded projects in 2006-07, 35 used PERT or the Quebec reporting tool. Data submitted were reviewed and analyzed to assess project-level success and progress in achieving outcomes. It should be noted that no data were collected using the PERT for projects funded in 2005-06.^{Footnote 2}

Document Review

A thorough review of documents and other relevant research pieces was conducted. The following documents were reviewed:

• Background documents, including the fully-Elaborated RMAF and various reports and presentations on the ISHLCD and CBP Functional Component;
• Documents related to 2006-07 funded projects;
• Administrative data relating to solicitations for Gs&Cs and priority-setting; and
• Other data sources, including reports from the First and Second Implementation Reviews of the ISHLCD (January 2007 and March 2008) (Performance Management Network Inc., 2007; Public Health Agency of Canada, 2008c).

In addition, documents generated from the Diabetes Policy Review,Footnote3 undertaken by a five-member Expert Panel appointed by the Minister of Health in October 2007, were reviewed to extract information relevant to the issues addressed in this evaluation (Diabetes Policy Review Committee chaired by the Honourable Brian Peckford, 2008a; Diabetes Policy Review Committee chaired by the Honourable Brian Peckford, 2008b).  The information included a review of program documentation and recent literature, an international environmental scan of diabetes strategies and approaches, and a series of stakeholder consultations undertaken between April and June 2008 with the following groups:

• Stakeholders who were involved with the original CDS (nine);Footnote 4
• Diabetes researchers (seven);
• Canadians living with diabetes (30);
• P/T governments (seven provinces and one territory);
• Health professionals (11);
• Diabetes educators (52);
• Research-based pharmaceutical companies (two); and
• Health-related non-governmental organizations (NGOs) (three).

A complete list of documents reviewed can be found in Annex D.

Early Outcome Evaluation of Diabetes Community-Based Projects (2006-07)

A sample of 17 funded projects participated in the evaluation of early project outcomes. This was designed to assess the success and progress in achieving outcomes by projects funded under Diabetes CBP. Projects were asked to complete evaluation-relevant questions from PERT. In addition, using a common framework developed by the consultant, each project completed a specific evaluation report. Data collection occurred in the six months immediately following the end of PHAC-funded project activities. The report synthesized findings using the common framework to identify areas where early outcomes could be reported.

The 17 participating projects were selected based on the following eligibility and selection criteria:

• Eligibility criteria: potential to be replicated; organizational capacity to complete additional evaluation requirements; recipient met reporting requirements as defined by the grant agreement; good working relationship between the funded project and PHAC; and demonstration of engagement of stakeholders and target population.
• Selection criteria: representation from diverse key target populations; a combination of rural and urban settings; a combination of diabetes prevention and diabetes management; and projects from each region.

The 17 participating projects consisted of 15 grants funded in 2006-07 and two contributions funded in the Quebec region in 2007-08.  These projects represented national (four projects) and regional (13 projects) community-based projects, with budgets ranging from $45K to $300K.  Additional funds were provided, through upward amendments, to these participating projects to support their capacity to participate in the evaluation exercise.

Interviews with Program Representatives

To solicit feedback on the evaluation issues, interviews were held with the Accountable Director of Diabetes, the CCDPC Manager of Diabetes Coordination in National Office, and the Regional Manager who oversees implementation of Diabetes CBP at the regional level. Each interview was less than two hours in length, and questions focused on relevance, roles/responsibilities, coordination, design and delivery, and success/progress. The interviews took place in mid-November 2008; responses were captured by a note-taker.

2.2 Methodological Limitations

There are potential biases with the evaluation approaches used in this report.  An objective third-party evaluator conducted the evaluation; however, staff members working within the Diabetes CBP, individuals who would have a vested interest in the program, provided data for the evaluation.  A potential self-reporting bias is important to acknowledge, as it is difficult to compensate for this.

Other limitations are related to the challenge of consistently applying common protocols for evaluation tool administration across such diverse community projects.  Attempts were made to counteract this by training staff, developing standardized methodologies suited to the community environment, and providing on-going support throughout data collection.

Key limitations resulting from each line of evidence used for this evaluation are described below.

Project Evaluation and Reporting Tool (PERT) Data Limitations

For this evaluation, performance monitoring data from 2006-07 projects were used to develop a profile of funded projects, as well as to assess the success and progress related to project outputs and early outcomes.

Data available for this evaluation were limited in scope due to a lack of systematically collected data related to Diabetes CBP.  Given that PERT was piloted in 2006-07, no data were collected using a standardized reporting tool for projects funded in 2005-06.  Furthermore, data for 2007-08 funded projects, which used a revised (and improved) PERT, were not available at the time this evaluation was conducted.  As a result, the formative evaluation used project data from a single fiscal year.

Due to the delay in funding approval for the solicitation process for Gs&Cs, the PHAC National Office encouraged the use of grants over contributions to allow funded projects to carry funding forward beyond the fiscal year end. PERT reporting is not mandatory for grant-funded projects and, as a result, data were not collected for projects that did not report - projects funded in Quebec (five), the Northern region (two), British Columbia (two) and Ontario (one) did not complete the PERT reporting for 2006-07.

The Quebec region provided data using a similar reporting tool for the cumulative 12-month project funding period for four of its projects. These data were incorporated as part of the 2006-07 PERT data analysis as well as the Early Outcome Evaluation.

Another limitation relates to the validity of data collected through PERT.  Although PHAC staff worked with funded projects to complete the six month and end of project PERT, some inconsistencies remained.  For instance, some data submitted by 2006-07 funded projects may include double counts due to repetition of questions at both reporting periods. The extent of any possible duplication is not known as there was no formal mechanism to verify the data. Hence, the combined data sets likely inflate some project-level results to an unknown extent.

Document Review Limitations

This evaluation did not include consultations with external stakeholders.  Instead, relevant feedback provided by stakeholders consulted through the Diabetes Policy Review were reviewed and used to support the assessment of the evaluation (Diabetes Policy Review Committee chaired by the Honourable Brian Peckford, 2008a; Diabetes Policy Review Committee chaired by the Honourable Brian Peckford, 2008b).  These stakeholders were selected by an Expert Panel to support their assessment of the renewed CDS; they were not asked to provide feedback on the specific issues of this evaluation.  Therefore, the feedback from these stakeholders may not be completely relevant to Diabetes CBP within ISHLCD, which is the focus of the present evaluation.

Early Outcome Evaluation Limitations

The Early Outcome Evaluation examined data on project results reported from 17 of the 41 projects.  The 17 projects do not, however, constitute a representative sample of the 41 projects funded in 2006-07.  Using the selection criteria as described, these projects likely demonstrate greater success in achieving outcomes than a true representative sample. Given the non-representative nature of the sample, the results of the Early Outcome Evaluation cannot be generalized to other projects.

Early Outcome Evaluation results indicated differing approaches among the evaluations undertaken by projects, and a varying degree of rigour in methods. Some of the method limitations from participating projects may be a result of projects being asked to participate in the evaluation exercise midway through their project implementation. In addition, there are potential self-reporting biases inherent within the approaches used by projects.

Interview Data Limitations

This formative evaluation involved only limited interview data to support the assessment of each evaluation issue. In-depth interviews were conducted with three program representatives, and as a result, views and opinions presented in this evaluation report may not reflect those of the majority of staff involved with delivery of Diabetes CBP.

Despite the fact that the majority of diabetes program staff were not included in the interview process, the opinions of key managers responsible for Diabetes CBP – that is, the Accountable Director of Diabetes, the Manager of Diabetes Coordination, and the Regional Diabetes Manager - were captured and included in the overall assessment of the evaluation issues.