Annex C – Details on evaluation methodology

The research methods^{Footnote 1} utilized to collect evidence for the evaluation are described below.

Document and Literature Review

Documentation and literature were reviewed to gather evidence to help address each of the evaluation questions. This involved a review of program documentation and reports, for example, the State of the Great Lakes reports on environmental indicators, updates on Lakewide Action and Management Plans, and COA Progress Reports. In addition, relevant departmental and federal government documents were reviewed, including ECCC Reports on Plans and Priorities (RPPs) and Departmental Performance Reports (DPRs); previous evaluations; Speeches from the Throne; and Federal Budgets. A review of selected literature focused on relevant studies related to the ongoing need for the program as well as some aspects of the efficiency analysis. The evidence was summarized, by evaluation question, in a template.

Review of Financial and Performance Data

Financial, project and performance data were reviewed to contribute to the assessment of evaluation questions related to the program’s efficiency/economy and effectiveness. This included a review of financial information (to assess any gaps between budgets and expenditures, as well as the administrative costs associated with disbursal of Gs&Cs), and performance information available in updates, progress reports, DPRs and the 2007–2012 COA database.

Grants and Contributions Project File Review

This component of the methodology involved a review of a sample of 40 project files. All of these projects were funded under the Great Lakes Sustainability Fund (GLSF) as part of the Great Lakes Action Plan (GLAP). This included a review of approval forms, contribution agreements, project activity reporting, financial files (e.g., requests for payment, cash flow statements, recipients’ accounting of expenditures), annual and/or interim reports, and final project reports. Data gathered from project files were recorded in a standard file review template.

In the evaluation timeframe, there were 167 funded projects which had been completed. A random sample of completed projects was selected, stratified by criteria such as dollar value of contribution funding and fiscal year when funding began, in order to ensure that a representative number of different types of projects was included in the review.

Key Informant Interviews

In order to obtain feedback related to all of the evaluation questions, in-depth interviews were conducted with key informants. A representative number of different types of relevant stakeholders, both internal and external to ECCC, were sampled from a larger list of interview candidates compiled with the assistance of program representatives. A total of 43 key informant interviews were completed with respondents in the following categories:

• ECCC program managers, staff and internal partners (n=14)
• Federal government partners in other departments (n=3)
• Provincial government partners (n=4)
• Project proponents and unfunded applicants (n=12)
• Other external stakeholders, including representatives of the IJC, conservation authorities, First Nations organizations and academic experts (n=10)

Interview guides consisting of open-ended questions were designed for each major type of key informant. Interviews were conducted by telephone or in person (in the National Capital Region) and were 45 to 90 minutes in duration, depending on the type of respondent. Findings from the interviews were captured in templates organized by evaluation question.

Case Studies

Case studies were conducted for an in-depth examination of two specific aspects of the Great Lakes program:

1. implementation of the 2012 Canada–US Great Lakes Water Quality Agreement, with a focus on the effectiveness of the governance structure and processes; and
2. the scientific process by which phosphorus targets are set for Lake Erie. The methodology for each case study involved a small number of interviews with stakeholders, a review of documentation and data, and preparation of a brief report.

Online Survey of Stakeholders

An online survey was conducted to obtain quantifiable responses from a large group of stakeholders, beyond those consulted in the key informant interviews. A survey questionnaire, consisting primarily of closed-ended questions, was developed to assess a number of evaluation questions related to relevance and performance. All stakeholders on lists provided by the program (excluding those consulted in interviews) were sent the questionnaire. Of 193 stakeholders invited to participate, responses were received from 108 stakeholders for a response rate of 55.9%.^{Footnote 2} For some survey questions, such as those on the achievement of outcomes for a specific program, only a sub-set of these stakeholders provided responses (i.e., those with sufficient knowledge and experience to respond, based on their role related to the Great Lakes program).

Survey respondents included representatives of various program-related committees, including the Great Lakes Executive Committee, COA Management Committee, GLWQA Annex Subcommittees and Extended Subcommittees, and BUI Working Groups. The distribution of the 10 respondents is as follows:

• ECCC (n=16)
• Other federal government departments (n=13)
• Provincial and regional governments (n=20)
• US federal government (n=12)
• US state governments (n=18)
• Other external stakeholders, including NGOs, international organizations and First Nations organizations (n=29)

Analysis of the survey data included the computation of overall descriptive statistics by question and cross-tabulations to assess differences by types of stakeholder organizations.