3. Evaluation design

3.1. Scope

The evaluation addresses the issues of relevance and performance (including effectiveness, efficiency and economy) of the Great Lakes sub-program, and in particular the program activities under GLNI. Less effort was expended on the evaluation of GLAP, as it had previously been evaluated. The assessment of the GLSRP was more limited and focused on progress to date because, as noted, it was too early to assess outcomes. The evaluation responds to the requirements of the Financial Administration Act and the Treasury Board of Canada Policy on Evaluation to evaluate all ongoing programs of grants and contributions and direct program spending at least once every five years.

The evaluation primarily covered the five-year period from 2010–2011 to 2014–2015, and it also included some updated information for 2015–2016. It considered and built on previous evaluations of the Great Lakes Action Plan IV (2010) and Freshwater Programs under the Action Plan for Clean Water (2011).

3.2. Evaluation Approach and Methodology

The research methods^{Footnote 1} utilized to collect evidence for the evaluation are briefly described in this section, and further details are available in Annex C.

Document and Literature Review: Documentation and literature were reviewed to gather evidence to help address each of the evaluation questions. This included program documentation (e.g., State of the Great Lakes reports, COA Progress Reports), departmental and federal government documents (e.g., ECCC Departmental Performance Reports (DPRs), previous evaluations, Federal Budgets), and selected research literature.

Review of Financial and Performance Data: Financial and available performance data were reviewed to contribute to the assessment of evaluation questions related to the program’s efficiency/economy (e.g., administrative costs associated with the disbursal of Gs&Cs) and effectiveness.

Grants and Contributions Project File Review: A review was conducted of the files of a random sample of 40 completed GLAP projects (of a total of 167 completed projects funded in the evaluation timeframe).

Key Informant Interviews: In order to obtain feedback related to all of the evaluation questions, in-depth interviews were conducted with a sample of 43 key informants, including representatives of different types of relevant stakeholders, both internal and external to ECCC.

Case Studies: Two case studies were conducted in order to examine two specific aspects of the Great Lakes program: (1) implementation of the 2012 Canada–US Great Lakes Water Quality Agreement, with a focus on the effectiveness of the governance structure and processes; and (2) the scientific process by which phosphorus targets are set for Lake Erie.

Online Survey of Stakeholders: An online survey was conducted to obtain quantifiable responses from a large group of stakeholders, beyond those consulted in the key informant interviews. Of 193 stakeholders invited to participate, responses were received from 108 stakeholders for a response rate of 56%. Survey respondents included representatives of various program-related committees, including the Great Lakes Executive Committee, the COA Management Committee, GLWQA Annex Subcommittees and Extended Subcommittees, and BUI Working Groups.

3.3. Challenges and Limitations

Challenges encountered in the conduct of this evaluation, as well as the associated limitations and strategies used to mitigate their impact, are outlined below.

• Performance measurement information was incomplete, as the sub-program does not currently have an overall performance measurement strategy with corresponding data collection in place for each of the outcome areas assessed in this evaluation. In addition, it was difficult to obtain outcome information from the final reports of funded projects in the file review, because the project reports focused primarily on activities and outputs. For these reasons, the information on the achievement of outcomes was incomplete. To the extent possible, gaps in the evidence of outcomes were filled by the available departmental Performance Measurement Framework data, other high-level performance information (e.g., in reports related to the GLWQA and IJC), and evidence obtained from other sources such as the document review and key informant interviews.
• Where there was insufficient evidence from documentation and performance data, the observations of stakeholders consulted in interviews and the survey were used. There is a possibility of bias stemming from the selection of stakeholders, given that it was not feasible to interview each and every relevant stakeholder. This risk was mitigated by carefully selecting the sample of 43 interviewees to ensure that all relevant perspectives, both within and outside ECCC, were adequately covered by knowledgeable respondents from each interviewee group; asking interviewees to provide evidence or concrete examples to support the views they expressed; and corroborating the interview findings with reliable evidence from documentation and data where possible. Similarly, the online survey included a range of 108 relevant stakeholders from various perspectives, and the overall response rate of 56% was strong.
• It was difficult to measure the achievement of the longer-term outcomes of the Great Lakes program related to restoration and remediation in the lakes/AOCs and to assess the program’s contribution to observed outcomes. The ecological issues in the Great Lakes are complex and evolving. Therefore, remediating or restoring many of these areas will occur over the long term and will be influenced by factors outside the control of the program. In the absence of up-to-date empirical or performance data, the evaluation evidence related to some longer-term outcomes of the program relied heavily on qualitative findings such as the observations of key informants and the views of survey respondents. No conclusions were drawn solely on the basis of one line of evidence, and all available evidence for each outcome was triangulated to arrive at a conclusion. In cases where there simply was insufficient evidence or it was premature to draw a conclusion, that is stated.