3. Evaluation design

3.1. Purpose and scope

The evaluation was identified in the 2015 Departmental Risk-Based Audit and Evaluation Plan and conducted in order to inform future program funding decisions as current funding ends in March 2017. The evaluation also responds to the requirements of the Financial Administration Act and the 2009 Treasury Board Policy on Evaluation^{Footnote 1}, to evaluate all ongoing programs of grants and contributions and direct program spending at least once every five years.

The evaluation covers the period from 2012–2013 to 2015–2016, which includes four of the five years of Phase II of the LWBI^{Footnote 2}. The scope of the evaluation covers activities related to each of the program’s three pillars: science (research, modelling and monitoring), stewardship and transboundary partnerships.

The evaluation was informed by a previous evaluation of the LWBI completed in 2011–2012 when the program was a component of the Freshwater Programs under the Action Plan for Clean Water^{Footnote 3}, as well as recent evaluations of other ECCC water management and freshwater programs. There were no recent audits by the Commissioner of the Environment and Sustainable Development (CESD) or internal ECCC audits directly pertinent to this evaluation.

3.2. Evaluation approach and methodology

The findings presented in this document are based on five data collection methodologies. Evidence drawn from these methods informed findings and conclusions:^{Footnote 4}

• Document review. A review of documents was conducted that included foundational documents, descriptive program information, departmental and Government of Canada publications related to policy and priorities, and other internal strategic, operational planning and past evaluations. Program performance measurement-related data were also reviewed.
• G&C file review. A review of 29 LWBSF project final reports was completed to examine project activities, outputs, outcomes and lessons learned. The project reports reviewed included all Phase II projects that were completed as of March 31, 2016.
• Key informant interviews. In total, 30 individuals were interviewed for the evaluation. The distribution of interviews by respondent category is shown below:
• Internal ECCC program representatives and management (n=10);
• External stakeholders (n=20)^{Footnote 5}.
• On-line survey of LWBSF funding applicants. An online survey of all organizations that applied for LWBSF funding was conducted, including both funded and never-funded applicants. The survey gathered applicants’ perspectives on awareness of the program, satisfaction with the eligibility criteria and application process, partnerships, adequacy and appropriateness of performance reporting, the impact of securing (or not securing) LWBSF funding, ongoing impacts of the project and sustainability. In total, 74 organizations were contacted by email to participate and 45 organizations completed the survey (61% response rate). Of the organizations that completed the survey, 28 were funding recipients and 17 were never-funded applicants. This data collection method addressed evaluation questions related to performance.
• Case studies. Three case studies were conducted to provide additional insight related to the relevance and performance of the LWBI:
• funding of the LWRC for the operation of the MV Namao research vessel;
• the LWBIN; and
• a comparative analysis of the LWBI and three other ECCC freshwater funding programs.
Case study methodologies included a document review and key informant interviews (n=1 or n=2 for the LWRC and LWBIN case studies).
Case study interviews were conducted separately and in addition to the key informant interviews described previously.

3.3. Challenges and limitations

Challenges experienced during the conduct of the evaluation, as well as the related limitations and strategies used to mitigate their impact, are outlined below.

• The water quality issues affecting Lake Winnipeg have developed over the past several decades and require long-term solutions. While science has enhanced the understanding of the state of the lake and contributing sources of nutrients, intended outcomes such as a reduction in the magnitude and extent of harmful algal blooms and restoration of the ecological integrity of Lake Winnipeg are complex and only expected to be measurable in the long term, beyond the evaluation timeframe. Where available, proxy measures were used to assess intermediate outcomes and key external factors that impact on the achievement of longer-term outcomes were identified.
• Measures of some intended outcomes for the program such as nutrient reductions are not precise due to the method of calculation used and challenges in measuring nutrient reductions that occur as a result of individual actions. Additionally, intended outcomes related to the setting of nutrient objectives are the responsibility of other jurisdictions and, therefore, beyond the control of the program. These issues have been identified and explained in the findings where appropriate.
• The survey invitation was emailed to all 74 unique organizations that applied for LWBSF funding, including those who received funding and those never funded. Although a reasonable response rate of 61% was achieved, the small total number of completed surveys limits additional analysis beyond disaggregating the responses of funded and unfunded applicants.

• Previous
• Table of contents
• Next