Evaluation of Water Resource Management and Use program: chapter 6

3. Evaluation Design

3.1 Purpose and Scope

The purpose of this evaluation was to assess the relevance and performance of the WRMU program. The evaluation is identified in the 2013 Risk-based Audit and Evaluation Plan, which was approved by the Deputy Minister. The evaluation was conducted to meet the 2009 Treasury Board Policy on Evaluation requirement that all direct program spending, excluding grants and contributions, be evaluated every five years.^Footnote20

The evaluation covers the five-year timeframe from 2009-2010 to 2013-2014; however, it excludes from consideration any activities which were no longer supported as of 2013-2014. The following activities were scoped out of the evaluation:

• Activities previously conducted by ESB with regard to the promotion of water conservation demand management, which were discontinued in 2012-2013;
• EC’s hosting and support of UNEP GEMS/Water, as this activity terminated at the end of 2013-2014 and there was no requirement identified on the part of management for a “lessons learned” assessment of this work; and
• G&Cs that have been re-aligned to a different program (e.g., Lake Winnipeg Fund).

The primary focus of the evaluation is on EC’s activities related to the water management boards. It should be noted that this is not an evaluation of the water management boards themselves. However, in those instances when it is not possible to separate the activities of EC from the work of the boards to which it contributes, the performance of the boards is also examined, as appropriate.

The evaluation also includes a limited assessment of EC’s activities related to the IRIA. This limited focus on IRIA activities was deemed appropriate given the low materiality of EC’s work in this area^Footnote21 and the regular oversight and accountability to which this area is subject through the preparation of publicly available annual reports.

3.2 Evaluation Approach and Methodology

The methodological approach and level of effort for this evaluation were determined using a risk-based approach. The following data collection methodologies were developed to adequately address the evaluation issues and questions. Evidence gathered was then analyzed to develop overall findings and conclusions.^Footnote22

• Document Review
• On-line Survey of External Water Board Members and IJC Board Advisors
  • Data was collected between March 17 and April 4, 2014.
  • Survey invitations were sent to 130 non-EC stakeholders; 51 stakeholders completed the survey, for a response rate of 39.2%.
  • Several of the 51 respondents responded regarding more than one water management board and, as a result, 91 detailed board-specific responses were received.
• Key Informant Interviews
  • In-depth interviews were conducted with 31 key informants, including 20 internal stakeholders (EC senior management and program managers) and 11 external stakeholders (non-EC board members and IJC representatives).

A more detailed description of the methodologies can be found in Annex 2. 

3.3 Limitations

The individual water management boards include both domestic and international boards and vary in terms of their specific mandates and issues. Additionally, almost every branch in the Department is involved in delivery of the activities of this program, and EC’s roles and commitments also vary by board. This posed a challenge for the conduct of the evaluation in terms of ensuring that all relevant perspectives and contexts were considered, while still providing an appropriate overall assessment of the program.

In order to address this challenge and minimize any associated limitations this might introduce, the evaluation committee that was formed to provide guidance for the evaluation included individuals representing a cross-section of organizational groups, boards, and roles. This diversity was also taken into consideration in the data collection when determining samples for the key informant interviews and the survey. Finally, the report distinguishes between the various contexts where appropriate, while attempting to present a more generalized assessment.