Evaluation of Water Resource Management and Use program: chapter 11

Annex 2: Evaluation Methodology

Document Review

Relevant documentation related to EC’s involvement in the water boards was reviewed, and the key points relating to the evaluation questions were identified and summarized. Examples of documents reviewed include:

• 2013 evaluation of the Hydrological Service and Water Survey Program, recent CESD audits related to water monitoring
• Program work plans, budgets and reporting presentations
• Program overview presentations or strategic documents
• Agendas and records of decision from EC water committee meetings, including relevant presentations and documents
• Agreements with provincial and territorial partners, IJC
• Websites for the various water boards and the IJC
• Policy and legislative documents
• Canada Water Act annual reports
• International River Improvements Act annual reports
• Departmental Performance Reports and Reports on Plans and Priorities
• Speeches from the Throne, budget documents

A summary worksheet was prepared to organize and cross reference key points in the documents to the evaluation questions as well as highlight any issues for follow-up in key informant interviews. Additional documents were identified and reviewed as work progressed on the evaluation, especially as a by-product of key informant interviews.

On-line Survey of External Water Board Members and IJC Board Advisors

An on-line survey of non-EC board members and IJC board liaisons was conducted as part of the evaluation. The survey was sent to all current non-EC members participating on domestic and international water boards as well as IJC engineering advisors and liaisons for the boards. The survey questionnaire was designed to capture feedback on the efficiency and effectiveness of both the secretariat services provided by EC and the performance related to EC’s board and committee membership. The survey primarily focused on the last three years, to place a greater emphasis on the most current years in the study period and to ease the burden on respondents. The survey also served to capture performance data relative to the client satisfaction indicator identified in the 2013-14 Performance Measurement Framework, as referenced in section 2.5. The recently designed survey of the National Administrators Table (for the Water Survey of Canada) was reviewed in the development of the questionnaire to identify any common questions that would benefit from being asked as part of the evaluation. The survey sample, including contact information, was developed from the board membership lists posted on the websites of the various water boards. The sample was reviewed and sorted to identify any individuals who participate on more than one board. The questionnaire design accommodated this possibility by asking respondents about their involvement in different water boards separately, to a maximum of three boards. In the event of any one individual being involved in more than three boards, the questionnaire was programmed to randomly select three water boards.

During the survey design, an effort was made to limit the length of the questionnaire by only including those questions that were important to addressing the evaluation issues. Minimizing respondent burden was important in order to maximize the response rate and because a small subset of individuals was also contacted to participate in a follow-up interview. The on-line survey was conducted prior to the key informant interviews so that issues identified in the survey could be explored in greater depth in the interviews.

Key Informant Interviews

In-depth key informant interviews offered the advantages of providing informed opinion and observations on the evaluation questions and assisting in the interpretation and understanding of qualitative and quantitative data from other lines of enquiry.

A total of 31 key informant interviews were conducted, with a little more than half of the interviews being conducted with internal stakeholders (i.e., EC employees) and the remainder with external stakeholders. The interviews provided insight on all evaluation questions, but had a particular focus on the questions related to the achievement of outcomes and efficiency and economy. The categories of respondents and the estimated number of interviews per category are shown below:

• EC senior management (DG, RDG, ADM): n = 3
• EC program managers, staff involved in the provision of secretariat services and support to the boards, and staff participating on boards: n = 17
• This included a mix of individuals from the different branches within the Department that are involved in this work, those involved in international and domestic boards, and those who participate as chairs, co-chairs or active board or related committee or task team members.   
• External water board members: n = 7
• This included external board members on international and domestic boards from various government jurisdictions and the private sector, e.g., the United States Army Corps of Engineers, provincial government representatives, power companies, and other federal government departments. Interviews with board members supplemented data collected in the survey of water board members described above.
• IJC representatives and staff: n = 4
• This included IJC secretaries and technical advisors (3 Canada, 1 US).

A customized, open-ended guide was developed for each category of interviewee. For interviews with EC staff, there were several individuals who fell into both the board member and program manager categories. As such, only one interview guide was designed for these two types of interviewees. Questions were asked or skipped as appropriate, depending on the individual respondent’s role and experience with the WRMU program. Interviewees received a copy of the interview guide in advance of the interview, to allow them to review and reflect on the questions. Notes from the interviews were organized by interview question/evaluation issue and interviewee category for analytic purposes. This facilitated the identification of themes and trends in responses, as well as any notable similarities or differences in the attitudes and opinions of different sub-groups of respondents.