Annex E – Details on Evaluation Methodology
The research methodsFootnote 36 utilized to collect evidence for the evaluation are described below.
Documentary evidence consisted primarily of documentation that spoke to the intent, design and policy rationale for the UTCs. The documents reviewed included departmental documents describing the UTCs (e.g., Terms and Conditions for each umbrella authority, Guide to Grants and Contributions at ECCC, ECCC Departmental Action Plan for Gs&Cs Reform), federal government documents (e.g., Action Plan to Reform the Administration of Grant and Contribution Programs), and previous audits and evaluations (i.e., a 2013 internal audit of Gs&Cs and a 2009 evaluation of ECCC’s former class Gs&Cs). Findings from the documents were captured in a standard template. The document review provided insights regarding evaluation questions related to the relevance, design and efficiency of the UTCs.
Administrative Data and File Review
The Gs&Cs database developed and implemented by ECCC’s Corporate Services and Finance Branch to monitor the distribution of funding agreements under the UTCs currently contains administrative data collected as part of the overall management of the UTCs. This data includes the date of project approval; the project’s stated purpose at the time of funding; the name of the umbrella authority; the name of the responsible ECCC manager; the region; and the overall funding allocation. As such, the information was used to provide key descriptive information on the UTCs, including expenditures by program (see Annex B).
The Gs&Cs database is limited to the stated intended goals of the agreements.Footnote 37 In order to assess the degree to which intended project goals were achieved, the evaluation also examined a sample of 85 project files. The file review sample was selected from a total of 635 projects completed in the timeframe from 2010–2011 to 2014–2015. The sample was stratified by relevant factors such as fiscal year when funding began, dollar value of the contribution and type of funding recipient, and it included a larger number of projects for the most heavily used UTC on Biodiversity – Wildlife and Habitat.
The distribution of the 85 project files across the six UTCs is as follows:
- Biodiversity – Wildlife and Habitat (n=32)
- Water Resources (n=9)
- Sustainable Ecosystems (n=10)
- Weather and Environmental Services for Canadians (n=11)
- Substances and Waste Management (n=13)
- Climate Change and Clean Air (n=10)
Data from the files were collected using a standard file review template. Key documents reviewed were the contribution agreement and the final report for each project. The primary focus of the file review was to assess the extent to which the projects funded under the UTCs met intended project goals within expected timelines, were delivered efficiently (e.g., resources leveraged from project partners), and contributed to the UTC expected results.
Key Informant Interviews
Key informant interviews were used to gather in-depth information related to all of the evaluation questions. A total of 17 key informant interviews were conducted with representatives from the following two groups:
- Departmental program managers with experience using one or more of the UTCs (n=15). A representative sample of managers for these interviews was selected utilizing Corporate Services and Finance Branch’s Gs&Cs database which contains managers’ names. Efforts were made to interview managers who had experience with larger numbers of agreements and/or agreements of a higher dollar value. Of 19 managers invited, 15 were able to participate in an interview. All six UTCs were represented in these interviews.
- Representatives of Corporate Services and Finance Branch involved in coordinating the management of the umbrella contributions (n=2). Of three representatives invited, two were available for an interview.
Interviews were undertaken in person in the National Capital Region or over the telephone, depending on the availability and preference of the respondent, using open-ended interview guides (one for each of the above respondent groups). Interview findings were summarized by evaluation question in a template.
An online survey of funding recipientsFootnote 38 was conducted to assess a number of evaluation questions related to performance. In particular, the survey assessed recipients’ views on ECCC’s delivery of the UTCs (e.g., the project application, funding and reporting processes) and the contribution of funded projects to environmental outcomes, both short-term project-level outcomes and UTC expected results.
A total of 493 projects were funded under all six UTCs over the four-year period from 2010–2011 to 2013–2014. For all of these projects, contact information for funding recipients was requested from the responsible ECCC program managers listed in the Gs&Cs database administered by Corporate Services and Finance Branch. Contact information was received for 294 (60%) of these projects. Of the 294 funding recipients invited to participate in the survey, responses were received from 57 recipientsFootnote 39 for a response rate of 19%.
The distribution of the 57 survey respondents across the six UTCs is as follows:
- Biodiversity – Wildlife and Habitat (n=26)
- Water Resources (n=5)
- Sustainable Ecosystems (n=2)
- Weather and Environmental Services for Canadians (n=14)
- Substances and Waste Management (n=4)
- Climate Change and Clean Air (n=6)
Due to the small number of respondents, it was not possible to compare the survey responses across the six UTCs. Therefore, the survey results are presented only in aggregate, for the overall sample of 57 funding recipients. Data analysis involved the computation of descriptive statistics for responses to each closed-ended question and a content analysis of responses to open-ended questions.
Report a problem or mistake on this page
- Date modified: