4. Findings

This section presents the findings of this evaluation by evaluation issue (relevance and performance) and by the related evaluation questions.

For each evaluation question, a rating is provided based on a judgment of the evaluation findings. The rating statements and their significance are outlined below in Table 1. A summary of ratings for the evaluation questions is presented in Annex F.

4.1 Relevance

4.1.1 Continued need for program

There is a demonstrated environmental need in each of the program areas covered by the six UTCs. There is also a need for Gs&Cs in general to help ECCC deliver on its priorities.

• There is a continuing environmental need in each of the areas covered by the six UTCs, including the following:
  • Biodiversity: A global decline in biodiversity (i.e., the variety of plants, animals and other life) is recognized as a serious environmental issue in view of biodiversity’s importance to human health, prosperity, security and well-being, and its contribution to essential goods and services that flow from healthy and diverse natural systems.
  • Water resources: Freshwater aquatic ecosystems are degraded and under pressure due to factors such as urban development and agricultural and industrial activities, eutrophication, degradation of wetlands, and lower-than-average water levels associated with climate change.
  • Sustainable ecosystems: Sustainable ecosystems serve to protect species and their habitats, preserve biodiversity and perform essential ecosystem services, which in turn have economic and health impacts, including agricultural productivity and improved air and water quality.
  • Weather and environmental services: Severe weather has significant human and economic consequences, therefore information regarding anticipated weather events is important for related planning, decision-making and mitigation of potential negative effects.
  • Substances and waste management: Chemical substances negatively impact human health^{Footnote 8} and the environment and increase future costs associated with water treatment, clean-up of contaminated sites, and treatment of illnesses related to chemical exposure. In addition, risks to the environment and human health stem from myriad sources of human-generated waste, including water pollution from a range of sectors (e.g., forestry, mining and agriculture), wastewater effluent from wastewater systems, and hazardous solid wastes.
  • Climate change and clean air: Climate change related to the release of greenhouse gases (GHGs) is warming the atmosphere and oceans and has the potential to affect the natural functioning of ecosystems, as well as weather conditions, biodiversity, hydrological systems and coastal infrastructure. Air pollution has been linked to a variety of adverse health outcomes, is carcinogenic, costs the Canadian economy billions of dollars each year due to premature deaths (approximately 21,000 Canadians annually), hospital admissions, emergency room visits and absenteeism^{Footnote 9} and contributes to adverse environmental effects such as ecosystem degradation, impacts on wildlife habitat and food from pollutants such as mercury, and impacts on vegetation from air pollutants such as ground-level ozone and sulphur dioxide.
• Document review findings indicate that, in general, there is a need for Gs&Cs as an instrument for federal government departments to deliver on their priorities and responsibilities to Canadians, for instance, by enabling non-government actors to do what they can do best and utilizing the work and skills of these actors for public policy purposes.^{Footnote 10Footnote 11}
• Almost all funding recipients surveyed (96%) agree (indicated 4 or 5 on a 5 point scale) that there is a continued need for federal contribution funding to support projects. Almost one-third (30%) indicated that without ECCC funding, their project would not have been carried out. This point was also reported by eight recipients in the project file review sample. A further 42% of survey respondents indicated that, in the absence of ECCC funding, their project would have been carried out but only if it were significantly reduced in size or scope.

4.1.2 Alignment with federal government priorities

Projects funded under the UTCs are aligned with federal priorities to protect the environment and with departmental strategic outcomes.

• The work funded under the UTCs is consistent with the Government of Canada’s expected outcome of “a clean and healthy environment.”^{Footnote 12} Moreover, one of the key current priorities of the Government of Canada is “a clean environment and a strong economy,” including issues targeted by the UTCs such as reducing carbon pollution, clean technology, and understanding and minimizing environmental impacts.^{Footnote 13}
• Each of the six umbrella authorities is aligned to a program in the departmental PAA and the associated ECCC strategic outcome. This linkage is specified in the Terms and Conditions for each umbrella.
• Most interviewees indicated that the objectives of the funded projects are aligned with federal and departmental priorities. Some key informants explained that the UTC project selection criteria, contribution agreement template, and explicit alignment of the UTCs with the department’s PAA help to ensure this.

4.1.3 Alignment with federal roles and responsibilities

The project activities funded under the UTCs are consistent with federal responsibilities specified in legislation such as the Department of the Environment Act. Each funded project is linked to the objectives of an ECCC program and to a departmental strategic outcome and, as such, is compatible with the department’s responsibilities.

• ECCC’s role in funding projects under the UTCs is compatible with the department’s responsibilities under the Department of the Environment Act, which specifies that the powers, duties and functions of the Minister include the preservation and enhancement of the quality of the natural environment, including water, air and soil quality, as well as renewable resources, migratory birds and other non-domestic flora and fauna. The Department of the Environment Act is specified as one of the legal authorities for each of the six UTCs. In addition, other federal legislation is identified as an authority for specific UTCs, for example, the Canada Wildlife Act and Species at Risk Act for Contributions in support of Biodiversity – Wildlife and Habitat, and the Canada Water Act for Contributions in support of Weather and Environmental Services for Canadians.
• The section on “purpose and expected results” in each UTC contribution agreement specifies the ECCC program which the funded activity is intended to support. Consistent with this, key informants indicated that the UTCs are used to support agreements that are directly aligned with program activities and objectives. Projects supported under the UTCs are used to meet short-term needs, to fill gaps in the department’s monitoring of environmental conditions or specific species, and to address knowledge gaps in environmental issues. These funded activities are consistent with departmental and federal responsibilities.
• Interviewees reported that it is appropriate for the federal government to provide project funding to facilitate the achievement of environmental outcomes because of the alignment of this work with ECCC’s federal mandate and responsibilities, particularly for certain areas (e.g., migratory birds, GHG reduction projects in developing countries, funding for Indigenous communities to deal with environmental issues), and the national and international scope of some projects. It was also felt by some key informants to be appropriate for the federal government to engage academics/researchers in the early stages of program development, as they may be better positioned to conduct initial exploratory and foundational research.

4.2 Performance – efficiency and economy

4.2.1 Program design

The UTCs are sensibly designed with clear expected results linked to PAA programs. They provide consistency in the funding approach across program areas in the department, flexibility to respond to new project opportunities in a timely manner, and reduce the number of Ts&Cs on which management is required to report. In addition, the UTCs appropriately leverage existing departmental mechanisms such as a structured Gs&Cs approvals process.

• The design of the UTCs is sensible in that each UTC is aligned to a PAA program and an associated ECCC strategic outcome and specifies the expected results to which it is designed to contribute. For each project funded under each UTC, the contribution agreement specifies the activities being funded, the applicable UTC results these activities are intended to support, and requirements for project performance reporting.
• Results of the project file review demonstrate that almost all funded project activities (97%) were judged to be well linked to the applicable UTC expected results (rated 4 or 5 on the 5 point scale), suggesting that this aspect of the design is sound.
• A majority of key informants indicated that there is a continued need for the UTC approach or mechanism because it provides consistency in the funding approach across program areas and flexibility to respond to new project opportunities in a timely manner, and it reduces the number of Ts&Cs which program managers are required to maintain, update and report on. They also indicated that projects funded under the UTCs provide access to highly technical/expert resources not available within ECCC and leverage additional project funding from outside the department.
• Key informants identified some alternative ways of supporting individuals, organizations or other levels of government in contributing to environmental outcomes, including contracting, partnerships, leveraging through collaboration and engagement with stakeholders, in-kind contributions and provision of one large lump sum contribution to the provincial government to administer all the individual projects in that province. However, the majority expressed a preference for the use of contribution agreements under the UTCs, for example, because this approach allows for flexibility in the timelines, scope and deliverables, provides a secure funding arrangement for the recipient, and encourages the leveraging of resources from project partners.
• The UTC design also leverages existing departmental mechanisms and tools to deliver program activities and facilitate the consistent and effective management of Gs&Cs under the UTCs, including
  • an ADM Committee on Grants and Contributions, the roles and responsibilities of which are specified in the committee’s terms of reference (see section 2.2);
  • a Gs&Cs approvals process, which outlines the roles, responsibilities and accountabilities of project managers and Finance representatives in the delivery of Gs&Cs funding using the UTCs (see Annex A); and
  • a departmental guide to Gs&Cs that includes principles, requirements, guidance, detailed procedures, standard templates, and step-by-step instructions for all stages of ECCC’s Gs&Cs funding process, including the UTCs.
• The vast majority (80% or more) of funding recipients who responded to the survey rated each of the various design features about which they were asked very highly (i.e., responded with a 4 or 5 on a 5 point scale), with the exception of the Recipient Guide to ECCC Contribution Agreements, which only 70% of respondents agreed was a useful reference tool (see Table 2). Corporate Services and Finance Branch is currently in the process of reviewing and updating this guide. Of the small proportion of respondents (11%) who disagreed that the conditions and requirements for funding were well suited to their project, most (60%) further reported that they were required to adjust their project approach to accommodate the conditions for funding to a large or great extent (indicated 4 or 5 on a 5 point scale). For example, some recipients needed to delay the purchase of lab equipment, delay the overall project, or find other means of paying researcher salaries.

Source: Survey of UTC Funding Recipients (n=57). Respondents rated their answer on a scale from 1 (Strongly Disagree) to 5 (Strongly Agree). Don’t Know and Not Applicable responses were removed.

4.2.2 Program efficiency and alternatives

A number of factors suggest that the UTCs support the efficient management of Gs&Cs, including the use of a single common set of Ts&Cs for a wide range of funding programs/initiatives aligned to different PAA programs, and the delivery of funded projects in collaboration and with leveraged resources from partners (i.e., 64% of project funding from partners). Funding recipients are highly satisfied with the delivery and efficiency of UTC funding; however, the timeliness of the notification and receipt of funding is a concern for some.

Efficiency of UTC approach

• ECCC has made efficiency improvements related to the UTCs and Gs&Cs in general by implementing management actions in response to recommendations from a previous evaluation and internal audit:
  • The 2009 evaluation of the former class Gs&Cs made five recommendations, and all of the associated management actions have been completed. For example, Corporate Services and Finance Branch offered training and developed new guidelines, procedures and tools for managers and finance personnel, and the department implemented a new risk-based approach to Gs&Cs management, developed step-by-step guidelines for the approval of contribution agreements (see Annex A) and, as already noted, created standardized tools including a contribution agreement template.
  • The 2013 internal audit of Gs&Cs also made five recommendations, some of which resulted in the implementation of a phased approach, aligned with departmental budget allocation and planning processes, to support the timely execution of Gs&Cs approvals.
• Results of the survey of funding recipients on efficiency-related aspects of UTC contribution programs are presented in Table 3. The responses are very positive, with large majorities of survey respondents (75% to 100%) providing positive ratings for all program components about which they were asked.
• File review evidence (i.e., open-ended responses in the final project reporting template) yields insight into the reasons for the positive survey responses and found that of 36 responses related to interactions with staff, 34 were positive and noted a highly collaborative relationship between recipient organizations and ECCC staff.
• There is a systematic process in place for prioritizing the projects to be supported by Gs&Cs, including those under the UTCs, which should help to ensure that project funding and activities are well targeted. As described in Annex A, as part of annual Gs&Cs planning, Directors provide proposals for priority Gs&Cs by sub-program, which are reviewed by Executive Directors and/or ADMs. ADMs then recommend priorities for their branch in the annual Gs&Cs plan. Finally, the ADM Committee on Gs&Cs provides oversight and recommendations on Gs&Cs to the Deputy Minister.
• A majority of the interviewees who responded to the issue of efficiency and economy indicated that the expected results of projects supported by UTCs have been achieved at a reasonable program cost (e.g., due to the leveraging of additional resources such as funding and expertise from project partners).
• In terms of the UTC mechanism overall, the majority of interviewees who responded indicated that the UTC approach confers a number of advantages, including administrative savings because there are fewer Ts&Cs to manage; flexibility to respond quickly to new priorities and project opportunities (e.g., development of a pilot program); consistency and standardization of the approach across programs; and assurance of alignment of contributions with the PAA. The key disadvantage noted in interviews was the lack of alignment of some research opportunities with the UTCs.

Source: Survey of UTC Funding Recipients (n=57). Respondents rated their answer on a scale from 1 (Strongly Disagree) to 5 (Strongly Agree). Don’t Know and Not Applicable responses were removed.

Timeliness of funding

• Despite mostly positive results, there is one area of program delivery in which evidence suggests there may be opportunity for improvement: timeliness of funding.
  • In the file review, of 25 final project reports that discussed inefficiencies related to the Gs&Cs process, 20 reports specifically mentioned inefficiencies related to funding timelines. In particular, these recipients highlighted that they experienced difficulties in leveraging additional funding due to delays in receiving notice of the acceptance of their proposal and that the receipt of funds often did not align with project timelines, having an adverse impact on the achievement of intended project objectives.
  • Approximately half of the key informants noted that delays may cause project timelines to be missed when projects are time-bound or seasonal, or proponents are no longer available.
  • Although the survey responses were very positive (see Table 3), funding recipients surveyed were somewhat less likely to agree that “the notice of funding was received in a timely manner” (75%), with a majority of those who disagreed (13%) indicating that this caused challenges such as delays in purchasing lab equipment or hiring staff and imposed financial risks. Recipients noted that the funding application timelines can be confusing and difficult to meet, as often there is a very short turnaround between the notification of funding and the deadline for use of funds.
• Recipient survey suggestions for improving efficiency include expediting the negotiation process and funding notification to allow sufficient time for meaningful use of funds prior to the fiscal year-end deadline; using contributions that are multi-year and allowing carry-over funding from one year to the next; and reducing administrative burden by simplifying the annual activity and financial reporting process. Overall, however, funding recipients feel that the contribution funding process worked well and was essential to their project’s success.

Project partnerships and leveraging of ECCC funding

• Findings from the project file review indicate that for 62 of the 85 projects funded by ECCC, recipients were able to leverage additional funding (cash or in-kind) from 251 other contributors, for an average of three non-ECCC contributors per project. Overall, considering both financial and in-kind contributions, projects were resourced at an average rate of 36% by ECCC and 64% by partners, for a leveraging ratio of 1.79. Note that 16 of these projects were conducted with no additional funding partners, and information on partner contributions was incomplete for 7 projects. An assessment of leveraged funding by umbrella authority is presented in Table 4 below.
• The most common types of project partners identified by funding recipients in the survey are academic institutions (32%), other federal departments/agencies (25%), environmental non-governmental organizations (24%) and other types of partners, such as other levels of government (19%). The most frequently cited form of assistance received from these organizations was funding/financial support (67%), followed by in-kind support in the form of research/technical advice (47%) and volunteer or staff support (46%). A majority (65%) of respondents reported that the contribution funding received from ECCC assisted them in obtaining additional funding from other sources to a large or great extent (indicated 4 or 5 on a 5 point scale).
• In the project file review, responses were available from 20 funding recipients concerning the effectiveness of collaboration with project partners and stakeholders. Among these responses, 18 project reports cited effective collaboration between partners as a key means to achieving project objectives, particularly when the scope of the project was quite large.

Note: The leveraging ratio was computed by dividing the total funding from partners by that from ECCC. Source: Project file review (n=85).

4.2.3 Performance measurement

Available performance information is incomplete. Information at the project level is provided in the final project reporting template for most, though not all UTC projects. However, final reports focus primarily on project activities and outputs, as opposed to intended environmental outcomes or expected results. While each UTC specifies expected results and associated performance indicators, the evaluation found no evidence of performance reporting against these expected results and indicators. Also, ECCC has no consolidated management information on UTC project performance.

• For PAA programs and sub-programs, performance information is collected against expected results and performance indicators as part of the department’s Performance Measurement Framework (PMF) and presented in Departmental Performance Reports. However, these are different from the UTC-specific results and indicators. Beyond the PMF, program performance data are also collected as part of program-level performance measurement strategies, though there are gaps in this regard for some program areas throughout the department.
• The Terms and Conditions for each umbrella authority specify expected results (listed in Annex C) and performance indicators, which are typically related to but distinct from the expected results and indicators in the department’s PMF. The Terms and Conditions note that these UTC-specific indicators will be used to monitor and report on the impact and effectiveness of contributions toward realizing program outcomes. Based on the evaluation evidence, this does not appear to be happening, as the UTC performance indicators are not identified in contribution agreements or final project report templates and most key informants appear to be unfamiliar with these indicators. The department’s former Environmental Stewardship Branch did develop a proposed performance measurement framework for the Biodiversity – Wildlife and Habitat authority using the expected results and indicators from the Ts&Cs, but there was no evidence that it was actually used. The evaluation did not identify any other similar performance measurement strategies utilizing the expected results and indicators for the other five UTCs.
• A majority of key informants and funding recipients in the survey (91%) indicated that performance indicators are developed for intended project deliverables and short-term outcomes, though a small number of informants observed that deliverables or outcomes are vaguely worded in some contribution agreements. In addition, 79% of funding recipients indicated that performance data are collected against project activities and intended short-term outcomes. Examples provided of types of project performance data include project reach and participation levels; status of project activities/deliverables; population estimates and other data on migratory birds; and quantitative data on project results. As would be expected, funding recipients most commonly shared performance data with ECCC through the department’s final project reporting templates (37%) and annual project reports (34%). Some other methods used for communicating data to the department include expert and technical reports, emails, phone/conference calls, meetings and workshops.
• For each funded project, the contribution agreement specifies the requirements for financial reporting (cash flow statement, financial forecast, annual and final report) and for project activity progress reporting (annual and final reports). To facilitate this reporting, Corporate Services and Finance Branch developed a final project reporting template (referred to above). Project managers are free to use this standard template, or with ECCC’s approval, to adapt it or use another report format specific to their reporting requirements. The template for the final project report does not, however, specifically request information on any UTC expected results that are reflected in the contribution agreements.
• Based on the project file review, completed final financial and project activity reports could not be located for 21 projects (25% of the sample). Although reporting templates were generally well completed for the remaining projects, open-ended questions relating to project results, lessons learned, project sustainability and feedback on interactions with ECCC staff were inconsistently answered. In most cases, the project’s contribution to the applicable UTC expected result(s), included in the contribution agreement, was only indirectly addressed. In contrast with the survey and interview findings cited above, which suggested that performance indicators are developed for projects’ short-term outcomes, the file review indicates that reports generally focused on project activities and outputs.
• The department currently has no consolidated information on project performance for Gs&Cs (i.e., data on status of project activities, outputs and outcomes). However, Corporate Services and Finance Branch maintains a Gs&Cs database. This database contains financial information for projects funded under each of the six UTCs and is used to track Gs&Cs expenditures and the advancement of G&G projects for payment purposes. However, this database includes no performance information on funded projects, nor was it designed to capture this type of information. Moreover, although it includes the responsible ECCC program manager and the project proponent, it does not include contact information for project proponents – information which would be useful to support performance measurement and evaluation (e.g., for undertaking a recipient survey, as was conducted in the present evaluation).
• The noted lack of consolidated project performance information is consistent with the Commissioner of the Environment and Sustainable Development’s (CESD’s) 2013 assessment of ECCC’s funding programs for Species at Risk. This audit found that although results from funded projects were tracked at the project level and sometimes compiled at a program level, no overall compilation and assessment of the cumulative contribution of funding programs to the achievement of higher-level outcomes was conducted. Such an assessment would be particularly useful for informing future funding priorities. Improvements to these funding programs were made in response to the 2013 CESD findings.

4.3. Performance – effectiveness

The evaluation found that projects’ planned activities and intended short-term outcomes are for the most part being achieved, although evidence for the latter is limited to the views expressed by program managers and funding recipients. In addition, while the evidence would suggest that projects are contributing to UTC expected results, definitive conclusions cannot be effectively drawn due to the lack of objective performance information. Also, no unintended outcomes of the UTC mechanism have been identified.

Project activities and short-term outcomes

• All lines of evidence suggest that most funded project activities and short-term outcomes are being achieved:
  • Most key informants indicated that the intended outcomes of completed projects funded under the UTC in their program area have been achieved to a large extent (i.e., 4 or 5 on a 5 point scale).
  • A total of 318 activities were initiated across the entire file review sample (85 projects). However, activity completion status could not be assessed for 9 projects (10%) due to unavailable project documentation.
  • Of the remaining 282 activities assessed (representing 76 projects), 216 activities (77%) were completed as planned during the timeframe established in the contribution agreement, 38 activities (13%) were partially completed, and 11 activities (4%) were modified during the course of the project. Only 9 activities (3%) were not completed.
  • In the survey, nearly nine in ten funding recipients reported that their project activities were fully (63%) or mostly (31%) completed as planned (4 or 5 on the 5 point scale). The same proportion indicated that the intended outcomes of their project were completely (63%) or mostly achieved (31%). (See Figure 1).
• Interviewees nonetheless identified some challenges and factors which have negatively influenced the achievement of intended project outcomes, including late delivery of project reports or products and changes in the financial contribution of other involved parties. On the other hand, factors which have positively influenced the achievement of project outcomes include good coordination with respect to work being undertaken by the associated ECCC program and additional funding from another party.

Source: Survey of UTC Funding Recipients (n=57); Don’t Know and Not Applicable responses were removed.

• Fifty-one of 85 funding recipients whose files were reviewed reported that their project would be sustained in whole or in part at the completion of ECCC funding, although 10 of these recipients reported doing so by applying for additional funding from the same or similar ECCC programs. Other means of sustaining projects specified by recipients include financial and in-kind contributions from provincial and territorial governments, other federal departments, international sources and NGOs.

UTC expected results

• Given the lack of results information, as part of the file review the evaluators rated the extent to which the projects’ completed activities appeared to contribute to the UTC expected results specified in the contribution agreement (on a scale ranging from “not at all” (1) to “a great extent” (5)). This rating exercise suggested a strong contribution to the 87 expected results specified for all projects in the sample. The projects’ contributions were judged to be large or great (4 or 5 on the scale) for 95% of expected results, and moderate (3 on the scale) for the remaining 5% of results. The average rating for each of the six UTCs ranged from 4.3 to 4.9 out of 5.
• Similarly, in the survey, funding recipients were asked to rate the extent to which their project contributed to the UTC expected result(s) specified in their contribution agreement (using the same 5 point scale as in the file review). Again, the findings suggest a contribution. Across a total of 206 responses on all applicable expected results combined, the majority of responses (75%) indicated that projects contributed to a large or great extent. The remaining responses indicated a moderate contribution (16%) or little or no contribution (9%). The average rating across all responses and expected results was 4.0 out of 5.
• However, due to limitations associated with the above lines of evidence (i.e., only partial and indirect evidence available in project files, and the possibility of bias among survey respondents asked to rate their own project’s performance), as well as the lack of actual performance data, it is not possible to draw definitive and objective conclusions with respect to the achievement of UTC expected results.

Unintended outcomes

No unintended outcomes of the UTC mechanism were identified by the evaluation.