Evaluation of the Environmental Damages Fund: final report

Final Report

Audit and Evaluation Branch October 2014

Report Clearance Steps

Planning phase completed - September 2013
Report sent for management response - April 2014
Management response received - April 2014
Report completed - April 2014
Report approved by Deputy Minister - October 2014

Acronyms used in the report

ADM
Assistant Deputy Minister

AEB
Audit and Evaluation Branch

CESD
Commissioner of the Environment and Sustainable Development

CWS
Canadian Wildlife Service

DFO
Fisheries and Oceans Canada

EC
Environment Canada

DG
Director General

EDF
Environmental Damages Fund

EEA
Environmental Enforcement Act

ESB
Environmental Stewardship Branch

EVAMPA
Environmental Violations Administrative Monetary Penalties Act

FTE
Full Time Equivalent

Gs&Cs
Grants and Contributions

MIS
Management Information System

MOU
Memorandum of Understanding

NCR
National Capital Region

NSERC
Natural Sciences and Engineering Research Council of Canada

O&M
Operations and maintenance

PAA
Program Alignment Architecture

PCA
Parks Canada Agency

PPSC
Public Prosecution Service of Canada

PWGSC
Public Works and Government Services Canada

R&D
Research and development

RDG
Regional Director General

S&T
Science and technology

TB
Treasury Board

TC
Transport Canada

Acknowledgments

The Evaluation Project Team would like to thank those individuals who contributed to this project, particularly members of the Evaluation Committee, as well as all interviewees and survey respondents who provided insights and comments crucial to this evaluation.

The Evaluation Project Team was led by Michael Callahan, under the direction of the Environment Canada Evaluation Director, William Blois, and included Lindsay Comeau.

The evaluation was conducted by Goss Gilroy Inc., and the final report was prepared by Goss Gilroy Inc. and the Evaluation Division Project Team, Audit and Evaluation Branch.

Table of Contents

Executive Summary

Background

This report presents the results of the evaluation of the Environmental Damages Fund (EDF) conducted by Environment Canada’s (EC) Audit and Evaluation Branch in 2013-14. The purpose of this evaluation was to assess the relevance and performance of the EDF.

This evaluation is part of EC’s 2012 Risk-Based Audit and Evaluation Plan, which was approved by the Deputy Minister. The evaluation was conducted in order to meet the coverage requirements of the Treasury Board Policy on Evaluation, which require that an evaluation of all direct program spending be conducted at least once every five years.

The EDF is a specified purpose account created by the Government of Canada in 1995 and administered by EC. The EDF provides a mechanism for directing funds received as a result of fines, court orders and voluntary payments related to environmental infractions to priority projects that will benefit Canada’s natural environment. Since 1998, the EDF has allocated or committed over $4.8 million and has funded 201 projects across Canada.

While the EDF is administered and delivered by EC, the Department works closely with other government departments, namely Fisheries and Oceans Canada, Transport Canada and Parks Canada Agency, which have statutes that contain sentencing provisions enabling judges to direct funds to the EDF.

EC provides oversight and accountability of monies directed to the EDF. The program solicits project proposals from eligible groups for projects that restore the natural environment or prevent harm to wildlife in the geographic region (local area, region or province) where the original incident occurred. To be eligible for funding, projects must address one or more of the following EDF categories: Restoration (highest funding priority); Environmental Quality Improvement; Research and Development; and Education and Awareness.

The two ultimate outcomes of the EDF program are:

The evaluation covers the timeframe from 2008-2009 to mid-2013-2014. The evaluation involved a review of documents, a structured review of project final reports, analysis of program administrative data, key informant interviews and a survey of stakeholders.

Findings and Conclusions

Relevance

According to evaluation evidence, there is a legislative need for the EDF program to manage financial penalties awarded under four federal departments’ statutes. The EDF funds restoration and environmental quality improvement projects that address an environmental need usually in the geographic area where the damage occurred. Documentation and interviews indicate that the role and function of the EDF program are unique.

The evaluation found that the program is aligned with federal priorities for sustainable ecosystems and environmental protection. In 2010, the priority of the program was demonstrated through the passage of the Environmental Enforcement Act, which included an amendment to several statutes for mandatory provision of fines and administrative monetary penalties to the EDF. While the program places a priority on respecting all court conditions of awards directed to the EDF, the flexible regional delivery of the program also permits departmental and regional priorities to be considered in funding decisions through the use of Regional Management Plans.

The EDF program is consistent with federal roles and responsibilities given the federal legislative basis of the program. The role of EC as lead department for the program on behalf of other federal departments is consistent with EC’s federal environmental coordination mandate as outlined in the Department of the Environment Act.

Achievement of Intended Outcomes

The evidence suggests that the program is making progress toward achieving its direct outcomes, though the way some intended outcomes have been articulated in the program logic model does not align well with the objectives and delivery of the program. While promotion of the EDF to the enforcement and legal communities has been uneven across regions, recommendations to judges for its use appear to be widespread. Almost all enforcement officials and Crown prosecutors who were consulted indicated that they would recommend the use of the EDF as a sentencing option to their colleagues. There is evidence that the number of awards directed to the EDF (though not their total value) has increased over the last five years, with a notable increase in the Prairie and Northern region. Nevertheless, there is interest in more frequent communications to the enforcement and legal communities to increase their understanding of the program and its impacts, and the transparency of the use of the funds.

Interest in EDF calls for proposals is strong among potential funding recipients, and program officials provide support to ensure quality proposals are received. Scientific and technical reviews of proposals and oversight ensure that projects are implemented in accordance with funding agreements. According to funding recipients, EDF funds were essential for the implementation of their project.

Projects funded by the EDF are contributing to intended environmental outcomes in the areas of restoration and environmental quality improvement. Project data from the EDF Management Information Systems (MIS) indicate that half of EDF-funded projects are under these two funding categories and completed projects with goals related to environmental outcomes (hectares restored or improved) have exceeded targets by about 15 to 20 per cent.

There is also evidence that the EDF program successfully funds projects under the education and awareness component that engage participants to enhance their awareness and understanding. The number of participants involved in these EDF projects again exceeds original targets (by 27 per cent), and the MIS data also suggest that behaviour change is taking place.

There is insufficient evaluation evidence on the achievement of the intended outcome related to increased knowledge due to the recent change to the funding mechanism for research and development (R&D) projects from regional to national delivery of this component. The new national process has yielded funding of only two projects since it was implemented and the amount of EDF fines assessed as appropriate for R&D has not met the original target. Performance information in the MIS is limited for R&D projects prior to the change in the funding mechanism.

Although the intended long-term program outcomes are less directly measurable, interviewees perceive that the program is contributing to improvements in environmental quality at the local level and prevention of future incidents. There are no unintended negative outcomes of the program and positive unintended outcomes include partnership development at the community level and economic impacts of projects.

Efficiency and Economy

The overall model of the EDF program as a Government of Canada program delivered by EC is widely held to be sound, with many advantages (e.g., the ability to combine small fines across acts administered by different federal departments to fund larger restoration projects, the expertise of EC staff in these types of projects). However, multiple changes to the program during the period under study have created some challenges in the efficient management of the program. Key changes include a shift from regional to national level funding of R&D component projects, consolidation of five EC regions into three, and location of the national coordination unit within the Environmental Stewardship Branch. The program is currently in a period of transition in governance as responsibility for national coordination and the R&D component has recently been transferred to the Atlantic and Quebec Region.

The full impact of the changes in the program is not yet known; however, there is evidence that the changes have yielded mixed results to date and the implementation of the 2009 EDF Management Framework has been impeded. There are divided views, for example, on the national delivery of the R&D funding component. Proponents of the model feel the 2013 MOU with the Natural Sciences and Engineering Research Council of Canada (NSERC) will lead to more impactful larger-scale investments in research, leveraging NSERC expertise for peer review and oversight. Others are concerned about the new approach which has led to an administratively burdensome process for assessment of funds, the fact that few funds have been assessed for and few projects funded under the R&D component, and the potential movement of EDF funds outside the local region to pool them for larger national projects (which could undermine the credibility of the program for the enforcement and legal communities if this is not seen as fully respecting court restrictions and addressing priorities in legislation). Other delivery challenges include: duplication in reporting due to the fact the responsibility for the EDF was shared by two Assistant Deputy Ministers during the study period, some lack of clarity in roles and responsibilities between the national coordinating unit and regions, inconsistent regional practices for engaging partner departments, lack of a clear protocol for the transfer of court-awarded funds to the EDF, and the MIS system and performance measures that do not fully meet the reporting requirements and decision-making needs at the program level.

Evidence on the efficiency of the program is mixed. At the project level, leveraging of EDF funding is strong, in spite of the absence of any requirement to secure additional sources of funds. According to administrative data, on average, EDF provided 29 per cent of all project-level funding, compared to 71 per cent that came from other sources. The majority of funded projects are also sustainable in at least some respect. Funding recipients are generally satisfied with the delivery of the EDF program. More than eight in ten agree that the funding process was timely and efficient and that the service provided by EDF staff met their needs. Funding recipients are generally less satisfied, however, with reporting requirements that they perceive to be onerous in comparison to the small amounts of funding provided by the program for some agreements. The administrative ratio of the program over the evaluation time frame is approximately 0.36, which is higher than the ratios for some EC grants and contributions (Gs&Cs) programs recently evaluated (ranging from 0.15 to 0.25). This higher ratio can be partially explained, however, by the greater administrative investment of the EDF program in managing both the incoming awards and the outgoing funds to EDF project recipients. As noted above, there are also some program management challenges detracting from the program’s efficiency.

Recommendations

The following recommendations are directed to the Regional Director General (RDG), Atlantic and Quebec Region, as the senior departmental official responsible for the management of the EDF.

Management Response

The responsible RDG agrees with all five recommendations and has developed a management response that appropriately addresses each of the recommendations. The full management response can be found in Section 6 of the report.

1.0 Introduction

This report presents the results of the Evaluation of the Environmental Damages Fund (EDF) which was conducted by Goss Gilroy Inc. under contract to Environment Canada’s (EC) Audit and Evaluation Branch (AEB) in 2013-2014. The evaluation covers the timeframe from 2008-2009 to mid-2013-2014.

The document is organized as follows: Section 2.0 provides background information on the EDF. Section 3.0 presents the evaluation design, including the purpose and scope of the evaluation, as well as the approach and methods used to conduct the evaluation. Section 4.0 and 5.0 lay out, respectively, the evaluation’s findings and conclusions. The recommendations and management response are presented in Section 6.0.

2.0 Background

2.1 Program Overview

Purpose and Goal of the EDF Program

The EDF is a Government of Canada (GoC) program established in 1995 by a Treasury Board decision pursuant to the Financial Administration Act to oversee and manage the disbursement of funds received as compensation for environmental damages. The program’s goal is to achieve restoration of the natural environment and wildlife conservation in a cost-effective way and in accordance with conditions specified by the courts. While the EDF is administered and delivered by EC, the Department works closely with Fisheries and Oceans Canada (DFO), Transport Canada (TC) and Parks Canada Agency (PCA), which have statutes directing funds to the EDF or that contain sentencing provisions enabling judges to direct funds to the EDF.

Environmental Legislative Framework of the EDF

Currently, six EC and three PCA statutes include a mandatory provision to direct all fines to the EDF. Another nine federal statutes contain sentencing mechanisms that may be used by judges to direct monies to the EDF. Finally, the Environmental Violations Administrative Monetary Penalties Act (EVAMPA) establishes, for 11 acts, a system of administrative monetary penalties (AMPs) that will direct all funds to the EDF. Additional details related to the environmental legislative framework of the EDF are provided in Section 4.1.3 of this report.

2.2 Program Delivery

Program Promotion

To encourage the directing of court awards and negotiated settlements to the EDF (in the absence of legislation that automatically directs them to the EDF), the program is promoted as a creative sentencing tool to the legal and the enforcement communities. Enforcement officers formulate recommendations to prosecutors regarding appropriate legal mechanisms to be used when an offence has occurred. Prosecutors are involved in plea-negotiations and make sentencing recommendations to the judge, who then makes the final ruling.

Project Delivery

EDF funds are receivedFootnote 1 by regional offices from a number of sources such as tickets (fines) issued by enforcement officers, provincial and federal civil or court awards, negotiated settlements and voluntary payments. Prior to allocating EDF funds, regional EC staff gather relevant background documents on funds received to identify any specific requirements attached to the funds. Regional offices complete an “Assessment of Funds Received” form for each individual monetary contribution to the EDF. This internal document serves as a record linking the original incident and court restrictions to recommendations for the best targeted use of the EDF contribution. EDF project categories are: Restoration; Environmental Quality Improvement; Research and Development; and Education and Awareness.Footnote 2 Best use of the funds and the category of projects to be solicited may also be informed by Regional EDF Management Plans. These plans, developed in consultation with departmental and external experts, identify program and departmental priorities as well as priority ecosystem “hot spots” relevant to each region.

Once funds are received and assessed, EC actively seeks applications from eligible organizations by posting a call for proposals on the program website and/or directly contacting potential recipients. Eligible organizations include non-profit organizations, Aboriginal organizations, universities and other academic institutions, as well as provincial, territorial and municipal governments. Projects must be cost-effective, technically feasible, scientifically sound and directly address any court or use restrictions.Footnote 3

A principle of the EDF program is that compensation directed to the EDF is used for restoration of the natural environment, including environmental quality improvement, and wildlife conservation, with priority given to projects that restore the environment in the geographic region where the original incident occurred and in accordance with conditions specified by the courts. While priority is given to restoration and environmental improvement projects, if such projects are not possible or not the best use of funds (e.g., fine for failing to submit or provide reports as required by regulations, use of tetrachloroethylene in dry cleaning), then both research and development projects and education and awareness projects related to restoration are considered.Footnote 4Project proponents are encouraged to build partnerships with other stakeholders. The maximum contribution to any single recipient and the total contribution related to any one award cannot exceed $6 million.Footnote 5

Once the proposal eligibility for funding is confirmed by EC staff, a technical review team, composed of experts from EC and other government departments, evaluates the applications for scientific and technical merit. Proposals that pass the initial screening process are reviewed by an advisory committee, which then makes funding recommendations to the Regional Director General (RDG) or delegated equivalent responsible for final approval.Footnote 6
Since 1998, the program has allocated or committed over $4.8 million in EDF funding to 201 approved projects across the five regions. Funding for new EDF projects was suspended between January and August of 2011 as part of an EC effort to develop a new national research and development (R&D) funding component that addresses priority research gaps to support and inform restoration practices. This program redesign initiative culminated in 2013 with EC signing a three-year pilot Memorandum of Understanding (MOU) with the Natural Sciences and Engineering Research Council of Canada (NSERC) aimed at integrating EDF R&D with the NSERC Strategic Project Grants Program. This new funding mechanism is intended to be a means for better achieving R&D objectives. In advance of the MOU, the first national R&D project to be funded by the EDF was approved in 2012. An initial target was set to direct 40 per cent of EDF funds to the national R&D component.

2.3 Governance Structure

In response to recommendations stemming from a 2009 evaluation of the EDFFootnote 7 , and in order to increase program effectiveness and respond to an anticipated increase in EDF awards, the Department developed an EDF Management Framework (2009) intended to address the following:

The EDF is managed and administered by EC. Each EC region (Atlantic and Quebec, Ontario, and West and North) is responsible for delivery of EDF restoration, environmental quality improvement and education and awareness projects. This means that the regions are responsible for regional program planning and management including management plans, project selection and approval process, information and communications management, project monitoring and evaluation, and EDF regional financial tracking. RDGs approve regional management plans (see discussion of program delivery in Section 2.2), projects and funding agreements.Footnote 8

Until recently, a national program manager within the Environmental Stewardship Branch (ESB) provided a program coordination role. With the change of program responsibility in January 2014 to the RDG of the Atlantic and Quebec Region, the national program coordination role and responsibility for the new nationally-delivered R&D component has shifted to this region.

A national DG Committee (with representatives from EC and other federal government departments) has recently been established to provide overall guidance on the implementation and administration of the EDF. Membership in this committee includes EC (ESBFootnote 9 , Science and Technology Branch, Enforcement Branch, RDGs), DFO, PCA and TC. The terms of reference for this committee state that the purpose of the EDF DG Committee is to provide oversight and direction for the administration of the EDF program.

2.4 Resources

As of 2012, the EDF had received over $6.5 million in funds from 197 awards.Footnote 10Per the Treasury Board Policy on Specified Purpose Accounts, monies directed to a specified purpose account may cover administrative costs when they are specifically authorized by the enabling authority. In the case of the EDF, enabling authorities include: the courts, whereby a judge may specify that money from an award may be used for administrative purposes; and any legislation that makes provisions to allow money from an award to be used to administer the EDF.Footnote 11There are draft program guidelines in place for the potential use of a portion of awards for administrative purposes. The EDF has, traditionally, been administered using minimal existing departmental resources. As indicated in Table 2.1, total departmental funding (Vote 1) allocated for EDF administration for EC over the five-year period of 2008-2009 to 2012-2013 is approximately $988,000.Footnote 12

A more detailed discussion of program resources can be found in Section 4.2.2 under the discussion of demonstration of efficiency (evaluation question 10).

Table 2.1: EDF Program Expenditures for 2008-2009 to 2012-2013 and Budget for 2013-2014
  2008-2009 2009-2010 2010-2011 2011-2012 2012-2013 2013-2014 Budget
FTEs 0.9 0.8 1.3 3.7 2.6 4.5
Salaries 43,928 35,779 96,399 289,299 250,862 335,949
O&M 37,708 74,037 63,631 43,794 52,765 69,000
Totals 81,636 109,816 160,030 333,093 303,627 404,949
  1.   FTE information for 2008-2009 to 2012-2013 extracted from EC’s Salary Management System.
  2.   Salaries and O&M for 2008-2009 to 2012-2013 from EC's financial system, as provided by Finance Branch. As EDF work is co-located with that of other EC funding programs, program management is concerned that actual salary usage for 2008-2009 to 2012-2013 may be more than reported in EC’s financial system. For example, during this timeframe staff may have incorrectly coded some of the EDF-related work they did under the EcoAction Community Funding Program. EDF program management plans to take measures to improve the accuracy of financial coding.
  3. 2013-14 Budget data from Environment Canada, New Program Model and Resources Dedicated to EcoAction and the Environmental Damages Fund (EDF), Draft April 2012 revised April 2013.

2.5 Program Logic Model

The logic model used for the purpose of the evaluation can be found in Annex 1. The final outcomes to which the EDF is ultimately intended to contribute are:

3.0 Evaluation Design

3.1 Purpose and Scope

The purpose of this evaluation is to assess the relevance and performance (including effectiveness, efficiency and economy) of the EDF. The evaluation covers the time frame from 2008-2009 to mid-2013-14.

The evaluation of the EDF is part of EC’s 2012 Risk-Based Evaluation Plan, approved by the Deputy Minister, and is intended to support evidence-based decision making in policy, expenditure management and program improvements. The evaluation was conducted in order to meet the coverage requirements of the Treasury Board Policy on Evaluation, which require that all direct program spending be evaluated at least once every five years.

3.2 Evaluation Approach and Methodology

The methodological approach and level of effort for this evaluation were calibrated, taking into consideration the statutory basis of the program, senior management feedback, and AEB assessment of risk. Findings of previous evaluations of the program have been positive, thus lowering the overall risk associated with the program. However, the implementation of the EDF Management Framework in 2009 and recent changes to the administration of the program were key considerations for evaluation scoping. Custom questions were included in data collection instruments to address the evolution of the EDF, including questions related to design, delivery and governance. With these considerations, the following data collection methodologies were employed and the evidence from these methods was triangulated to develop findings and conclusions.Footnote 13

Document Review

Key documents were gathered and reviewed using a document review template. Documents included: descriptive program information (e.g., EDF Management Framework, regional management plans, funding agreement Terms and Conditions, program guidelines), departmental and Government of Canada publications related to policy and priorities, and performance and evaluation reporting documents (the EDF Evaluation Plan, 2009 Evaluation of the EDF).

Management Information System (MIS) Data Analysis

An extract of EDF administrative data from the MIS database that included all relevant project data for the study timeframe was obtained for analysis. During the period under study, 100 projects were funded, including 72 completed projects and 28 ongoing projects. The database contains information on past and current projects, including information from application forms and project performance reports. Among other fields, the database includes descriptive information on program activity, financials (awards and project funding), and data on both targets and actual performance related to key environmental and capacity-building indicators. These data were used primarily for an assessment of the program’s achievement of outcomes.

Review of Project Final Reports

A review of a sample of project final reports was completed to examine project activities, outputs, outcomes and lessons learned in more depth. This method involved reviewing a sample of 33 completed EDF project final reports. Projects were selected to broadly represent overall program activity on a number of criteria: EDF funding category; project size (i.e., dollar value); fiscal year; funding recipient type; and region. Findings from the file review addressed the issue of performance.

Key Informant Interviews

Key informant interviews were conducted either in person or by telephone with a total of 23 respondents. All relevant stakeholder perspectives were considered in the key informant interview analysis. Other lines of evidence (i.e., survey of stakeholders) also gathered the perspectives of individuals not directly accountable for the program, providing a balanced blend of views on program performance. The following provides a breakdown of the interviewees:

A customized, open-ended guide was developed for each interviewee category. Interviewees received a copy of the interview guide in advance of the interview. The findings from the key informant interviews addressed all evaluation questions, but were particularly important for the performance issue.

Online Survey of Stakeholders

An online survey was conducted of EDF stakeholders, including 1) recipients of EDF funding; 2) enforcement officers, managers and regional directors; and 3) Crown prosecutors. The approach to conducting each of these surveys is described below.

Data collection for the survey took place between February 3 and March 3, 2014. In total, 40 survey questionnaires were completed by funding recipients for a response rate of 43 per cent and 82 surveys were completed by enforcement personnel for a response rate of 32 per cent. Completed surveys were received from 11 Crown prosecutors. However, as the total number of Crown prosecutors that received a forwarded link is unknown, a response rate cannot be calculated. The detailed findings for key survey questions are presented in Annex 2.

3.3 Limitations

Challenges experienced during the conduct of the evaluation, as well as the related limitations and strategies used to mitigate their impact, are outlined below.

1. Evolving environment

The EDF operational program environment has been affected by many changes during the period under study, including consolidation of regional operations from five regions to three, introduction of a new national mechanism to disburse research and development funds, legislative amendments pertaining to use of the EDF under several pieces of legislation and, during the conduct of the evaluation, a shift in the responsibility centre for the program from the Strategic Priorities Division, ESB, to the Canadian Wildlife Service (CWS), ESB, and finally to the Regional Director General’s Office (RDGO) in the Atlantic and Quebec Region. The lack of stability in the program during the period under study affected the analysis and assessment of several of the outcome indicators. This impact has been noted throughout the report where relevant.

2. Quality of performance data

While the EDF program has a performance measurement strategy linked to the outputs and outcomes in the logic model, regular reporting (e.g., an annual EDF performance report) is not yet occurring. As well, the performance data available in the MIS were limited. As the number of completed EDF projects is small and highly variable, and not all completed projects had performance information, there were few observations available for each of the many performance indicators. Only those performance indicators that had data for nine projects or more (selected as a threshold based on the distribution of the observations) were included in the analysis of performance data.

4.0 Findings

This section presents the findings of this evaluation by evaluation issue (relevance and performance) and by the related evaluation questions. For each evaluation question, a rating is provided based on a judgment of the evaluation findings. The rating statements and their significance are outlined below in Table 4.1. A summary of ratings for all evaluation questions is presented in Annex 3: Summary of Findings.

Table 4.1: Definitions of Standard Rating Statements
Statement Definition
Acceptable The program has demonstrated that it has met the expectations with respect to the issue area.
Opportunity for improvement The program has demonstrated that it has made progress to meet the expectations with respect to the issue area, but continued improvement can still be made.
Attention required The program has not demonstrated that it has made progress to meet the expectations with respect to the issue area and attention is needed on a priority basis.
Not applicable There is no expectation that the program would have addressed the evaluation issue.
Unable to assess Insufficient evidence is available to support a rating.

4.1 Relevance

4.1.1 Continued Need for Program

Evaluation Issue: Relevance Rating
1. Is there a continued need for the EDF program? Acceptable

There is a demonstrated need for the EDF program to manage financial awards which are directed to the program under multiple federal statutes. There is evidence that investments in restoration projects of the type funded by the EDF continue to be required to ensure healthy ecosystems and conserve wildlife. The EDF program addresses this need and is unique at the federal level.

4.1.2 Alignment with Federal Government Priorities

Evaluation Issue: Relevance Rating
2. Is the EDF program aligned with federal government priorities? Acceptable

The EDF program is consistent with federal and departmental priorities related to environmental protection and sustainable ecosystems. Recent legislative amendments have raised the profile of the EDF by including a mandatory provision to direct fines and administrative monetary penalties to the EDF, which support federal priorities to protect Canada’s water and land. While the program places a priority on respecting all court restrictions on the awards in the disbursement of funds to projects, the flexible nature of the program and regional delivery also permit departmental and regional priorities to be considered in funding decisions.

4.1.3 Consistency with Federal Roles and Responsibilities

Evaluation Issue: Relevance Rating
3. Is the EDF program consistent with federal/departmental roles and responsibilities? Acceptable

The EDF program is consistent with federal/departmental roles and responsibilities given the statutory basis of the program and EC’s mandate articulated in the Department of the Environment Act.

4.2 Performance

4.2.1 Achievement of Intended Outcomes

Evaluation Issue: Performance Rating
4. To what extent have intended direct outcomes been achieved as a result of the EDF program? Acceptable

There is evidence that the EDF program is addressing intended direct outcomes, including the promotion and use of EDF as a creative sentencing option by the enforcement and legal communities. Interest in EDF calls for proposals is strong among potential funding recipients and program officials provide support to ensure quality proposals are received. Scientific and technical reviews of proposals and oversight ensure projects are implemented in accordance with funding agreements.

Increased promotion of the EDF by enforcement officials as a sentencing tool to Crown prosecutors - Acceptable

Increased use of the EDF by Crown prosecutors in sentencing recommendations, plea bargains and negotiated settlements - Acceptable

Recognition by judges of the value of the EDF for environmental restoration and wildlife conservation - Unable to assess

Increased number of quality project proposals submitted - Acceptable

Projects implemented in accordance with funding agreements - Acceptable  

Evaluation Issue: Performance Rating
5. To what extent have intended intermediate outcomes been achieved as a result of the EDF program? Acceptable

Program data suggest an increased use of the EDF as a sentencing tool. There is evidence that the program is funding projects in the priority areas of restoration, environmental quality improvement and education and awareness. Performance targets for these projects are being exceeded. It is not possible to assess the program’s contribution to increased knowledge at this time as only limited performance information was available and a new national funding component for R&D projects has only recently been implemented.

Increased use of EDF as a sentencing tool - Acceptable

Table 4.2: Number and Value of Court Awards by Region (2008-2012)

Number of Court Awards
Region 2008 2009 2010 2011 2012 Total Percent
Atlantic 5 1 2 3 3 14 15%
Quebec 0 0 1 2 2 5 5%
Ontario 6 7 8 11 4 36 38%
PNR 2 8 7 7 9 33 35%
PYR 2 1 2 2 0 7 7%
Total 15 17 20 25 18 95 100%
Value of Court Awards ($000)
Region 2008 2009 2010 2011 2012 Total Percent
Atlantic 78 5 50 90 153 376 13%
Quebec 0 0 5 9 42 56 2%
Ontario 62 126 68 121 75 452 16%
PNR 28 79 669 500 499 1,775 62%
PYR 34 14 57 117 0 222 8%
Total 202 224 849 837 769 2,881 100%
  1. Source: EDF Management Information System.

Restoration of natural environments affected by environmental damage and improved environmental quality in areas where environmental damage or harm to wildlife occurred - Acceptable

Table 4.3: Environmental Performance Indicators: 2008-2009 to 2012-2013
Performance Indicator Number of Projects % of Target Achieved
Results of Completed Projects - Goal Results of Completed Projects - Actual
Number of hectares where restoration activities have been implemented 16 814 970 119%
Number of hectares of natural environment created or enhanced 17 2,050 2,404 117%

Increased knowledge of long-term and/or cumulative impacts of environmental damage or harm to wildlife - Unable to assess

Increased awareness and understanding related to environmental restoration and/or compliance with environmental regulations - Acceptable

Table 4.4: Participation Performance Indicators: 2008-2009 to 2012-2013
  Performance Indicator Number of Projects % of Target Achieved
Results of Completed Projects - Goal Results of Completed Projects - Actual
Participants attending project activities 23 7,946 10,098 127%
Target audience that confirmed modification in behaviour as result of project activities 9 2,084 2,403 115%
Evaluation Issue: Performance Rating
6. To what extent have intended final outcomes been achieved as a result of the EDF program? Acceptable

Although an assessment of these outcomes is less directly measurable and therefore relies to a much larger extent on subjective feedback, EDF-funded projects are generally thought to be leading to improvements in environmental quality and prevention of future incidents. However, these impacts are of a very local nature given the small size of EDF projects.

Environmental quality in affected or similar areas is comparable to pre-incident conditions - Acceptable  

Prevention of future incidents of environmental damage or harm to wildlife - Acceptable

Evaluation Issue: Performance Rating
7. Have there been any unintended outcomes of the EDF Program, positive or negative? Not applicable

Progress toward achievement of intended outcomes is being achieved without unintended negative outcomes. Reported positive unintended outcomes relate to increased partnerships and the economic impacts of projects and restored natural environments.

Overall, the majority of unintended outcomes of the EDF program reported by key informants or in the project final reports were positive. A few key informants and the review of project final reports noted that, although partnerships and leveraged funding are not a condition of program funding, almost all projects are very successful in establishing partnerships, which leads to sustainability (discussed in more detail in Section 4.2.2). Other examples of unintended outcomes mentioned by key informants are employment and the economic impacts of EDF projects, including the employment of summer students and the impact of a restored natural environment for economic activities (e.g., tourism).

4.2.2 Efficiency and Economy

Evaluation Issue: Performance Rating
8. Are effective governance and accountability structures in place to guide and coordinate EDF priorities and activities? Attention required

Governance of the program has undergone a series of changes during the study period and during the conduct of the evaluation. Challenges around program management and governance relate to the national mechanism for funding R&D projects, dual location of program responsibility, and some lack of clarity in roles and responsibilities between the national coordinating unit and regions. Engaging partner departments at the regional level is inconsistent and, at the national level, the relationships are underdeveloped with partners who wish to participate more fully but find the tools and guidance (e.g., clear procedures for directing fines, etc. to the EDF) lacking for them to do so. Other program management issues include: 1) the requirement for tracking of financial awards which is a necessary, but burdensome and largely unrecognized activity now undertaken by the regions; and 2) the MIS system and performance measures that do not fully meet the reporting requirements and decision-making needs at the program level.

Governance and Management

Context. Between late 2009 and early 2014, a national coordination unit within the ESB was in operation, based in the National Capital Region (NCR). The coordination unit was responsible for national EDF management, promotion, national consistency of program implementation and delivery, maintaining the MIS, and overseeing the performance of national program delivery. Since 2012, the NCR national coordinating unit was also responsible for project selection and approval, funding agreements and project monitoringFootnote 28and evaluation of the new national R&D component. The regions were (and remain) responsible for disbursement of EDF funds. In early 2014, as the evaluation was underway, there was a change in the EDF management structure; the ESB is no longer involved with the management of the program and national coordination of the EDF and responsibility for the national R&D funding component were transferred to the office of the RDG - Atlantic and Quebec Region.

Among the minority of respondents who have been involved in and favour the new national mechanism, they feel the national process holds promise to fund leading edge research with greater impact and with a stronger tie to federal priorities and research gaps related to restoration by funding larger-scale projects approved by a national research authority (NSERC).

Engagement and Collaboration with Partner Departments

Protocols/Roles and Responsibilities for Financial Transfers to EDF

Performance Measurement and Reporting

Evaluation Issue: Performance Rating
9. To what extent has progress been achieved in the implementation of the 2009 EDF Management Framework? Opportunity for improvement

There has been some progress in implementation of the 2009 EDF Management Framework, though this has been impeded by changes to the management of the program and broader departmental changes. A deficiency in program administrative resources, identified in the 2009 framework, has been addressed as the EDF program had more administrative resources during the evaluation time frame than in previous years. On the other hand, enhancing national consistency in some program processes and increasing promotion of the program to key constituencies remain outstanding items for the implementation of this framework.  

Evaluation Issue: Performance Rating
10. Is the EDF Program undertaking activities and delivering products in the most efficient manner? Opportunity for improvement

Results related to program efficiency are mixed. At the project level, leveraging of EDF funding is strong and the majority of funded projects are also sustainable in at least some respect. The administrative ratio of the program is higher than that for some EC Gs&Cs programs recently evaluated, though this can be partially explained by the additional work of the program in assessing and tracking financial awards. Key informants, however, noted several aspects of the program that detract from efficiency, including internal processes related to assessment of funds even for small-value awards, and inefficient approval and reporting processes (related to the former dual responsibility centres for the program). Moreover, as discussed earlier under evaluation question 8, roles and responsibilities for the transfer and tracking of EDF financial awards are currently unclear which leads to inefficiencies. The efficiency of the national R&D component was praised by some and questioned by others.

Leveraging

Table 4.5: Leveraged Funding for Completed Projects: 2008-2009 to 2012-2013
Total Project Value Average (average $ value across projects) Percentage Total across all projects (sum of $ across all projects) Count (count of projects that have a contribution)
EDF contribution $17,637.66 29% $1,269,911.23 72
Table 4.5b: Leveraged Funding for Completed Projects: 2008-2009 to 2012-2013 - Partner contribution
Total Project Value Average (average $ value across projects) Percentage Total across all projects (sum of $ across all projects) Count (count of projects that have a contribution)
  Cash $24,229.02 29% $1,259,909.00 52
  In-kind $27,359.41 42% $1,860,440.00 68
Total $69,226.09 100% $4,390,260.23 72

Administrative Ratio

Table 4.6: EDF Expenditures and Administrative Ratios, 2008-2009 to 2012-2013
  2008-2009 2009-2010 2010-2011 2011-2012 2012-2013 Total
Salaries 43,928 35,779 96,399 289,299 250,862 716,267
O&M 37,708 74,037 63,631 43,794 52,765 271,935
Total Direct Program Cost 81,636 109,816 160,030 333,093 303,627 988,202
Awards Received 220,364 306,200 998,934 756,639 706,328 2,988,465
Funding Disbursed 594,880 446,345 652,522 371,570 714,671 2,779,988
Standard Administrative Ratio
(total direct program cost/funding disbursed)
0.14 0.25 0.25 0.90 0.42 0.36
Adapted Administrative Ratio
(total direct program cost/(awards received+funding disbursed)
0.10 0.15 0.10 0.30 0.21 0.17
Total Program Cost $676,516 $556,161 $812,552 $704,663 $1,018,298 $3,768,190
  1. Figures on salaries, O&M, awards received and funds disbursed from EC's financial system, as provided by Finance Branch.  

Processing of Awards

Table 4.7: EDF Revenues and Expenditures, 2008-2009 to 2012-2013
Fiscal Year  Opening Balance  


(A)
Revenues
(Awards Received)

(B)
Total Budget
(Funding
Available)
(C=A+B)
Expenditures (Funding Disbursed)


(D)
Closing Balance  



(E=C-D)
% Expenditures /Budget  


(F=D/C)
2008-2009 2,260,160 220,364 2,480,524 594,880 1,885,644 24%
2009-2010 1,876,865 306,200 2,183,065 446,345 1,736,720 20%
2010-2011 1,736,720 998,934 2,735,654 652,522 2,083,132 24%
2011-2012 2,083,133 756,639 2,839,772 371,570 2,468,201 13%
2012-2013 2,468,201 706,328 3,174,529 714,671 2,459,858 23%
  1. Figures from EC’s financial system, as provided by Finance Branch.

Sustainability of Projects

Evaluation Issue: Performance Rating
11. Is the program achieving its intended outcomes in the most economical manner? Acceptable

Notwithstanding the challenges with the efficiency of the current program design, the overall model of the EDF program as a GoC program delivered by EC is widely held to be economical, with many advantages.

Evaluation Issue: Performance Rating
12. Moving forward, what are the lessons learned as a result of the EDF Program? Not applicable

Lessons learned point to potential areas for improvement in the management of the program to ensure the continued confidence in the EDF of the enforcement and legal communities, as they play an important role in the ongoing viability of the program.

5.0 Conclusions

Relevance

Achievement of Intended Outcomes

Efficiency and Economy

6.0 Recommendations and Management Response

The following recommendations are directed to the RDG Atlantic and Quebec Region, as the senior departmental official responsible for the management of the EDF.

Recommendation 1: Clarify roles and responsibilities related to national coordination and program management to ensure that challenges in governance and management are being addressed.

The evaluation identified a number of challenges related to governance and management, including a lack of national consistency in program delivery across regions and a need for clarification of the responsibilities of the national coordination function and the regions delivering the program for key activities. The transfer of responsibility for national coordination to the Atlantic and Quebec region has occurred very recently and the program is now undergoing the transition in governance and roles and responsibilities. While undergoing this transition, EDF senior management should ensure:

Statement of Agreement/Disagreement with the Recommendation
The RDG Atlantic and Québec Region agrees with the recommendation.

Management Action

Transfer of national coordination to RDG Atlantic and Québec region provides an opportunity to address these issues. The national coordination staff in RDGO Atlantic and Québec Region will lead meetings with regions to address this recommendation.

Implicit in the transfer and elimination of the Environmental Stewardship Branch from the EDF is a streamlined approval process.  Approval processes will be documented.

Roles and responsibilities will be clarified especially as they relate to national coordination and regional delivery, including but not limited to awards tracking, information management, and awards assessment.

Opportunities for increasingly consistent program delivery will be identified and implemented as appropriate, taking into consideration regional uniqueness. This will build on regional initiatives already underway in the amalgamated regions (West and North and Quebec and Atlantic) to harmonize and increase consistency of program delivery. 

The agreed upon approval processes, roles and responsibilities, and delivery practices will be documented and integrated into the EDF Operational Guidelines, as well as the EDF Management Framework.  These will be approved once all revisions arising from the Evaluation are finalized in these documents.

Timeline Deliverable(s) Responsible Party
December 2014 New approval processes are documented and communicated to regional managers RDG Atlantic and Québec
December 2014 National coordination and regional roles and responsibilities are documented and communicated  RDG Atlantic and Québec
 March  2015 Recommendations for improving program consistency  RDG Atlantic and Québec
December 2015 Revised EDF Operational Guidelines RDG Atlantic and Québec
December 2015 Revised EDF Management Framework RDG Atlantic and Québec

Recommendation 2: Clarify protocols, roles and responsibilities for the transfer and tracking of court-ordered awards for purposes of program management.

There is currently a lack of protocols to ensure the effective and timely transfer of awards made by the courts to the EDF and their deposit in program regional accounts, and a lack of clarity and understanding of roles and responsibilities among those involved (e.g., provincial courts, enforcement officials, PWGSC and EC). This creates inefficiencies for the program in tracking the source of awards and difficulties in guiding partner departments as to how they may participate in the program, and can result in a lack of timeliness in the receipt of funds. Clear protocols should be developed and roles and responsibilities clarified to ensure that this important aspect of the program can be managed more effectively.

Statement of Agreement/Disagreement with the Recommendation
The RDG Atlantic and Québec Region agrees with the recommendation.

Management Action

EDF will engage in a series of meetings with EC Legal services, EC Enforcement Branch, EC Finance, PWGSC and other government departments with legislation that directs funds to the EDF to determine and document how funds currently find their way to the EDF.

New protocols will be jointly developed by those involved in awards reception and tracking activities and documented. Once the protocols are in practice, the movement of awards will be recorded and tracked in a spreadsheet, and eventually in the new information management system.   These activities will be documented.

The protocols will also be integrated into the EDF Operational Guidelines and Management Framework and approved once all revisions arising from the Evaluation recommendations are finalized in these documents.

Timeline Deliverable(s) Responsible Party
October 2014 Current practices are recorded and communicated RDG Atlantic and Québec
December 2014 New protocols are documented and  communicated RDG Atlantic and Québec
December 2014 Implementation - awards are tracked based on new protocols RDG Atlantic and Québec
December 2015 Implementation of new information management system RDG Atlantic and Québec
December 2015 Revised EDF Management Framework RDG Atlantic and Québec
December 2015 Revised EDF Operational Guidelines RDG Atlantic and Québec

Recommendation 3: Improve program promotion and communications to enhance awareness and understanding of the EDF.

The evaluation evidence indicates that there is a need to improve awareness and understanding of the EDF among all stakeholders, particularly the enforcement and legal communities. This would include explaining the EDF’s goals and results and highlighting success stories. Improved communications (e.g., in an annual report, brochure and newsletters) would help to ensure that all stakeholders have an adequate understanding of the program. 

Statement of Agreement/Disagreement with the Recommendation
The RDG Atlantic and Québec Region agrees with the recommendation.

Management Action

The program will develop a communications plan to promote the EDF to the judiciary, EC Enforcement Branch, OGD partners, potential recipients and the general public.  Program goals, operations, and achievements will be a focus. 

The plan will identify best vehicles for communication - reports, fact sheets, brochures, web, training, meetings, etc. at the national and regional levels. Sharing of best practices between regions will inform the plan. Implementation timelines will be identified. Certain EDF promotional materials are known to be in need of revision, for example due to changes in Legislation the website and regional fact sheets require updating.

The EDF will develop a report to convey results achieved which can be updated annually.  The information will form the basis of communication with different audiences as articulated in the communications plan. 

Timeline Deliverable(s) Responsible Party
October 2014 Develop EDF Communications Plan RDG Atlantic and Québec
November 2014 EDF report completed RDG Atlantic and Québec
December 2014 Update EDF fact sheets RDG Atlantic and Québec
October 2015 Review and update EDF website including program changes and success stories RDG Atlantic and Québec

Recommendation 4: Clarify and communicate the national funding mechanism and use of the NSERC MOU for the R&D component to ensure that it maintains the confidence of the enforcement and legal communities in the program and permits R&D projects to move forward.

The national R&D category has not rolled out as anticipated. Regional program staff finds the assessment of funds process to be burdensome and there are challenges in identifying court awards that are suitably directed from the regions to the national R&D funding category while complying with the original court order and the spirit of the program. As a result, initial targets for funding of this category (40 per cent of program funding) have fallen short. Yet, it has also been suggested that the national funding approach holds promise for funding larger, higher impact R&D projects, using the expertise of a national science granting agency. A review of the MOU is scheduled for 2015. In the interim, EDF program management should review and clarify how EDF funds can be more efficiently assessed and directed to a national R&D component and the potential role of the regions to continue to fund R&D under certain circumstances.

Statement of Agreement/Disagreement with the Recommendation
The RDG Atlantic and Québec Region agrees with the recommendation.

Management Action

The R&D component will be re-examined with relevant stakeholders to determine the most efficient and effective approach to deliver R&D in support of restoration, both at the local level and with the NSERC MOU at the national level.  It will examine challenges and choices such as:

The approach will be in place prior to the official review of the NSERC MOU in late 2015. The agreed upon approach will be communicated to the judiciary, and other relevant stakeholders.  It will be guided by the Communications plan, and may include meetings or information materials.   Engagement with NSERC officials will form a key part of the Communications plan in order to explain and clarify expectations in advance of the review of the MOU.  Ultimately it will be integrated into the EDF Management Framework and EDF Operational Guidelines and approved once all revisions arising from the Evaluation recommendations are finalized in these documents.

Timeline Deliverable(s) Responsible Party
June 2014 Recommended R&D approach RDG Atlantic and Québec
November 2014 Communication to internal partners RDG Atlantic and Québec
November 2014 Communication to external stakeholders (informational material) RDG Atlantic and Québec
December 2015 Revised EDF Management Framework RDG Atlantic and Québec
December 2015 Revised EDF Operational Guidelines RDG Atlantic and Québec

Recommendation 5: Refine the program logic model and performance indicators to reflect the objectives and delivery of the program.

Some shortcomings in the program’s logic model were in evidence, including findings which indicate that the logic model: 1) does not adequately reflect the work of the program in the tracking of financial court awards and ensuring that court restrictions are respected in the disbursement of funds; and 2) incorrectly suggests that EDF funds are used to clean up and/or restore the site where the environmental damage or harm to wildlife occurred and to ensure environmental quality is comparable to pre-incident conditions. In fact, the EDF award is typically made in addition to this clean-up work. The logic model and performance indicators should be re-examined with a view to better reflecting the activities, scope and intended outcomes of the program.

Statement of Agreement/Disagreement with the Recommendation
The RDG Atlantic and Québec Region agrees with the recommendation.

Management Action

The program recognizes the need to revise the program logic model and intended outcomes of the program, and to review performance indicators to ensure that they are measureable and aligned with goals. 

Working with the indicators since their finalization in 2010 has allowed the program to see where they may need modification. In addition, external factors such as the 2010 Environmental Enforcement Act that resulted in changes to legislation used to direct awards to the EDF has impacted the usefulness or accuracy of certain indicators and outcomes. 

The Logic Model and Performance Management Framework will be reviewed to address, for example: 

Page details

2018-12-06