Evaluation of Lake Simcoe/South-eastern Georgian Bay Clean-up Fund: Lessons learned

Acknowledgements

The Evaluation Project Team would like to thank the individuals who contributed to this project. In particular, we would like to acknowledge the contribution of members of the Evaluation Committee, as well as the interviewees who provided insights and comments crucial to this evaluation.

The evaluation was conducted in accordance with the 2016 Treasury Board (TB) Policy on Results.  It was identified in the 2015 Departmental Risk-Based Audit and Evaluation Plan.

This report was approved by the Deputy Heads of Environment and Climate Change Canada on March 16, 2018. It is available on the Environment and Climate Change Canada website in both official languages.

The evaluation report was prepared by the Evaluation Division of the Audit and Evaluation Branch.

Executive summary

Background

Lake Simcoe and South-eastern Georgian Bay are in Southern Ontario. Lake Simcoe is located north of Toronto and is the largest lake in Ontario outside of the Great Lakes system. The Lake Simcoe/South-eastern Georgian Bay Clean-Up Fund (the Fund) provided financial and technical support to implement priority projects through contributions to citizens, non-governmental organizations, provincial ministries, conservation authorities, land owners, First Nations, universities and industry. The key components of this Environment and Climate Change Canada (ECCC) program included:

• the management and administration of the grants and contributions (G&C) provided under the Fund to support community-based projects focused on specific priorities such as reducing phosphorous input from urban and rural sources
• research (science and monitoring) in the Lake Simcoe/South-eastern Georgian Bay area

The evaluation of the Fund covered the period from 2012-2013 to 2015-2016, with updated information for 2016 to 2017. The Fund ended on March 31, 2017.

Lessons learned

Based on an analysis of the information gathered during the evaluation project, the evaluation team developed two broad lessons learned to help inform the design and delivery of future similar initiatives.

Governance structure

Clearly define the roles and responsibilities of the governance committees so that they work together to achieve the best results for the program.

• An effective governance structure could include a technical review committee to ensure that recommended projects are aligned with the best available science, and a public advisory committee to ensure that all relevant stakeholders are engaged. As well, it is suggested that programs using this governance model consider the views of both committees in determining recommendations on project funding priorities.

Performance measures

Develop cost-effective performance indicators and a program database that allow the program to track all its key expected results.

• To receive funding, most programs must present a logic model that sets out the expected results. The program database should be able to track more than just the basic project financial and results information. It should also provide data to report on all key expected program results.

Context

Lake Simcoe and South-eastern Georgian Bay are in Southern Ontario. Lake Simcoe is located north of Toronto and is the largest lake in Ontario outside of the Great Lakes system. It supplies drinking water to eight municipalities. As shown in Figure 1, Georgian Bay is connected to Lake Huron and the Great Lakes system and includes the UNESCO-designated Georgian Bay Biosphere Reserve. Both water bodies are major recreational areas that support and generate millions of dollars per year in tourism revenue.

Figure 1: Lake Simcoe\South-eastern Georgian Bay Boundary Map

The water quality and ecosystem health in parts of these areas are under threat from urban and shoreline development and agricultural activities. These activities contribute to excessive inputs of phosphorous, the growth of nuisance and toxic algae and the introduction of invasive species resulting in habitat loss. High levels of phosphorus can starve the water of oxygen, which affects cold-water fish, wildlife and overall water quality.

This report presents the lessons learned from the evaluation of the Lake Simcoe/South-eastern Georgian Bay Clean-up Fund conducted in fiscal years 2015 to 2016 and 2016 to 2017. The Fund ended on March 31, 2017.

The Lake Simcoe/South-eastern Georgian Bay Clean-up Fund

The Lake Simcoe/South-eastern Georgian Bay Clean-Up Fund (the Fund) provided financial and technical support to implement priority projects through contributions to citizens, non-governmental organizations, provincial ministries, conservation authorities, land owners, First Nations, universities and industry. The key components of this Environment and Climate Change Canada (ECCC) program included:

• the management and administration of the grants and contributions (G&C) provided under the Fund to support community-based projects focused on specific priorities such as reducing phosphorous inputs from urban and rural sources
• research (science and monitoring) in the Lake Simcoe/South-eastern Georgian Bay area

Overall accountability for the delivery of activities under this program rested with the Associate Regional Director General (RDG), Ontario Region. Day-to-day operations were delivered from the Lake Simcoe/South-eastern Georgian Bay office in the Ontario Region. The Fund was active from 2012‑13 to 2016-17. Program expenditures over this five-year period totalled $28.9 million.

The objectives of the Fund were to:

• improve scientific understanding of the watersheds for decision making
• rehabilitate priority habitats to restore the health of the aquatic ecosystem and the coldwater fishery

To meet its objectives, the Fund set the following funding priorities:

• enhance research and monitoring to improve environmental information for decision making
• conserve critical aquatic habitat and associated species
• reduce rural and urban non-point source^{Footnote 1}  phosphorus and other nutrients
• reduce point sources of phosphorus and other nutrients

About the evaluation

The evaluation of the Fund covered the period from 2012-13 to 2015-16, with updated information for 2016-17. Management processes, governance structures, G&C and program activities of the Regional Director General’s Office (RDGO) Ontario and the Science and Technology Branch were included within the scope of the evaluation.

The evaluation team gathered information using a variety of research methods, including a review of documents, literature and financial and performance data, an online survey of 36 funding recipients, interviews with 29 key informants and 10 unfunded applicants, a file review of 21 completed projects and four case studies.

Since the Fund ended on March 31, 2017, no recommendations are made for the program. Instead, the evaluation team looked at the elements of the program that worked well and those that could be improved. Based on this analysis, they then developed some lessons learned to help inform the design of future similar initiatives.

What worked well

The Fund supported projects that contributed to the reduction of phosphorus discharges from point and non-point sources. According to program reports, as of March 2017, funded projects reduced an estimated 14,343 kg of phosphorus loadings to the Lake Simcoe and South-eastern Georgian Bay watersheds, thus surpassing the 4,000 kg target reported in the 2016-19 Federal Sustainable Development Strategy for this outcome.

The evaluation team looked at those aspects of the funding program that worked well and allowed it to meet and surpass its stated outcomes.

Increased understanding of reduction needed

The program supported a number of research activities that increased the knowledge and understanding of the phosphorus reduction required to protect water quality and ecosystem health, especially in South‑eastern Georgian Bay. The Science Plan for South-eastern Georgian Bay helped to develop an understanding of the Georgian Bay and the impact of its water levels on nutrients and aquatic habitat. Key informants observed that there had been limited research funds available for South-eastern Georgian Bay prior to the Fund. The Fund resulted in a surge of research activities that have contributed to a better understanding of South-eastern Georgian Bay.

Leveraging funds

The approach used was very successful at leveraging funds from other participants like the Province of Ontario, local governments, conservation authorities and the private sector. The program enabled groups to bring funds to the table and to encourage other stakeholders to support the projects. The review of financial and performance data demonstrated that in the first three funding rounds, the program leveraged $31,288,573 in in-kind and cash contributions, which is 61% of the total value of projects ($51,332,817). This was within the identified target range of $12 million to $49 million in leveraged funds.

Fostering collaboration

Support from the program led to more collaboration and sharing of resources among a variety of partners within the region. Almost all interviewees stated that the available funds enabled groups to work together that had not done so before because of a lack of funds or opportunity. A number of survey respondents noted that the key benefit of collaboration with other partners was the sharing expertise and knowledge among a variety of experts and an increased access to data.

Two-pronged program design

The design of the Fund included a research component and stewardship activities. This two-pronged approach was viewed by almost all interviewees as a good way to address the issues of phosphorus and nutrient loading. The research component was seen as being necessary to gain a better understanding of the watershed and to ensure that the stewardship activities were aligned with current needs. The program design was similar to that of other ECCC programs with similar objectives, such as the Lake Winnipeg Basin Initiative.

Governance structure

The governance structure was clear and appropriate to support the program. The governance model was composed of the Technical Review Committee (TRC), to provide scientific expertise, and the Public Advisory Committee (PAC), to ensure that all relevant stakeholders were involved. The roles and responsibilities of both committees were clearly defined and they operated as planned. The majority of interviewees supported and valued the role of PAC in the project review process. TRC was viewed by most interviewees as a valuable addition to the project review process. It was an effective way to engage stakeholders from other federal departments and for the provincial government to provide input on the best available science.

Documentary evidence suggested that the governance structures related to research were clearly defined for South-eastern Georgian Bay. These structures built on initial efforts to identify priority areas for science and monitoring, and associated information needs. A memorandum of understanding between the RDGO and the Water Science and Technology Directorate was updated annually to confirm commitments for activities undertaken by the directorate under the science component of the Fund.

Program delivery

Program delivery was efficient and carried out in a timely manner.

• Eligibility criteria were clear.
• Time allotted for submitting project applications was sufficient.
• The website provided sufficient and useful information.
• The two-stage application process, namely  a letter of intent followed by a project proposal, was seen as a good approach.
• The project proposal review process was regarded as rigorous and cost-efficient. Although the number of program staff was limited between 2012 and 2016, they were successful in meeting their service standards within the mandate of the RDGO.
• Payment of expenditures was efficient and timely. Financial reporting templates were easy to use and the level of detail requested in the project activity templates was reasonable.

Collaboration and coordination

Given the geographic scope of the program, the RDGO played an important role in fostering collaboration among stakeholders and coordinating activities. Funding recipients viewed positively the level of communication and collaboration with the regional office.

What could be improved

Overall, the program was successful in providing support to a number of projects that were directly focused on the protection and restoration of aquatic habitat and that had some immediate outcomes. However, as with all such activities, there are some aspects that could have been done better.

Developing and measuring outcomes

Work on the Fund was guided by a logic model. A program database was developed to track basic project financial and results information. However, there was no formal performance measurement strategy in place that provided data for reporting on all expected program results. The available performance data was useful to measure key outcomes such as leveraging of funds and reductions in phosphorus inputs, but there was insufficient information to determine whether all intended results had been achieved.

Governance

Most interviewees mentioned that the addition of TRC had a positive impact on the integration of science in the program. ECCC representatives and interviewees who had participated on TRC noted, however, that the project recommendations to the Minister made by PAC sometimes conflicted with those of TRC. In the view of these key informants, this approach does not always ensure that projects recommended to the Minister are aligned with the best available science.

In interviews, ECCC scientists and external experts agreed that the integration of science in the program was improved through the development of the Science Plan for South-eastern Georgian Bay. However, they pointed out that Lake Simcoe didn’t have a similar science plan to guide the science activities.

Communications

The evaluation found that there was a lack of ongoing communication about the program and project results. The Fund provided relatively few communication materials intended directly for policy makers, stakeholders and members of the public who were involved or interested in stewardship-based activities in the Lake Simcoe and South-eastern Georgian Bay watersheds. RDGO couldn’t post the results of ongoing projects. And once the projects had been completed, RDGO couldn’t post project products because they were not ECCC’s property. A number of funding recipients posted their results but the outreach that resulted was usually limited to the local community. The Fund eventually communicated project results in a summary report and at a Science and Stewardship Forum in 2017.

Timeliness of funding approvals

Project approval notices were not always communicated in a timely manner. This proved to be an issue for funding recipients who needed to undertake field work. The case studies showed that the timeliness of the announcement of funding was a challenge, particularly for those projects that involved field work and needed to hire summer students. Some of the case study projects received their notice midsummer, when a number of summer students were no longer available. As a result, many organizations had to risk-manage by hiring students prior to receiving funding approval.

Lessons learned

Based on an analysis of the information gathered during the evaluation project, the evaluation team developed two broad lessons learned to help inform the design and delivery of future similar initiatives. As well, some suggested strategies are included for consideration.

Governance structure

Clearly define the roles and responsibilities of the governance committees so that they work together to achieve the best results for the program.

Suggested strategy

An effective governance structure could include a technical review committee, to ensure that recommended projects are aligned with the best available science, and a public advisory committee to ensure that all relevant stakeholders are engaged. As well, to ensure that a proper balance is maintained and that one committee can’t override the decisions of the other committee, the Terms of Reference for the two committees could specify that the recommendations from both committees on project funding priorities are equally weighted by ECCC program staff in determining the recommendations to the Minister. This would ensure that input from both committees receives due consideration in determining projects to be supported. It would also help to ensure that funded projects are aligned with the best available science.

Performance measures

Develop cost-effective performance indicators and a program database that allow the program to track all its key expected results.

Suggested strategy

To receive funding, most programs must present a logic model that sets out the expected results. The program database should be able to track more than just the basic project financial and results information. It should also provide data to report on all key expected program results.

The project reporting template could be refined to ensure that measurements are better aligned with the expected results stated in the contribution agreement. ECCC’s Corporate Services and Finance Branch has developed a final project reporting template that focuses on the completion of project activities and financial information. Funding programs can use the template as is. Or, with ECCC’s approval, they can adapt it to their specific reporting needs. The template could be modified to include key performance indicators for outcomes specified in project proposals and contribution agreements.

[1] A “point source” refers to a single, identifiable source of phosphorus and nutrients. A “non-point source” refers to inputs and impacts that do not have one specific source, but come instead from the cumulative effect of a number of factors or activities.