Evaluation of the Environment and Climate Change Canada components of the Oceans Protection Plan

Acknowledgements

The Evaluation Project Team would like to thank the individuals who contributed to this project. In particular, we would like to acknowledge the contribution of ECCC’s Oceans Protection Plan (OPP) Secretariat and the program representatives who provided input and comments to inform the findings and recommendations of this evaluation.

This report was prepared by the Evaluation Division of the Audit and Evaluation Branch, Environment and Climate Change Canada.

This report was approved by the Deputy Ministers on February 23, 2021

List of tables

• Table 1: the five sub-initiatives of the Oceans Protection Plan. 3
• Table 2: key ECCC accomplishments as of September 2019. 4

List of acronyms and abbreviations

• ADM: Assistant Deputy Minister
• ARM: Alternative response measure
• CCG: Canadian Coast Guard
• CSFB: Corporate Services and Finance Branch
• CWS: Canadian Wildlife Service
• DFO: Department of Fisheries and Oceans
• DG: Director General
• EB: Enforcement Branch
• ECCC: Environment and Climate Change Canada
• EPB: Environmental Protection Branch
• GBA+: Gender-based Analysis Plus
• MSC: Meteorological Service of Canada
• NRCan: Natural Resources Canada
• OPP: Oceans Protection Plan
• PIAMSB: Indigenous Affairs and Ministerial Services Branch
• STB: Science and Technology Branch
• TC: Transport Canada

Executive summary

Context

This evaluation report presents the mid-term results concerning Environment and Climate Change Canada’s (ECCC) components of the Oceans Protection Plan (OPP). The evaluation period covered was from April 2017 to September 2019. The evaluation examined:

• the extent to which ECCC’s OPP components were implemented as planned
• the effectiveness of the OPP governance structure and mechanisms within ECCC
• other relevant aspects, such as performance information, and implementation challenges

ECCC received $69.5 million over five years (fiscal year 2017 to 2018 to fiscal year 2021 to 2022) to conduct activities under five OPP sub-initiatives related to the State-of-the-art Marine Safety pillar:

• Marine Weather Information Services Demonstration Project
• Baseline Environmental Data on the Northern British Columbia Coast
• Regional Response Planning
• 24/7 Emergency Response Capacity
• Alternative Response Measures

Findings and conclusions

Implementation status

This evaluation found that although delays occurred with activity implementation over the first half of the OPP, ECCC did complete approximately half of its outputs. While four of the five sub‑initiatives were on track to deliver the remaining outputs by the end of the five years, the Alternative Response Measures (ARM) sub-initiative did not complete its outputs as planned and is at risk of not completing them at the end of the OPP.

Governance structure and mechanisms

Overall, ECCC established governance mechanisms to support the implementation of the OPP, which enabled the reporting of OPP sub-initiatives within the Department. However, some areas for improvement were identified. For example, the ECCC OPP Secretariat could play a stronger role in its oversight function and further challenge the information provided by program areas involved in the OPP within ECCC.

Performance measurement and Gender-based Analysis Plus

ECCC has collected performance measurement information at the output level for the five sub-initiatives and has measured progress at the outcome level. However, there is a need to better articulate how Gender-based Analysis Plus (GBA+) can be considered in the context of the OPP. Up to the OPP mid-term, a full GBA+ analysis had not been conducted to assess whether any measures or strategies would be necessary to ensure the participation of specific population groups in the context of the OPP.

Recommendations

Two recommendations are directed to ECCC’s Associate Assistant Deputy Minister (ADM) of Environmental Protection Branch, as the senior departmental official responsible for the OPP Secretariat.

Recommendation 1: The ECCC Oceans Protection Plan Secretariat and Environmental Protection Branch senior managers should provide more strategic oversight, be more proactive in their coordination efforts and provide a challenge function on information provided by program areas.

The ECCC OPP Secretariat currently provides a coordination function for ECCC branches involved in the OPP. The Secretariat should be more proactive by coordinating joint activities (for example, across programs/branches) and crosscutting issues (for example, Indigenous engagement, funding reallocation). For example, current reporting tools could be better leveraged for the purposes of providing oversight. When required, information provided to the Secretariat should be scrutinized and challenged by both the Secretariat and EPB senior managers.

Recommendation 2: The ECCC Oceans Protection Plan Secretariat and Environmental Protection Branch senior managers should collect and analyze data on Gender-based Analysis Plus to ensure that vulnerable populations participate equally in the OPP.

There is a need to better articulate how GBA+ can be considered in the context of the OPP and to collect data on activities undertaken to address these considerations. The Secretariat should leverage their lessons learned to date and track, for example, how women and Indigenous people participate in the OPP.

1. Context

This mid-term evaluation report presents findings related to Environment and Climate Change Canada’s (ECCC) activities under the Oceans Protection Plan (OPP) for the period from April 2017 to September 2019. The evaluation examined:

• the extent to which ECCC’s OPP components were implemented as planned
• the effectiveness of the OPP governance structure and mechanisms within ECCC
• other relevant aspects, such as performance information, and implementation challenges

For the purposes of this evaluation, different methodologies were applied, including a review of key documents, financial and administrative data and 22 interviews with 37 ECCC staff involved in the OPP (see Appendix A for more details).

1.1. The Oceans Protection Plan

The OPP is a five-year $1.5 billion investment aimed at building a world-leading marine safety system to protect Canada’s marine ecosystems. Launched in fiscal year (FY) 2017 to 2018, the OPP is a comprehensive and transformative whole-of-government strategy. It is composed of 58 sub-initiatives grouped under four pillars, each targeting a different expected outcome.

Transport Canada (TC) is leading the implementation of the OPP, in collaboration with Fisheries and Oceans Canada (DFO), the Canadian Coast Guard (CCG), Natural Resources Canada (NRCan) and Environment and Climate Change Canada (ECCC). 

ECCC’s activities in the OPP fall under the State-of-the-Art Marine Safety System Pillar. Activities under this pillar aim to better position Canada’s marine safety system to prevent and respond to marine safety and pollution incidents.

1.2. ECCC’s activities under the State-of-the-Art Marine Safety System Pillar

ECCC received $69.5 million over five years (FY 2017 to 2018 to FY 2021 to 2022) to advance the objectives of the State-of-the-art Marine Safety Pillar. ECCC’s funding  is being used to conduct activities within this pillar. While there are seven ECCC branches^{Footnote 1} involved in the implementation of the OPP, the Environmental Protection Branch (EPB) is the lead on ECCC’s activities. ECCC coordination functions are carried out through the ECCC OPP Secretariat. Five OPP sub-initiatives involve ECCC:

Marine Weather Information Services Demonstration Project

This project aims to improve marine weather services in high-risk areas by providing more detailed, up-to-date and frequent weather information. Funding is aimed at supporting the deployment of five new buoys to collect weather information. This information, along with the most advanced numerical models, support the development of new forecasts, providing mariners with enhanced weather information such as short-term forecasts of wind speed, wind direction and wave height.

Baseline Environmental Data on the Northern British Columbia Coast

The objective of this sub-initiative is to collect and collate baseline environmental data, including data on marine birds, species at risk and coastal characteristics. This data is expected to be used for preparedness planning and emergency response.

Regional Response Planning

ECCC is mandated to conduct bird population studies and to collect and coordinate scientific data and information on environmental sensitivities. This information is provided to the CCG and TC to support the evaluation of risks and impacts of ship-source oil spills. In turn, these evaluations are expected to help with emergency planning. ECCC also supports TC and the CCG in advancing regional emergency response planning based on risk and stakeholder engagement in the Northern Coast of British Columbia.

24/7 Emergency Response Capacity

To enhance its emergency response capacity, ECCC received funding to hire additional environmental emergency officers on the Pacific and Atlantic coasts, enforcement officers in British Columbia, wildlife biologists and emergency communications staff, as well as to develop 24/7 oil spill modelling.

Alternative Response Measures

This sub-initiative is aimed at supporting the development of a legislative and operational framework that expands the options of effective emergency response techniques (for example, spill treating agents, in-situ burning, decanting and oil translocation). These alternative response measures are expected to complement conventional mechanical measures, to enhance clean-up and ultimately achieve a net environmental benefit. In addition, ECCC is conducting scientific research on the use of alternative response measures (ARM) to support the operational use of ARMs.

As shown in the Table 1, of the five sub-initiatives, two are led by ECCC and three by the CCG, in collaboration with other departments.

*Regional Response Planning: funding was allocated from April 2017 to March 2019.
**24/7 Emergency Response Capacity: $5 million ongoing
***Alternative Response Measures: $4.5 million ongoing

2. Findings

2.1. Progress made in implementing OPP activities

As outlined in the program authorities, ECCC had planned to complete 66 outputs through its five sub-initiatives. At the mid-term, ECCC had completed 31 outputs, which represents about half of the planned outputs. Four of the five sub‑initiatives were generally progressing within the planned timelines and were on track to complete their remaining outputs at the end of the five-year OPP funding period.

The ARMs sub-initiative experienced significant delays with the tabling of legislative amendments. As a result, none of its seven planned outputs were completed at mid‑term. That said, as shown in Table 2, all five sub-initiatives, including ARMs, resulted in deliverables that contributed to the achievement of OPP’s objectives.

In addition to the achievements accomplished by branches and program areas involved with the OPP, most interviewees noted that ECCC’s Communications Directorate provided an effective enabling function to support OPP-related activities. This included developing communication strategies, drafting communication products in collaboration with TC, sharing OPP announcements and achievements in social medias and tracking the uptake of communication products.

When discussing ECCC’s 24/7 emergency response capacity, some program staff reported that they had experienced IT-related challenges. Some of these challenges included unreliable internet and phone access, as well as a lack of platforms to exchange sensitive information.

2.2. Progress made in deploying allocated funding

According to financial documents, about 80% of approved funding was spent in FY 2017 to 2018 (mainly due to delays in staffing). In FY 2018 to 2019, about 96% of the approved funding was spent due to a reduction in the operating costs of the Next Generation Environmental Simulator of the ARMs sub-initiative. In FY 2019 to 2020, a surplus of $830,000 (that is 7% of the approved annual funding) was forecasted for year-end since the legislative amendments to enable the use of ARMs did not take place as planned.

Based on information gathered during interviews, unspent OPP funding was not reallocated across the sub-initiatives. As a result, ECCC had to invest additional resources to complete some of its planned outputs. In general, additional resources were allocated to Indigenous engagement and capital investments under the 24/7 Emergency Response Capacity sub‑initiative. Additional resources were also allocated to meet increased demands to support partner departments that requested ECCC’s expertise to support scientific research and environmental emergency response capacities.

2.3. Effectiveness of the governance structure

TC, as the lead department for the OPP, developed the overall governance structure and put in place various initiative-specific working groups and senior management committees in which ECCC participated. This evaluation focused only on the effectiveness of the OPP governance within ECCC.

At the departmental level, ECCC established the OPP Secretariat, which is housed within EPB. According to ECCC’s Initiative Charter and Implementation Plan, the Secretariat was expected to coordinate reporting for each of the five sub-initiatives and to exercise a challenge function on information provided. It was also expected to lead biweekly internal meetings to facilitate OPP implementation and information sharing.

During the first half of the funding period, ECCC’s Secretariat coordinated the monthly reporting process to update the OPP Sub-initiative Tracking Sheets. It sent out call letters and consolidated input from the program areas before sharing the tracking sheets with TC. It also led biweekly meetings with participants from all OPP program areas and ECCC’s Communications services.

While acknowledging the enabling function of the OPP Secretariat, interviewees believed that it could play a more proactive role in coordinating joint activities (for example, across programs and branches) and crosscutting issues (such as Indigenous engagement and funding reallocation). There was a belief that the Secretariat could provide greater oversight concerning implementation activities by exercising a challenge function on reported information. Other reported challenges included minimal resources allocated to the Secretariat, employee turnover and a need to improve regional issue awareness.

At the sub-initiative level, ECCC’s governance approaches varied. Some sub-initiatives had more established governance structures, such as working groups, while others collaborated informally using existing relationships or forums.

While roles and responsibilities were described for each sub-initiative at the onset of the OPP via a project charter and implementation plan, interviews revealed that many program staff were not familiar with these documents. Some interviewees reported that they felt that they were working in silos and that there could be more coordination among program areas involved in joint sub-initiatives and outputs. 

At the senior management level, ECCC put in place an OPP Director General Committee. This committee was expected to provide strategic advice, coordination and information sharing related to ECCC’s implementation of the OPP and its environmental emergency roles. It was scheduled to meet every two months. Meetings took place as scheduled during the first year of the five-year funding cycle. However, the committee was less active during the second year and met only on an ad-hoc basis. Interviews suggested that there was a lack of awareness concerning the existence of this committee and that it was perhaps not a useful mechanism to foster senior level discussions.

2.4. Reporting on Oceans Protection Plan implementation

TC required partner departments to update the OPP Sub-Initiative Tracking Sheets on a monthly basis. The tracking sheets were used as a reporting tool to collect and compile information regarding the implementation status of the OPP.

In August 2018, TC revised the OPP Tracking Sheet template and required participating departments to report on OPP outputs (instead of milestones), so that it could track progress on commitments. At the time of conducting this evaluation, ECCC had not completed this exercise because the output list was incomplete. Specifically, not all of ECCC’s expected outputs were included on the tracking sheets.

Interviews revealed that some program staff did not have a good understanding of who should report or how to report on joint outputs involving more than one ECCC program or branch. There was a belief among some of the interviewees that ECCC’s OPP Secretariat should take on more of a challenge function when information is provided to them and target dates are revised.

2.5. Performance measurement

ECCC must have a well-designed performance measurement framework to demonstrate that OPP activities are progressing towards their ultimate outcomes. In time, such performance measurement information will be required to support the TC-led horizontal evaluation.

Under the State-of-the-Art Marine Safety System pillar, ECCC sub-initiatives support the achievement of the following outcome: Canada’s marine safety system is better positioned to prevent and respond to marine safety and pollution incidents. ECCC provides annual input into the OPP Delivery and Expected Results for Horizontal Initiatives: Progress Against Targets tracker. This reporting on program performance indicators assists in determining if initiative targets are on track. A review of available performance information showed that ECCC has also been collecting information at the output level through monthly updates to the OPP Sub‑Initiative Tracking Sheets.

2.6. Gender-based Analysis Plus

Program documents indicate that the OPP aims to address the underrepresentation of women in the marine sector. This is being accomplished through a broader initiative that aims to increase engagement, partnerships and training opportunities for Indigenous, coastal and Northern groups, including women. The documents also mention that OPP participating departments intend to pursue measures and strategies, as necessary, to ensure that both women and men will be able to participate equally in the OPP.

Although GBA+ was considered as part of the program development process, a full GBA+ analysis was not conducted. Interviewees indicated that there was a lack of understanding on what GBA+ meant in the context of the OPP and its implementation.

3. Conclusions, recommendations and management response

3.1. Conclusions

Overall, ECCC has made substantial progress in implementing the OPP during the first half of the funding period. At the mid-term, ECCC had completed about half of its planned outputs. With the exception of the ARMs, all of ECCC’s sub-initiatives were on track to complete their outputs by the end of the five-year funding period.

ECCC established a governance structure at various organizational levels to support the OPP. While this enabled the implementation of OPP sub-initiatives, some areas were identified as needing improvement. For example, the ECCC OPP Secretariat and EPB senior managers could play a stronger role in program oversight and provide a challenge function regarding the information submitted by program areas involved in the OPP within ECCC.

ECCC has collected performance measurement information at the output level for the five sub‑initiatives and has measured progress at the outcome level. While it has collected this information, there is a need to better articulate how GBA+ can be considered in the context of the OPP and to collect data on activities undertaken to address these considerations.

3.2. Recommendations and management response

Two recommendations are directed to ECCC’s Associate Assistant Deputy Minister of the Environmental Protection Branch, as the senior departmental official responsible for the OPP Secretariat.

Recommendation 1

Recommendation 1: The ECCC Oceans Protection Plan Secretariat and Environmental Protection Branch senior managers should provide more strategic oversight, be more proactive in their coordination efforts and provide a challenge function on information provided by program areas.

The ECCC OPP Secretariat currently provides a coordination function for ECCC branches involved in the OPP. The Secretariat should be more proactive by coordinating joint activities (for example, across programs/branches) and crosscutting issues (for example, Indigenous engagement, funding reallocation). For example, current reporting tools could be better leveraged for the purposes of providing oversight. When required, information provided to the Secretariat should be scrutinized and challenged by both the Secretariat and EPB senior managers.

Statement of agreement or disagreement

The Associate Assistant Deputy Minister of the Environmental Protection Branch agrees with the recommendation.

Management response

The Environmental Emergencies Division oversees the OPP Secretariat. The OPP Secretariat has over the past year instituted a new governance structure and is putting in place measures to strengthen oversight, program coordination and integration. The Secretariat will also provide a challenge function within the Department. It will ensure effective monitoring, tracking and reporting as well as use governance mechanisms to discuss and resolve issues.

Recommendation 2

Recommendation 2: The ECCC Oceans Protection Plan Secretariat and Environmental Protection Branch senior managers should collect and analyze data on Gender-based Analysis Plus to ensure that vulnerable populations participate equally in the OPP.

There is a need to better articulate how GBA+ can be considered in the context of the OPP and to collect data on activities undertaken to address these considerations. The Secretariat should leverage their lessons learned to date and track, for example, how women and Indigenous people participate in the OPP.

Statement of agreement or disagreement

The Associate Assistant Deputy Minister of the Environmental Protection Branch agrees with the recommendation.

Management response

GBA+ was applied in the planning phase of OPP through an assessment of the general impacts of the program on women and other groups benefiting from or participating in OPP. It was determined that Indigenous people had a desire to be much more involved in the marine oil spill regime, especially in British Colombia, and that all OPP departments needed to find ways to allow greater Indigenous participation.

Work will continue in FY 2021 to 2022 to support the renewal of the OPP. Women and men may participate in the proposed initiatives and activities differently. The Environmental Emergencies Division will design measures to monitor and track impacts or differences in engagement until the end of OPP in March 2023 and use this information to develop engagement and implementation strategies to ensure that both women and men will be able to participate equally in the next phase of OPP. ECCC will also design future OPP initiatives to allow Indigenous people to participate, and where appropriate, help design future OPP objectives and work plans.

Appendix A: evaluation scope, methodology and limitations

Scope

The evaluation was conducted to address a need specified by the senior management to assess the OPP implementation status within ECCC at its mid-term. The evaluation covered the period from April 2017 to September 2019.

The evaluation examined:

• the extent to which ECCC’s OPP components were implemented as planned
• the effectiveness of the OPP governance structure and mechanisms within ECCC
• other relevant aspects, such as performance information, implementation challenges and missed opportunities

Key questions to be addressed by this evaluation are:

• Have the funds been spent as planned? If not, what caused the variances?
• Have the initiatives and sub-initiatives been implemented as planned? If not, what caused the variances?
• Have there been any additional activities conducted in addition to what was committed in the Treasury Board submission?
• Have the performance data been collected according to the Data Strategy specified in the Delivery and Expected Results Annex for the OPP? If not, what were the reasons?
• What are the areas that would require corrective actions to ensure delivery of the remaining OPP components and achievement of the expected results by ECCC?

Approach and methodology

The evaluation used multiple lines of evidence for data collection:

Document review: review of program key documents, including program authorities, implementation plans, internal briefing documents and departmental reports

Financial and administrative data review: review of departmental reporting documents including OPP monthly tracking sheets and updates on OPP financial information

Key informant interviews: 22 interviews were conducted with 37 interviewees from the seven branches, including regions and the Secretariat. The interviewees were mostly with program managers and staff.

Limitations

The evaluation encountered some limitations in relation to data availability and quality, which are common challenges when conducting evaluations. The evaluation team mitigated these challenges by examining multiple lines of evidence.