Evaluation of ECCC’s contribution to the Emergency Management Strategy

List of tables

• Table 1. Planned and actual spending by initiative from 2019-2020 to 2022-2023

List of figures

• Figure 1. Representation of MSC, EMS, the initiatives, and the partners
• Figure 2. Prototype of PACF communication product

Acknowledgements

The Evaluation Division would like to thank the individuals who contributed to this project. In particular, we would like to acknowledge the contribution of program representatives, partners and interested parties who provided input and comments throughout this evaluation.

This report was prepared by the Evaluation Division of the Audit and Evaluation Branch, Environment and Climate Change Canada.

This report was approved by the Deputy Minister on November 6, 2023.

List of acronyms and abbreviations

ADM

Assistant Deputy Minister

CoP

Community of Practice

ECCC

Environment and Climate Change Canada

EMS

Emergency Management Strategy

FHIMP

Flood Hazard Identification and Mapping Program

FPT

Federal-Provincial-Territorial

FTE

full time equivalent

H&H guidelines

Federal Hydrologic and Hydraulic Procedures for Flood Hazard Delineation

ISC

Indigenous Services Canada

MSC

Meteorological Service of Canada

NRCan

Natural Resources Canada

PACF

Predicting and Alerting for Coastal Flooding

PS

Public Safety

PTs

Provinces and Territories

SOREM

Senior Officials Responsible for Emergency Management

Introduction

This report presents the findings from the evaluation of Environment and Climate Change Canada’s (ECCC) contribution to the Emergency Management Strategy (EMS). This evaluation covers the 4-year period from 2019-2020 to 2022-2023 under the following themes: relevance; governance; program design and delivery; effectiveness and results achieved; and use of performance information. The evaluation was conducted as required by the 2016 Treasury Board Policy on Results.

1.1 Initiatives overview

The Emergency Management Strategy (EMS) is a whole-of-government initiative with multiple objectives targeting five priority areas, as approved by federal, provincial, and territorial (FPT) Ministers:

1. Enhance whole-of-society collaboration and governance to strengthen resilience;
2. Improve understanding of disaster risks in all sectors of society;
3. Increase focus on whole-of-society disaster prevention and mitigation activities;
4. Enhance disaster response capacity and coordination and foster the development of new capabilities; and
5. Strengthen recovery efforts by building back better to minimize the impacts of future disasters.

To support these objectives, which are led federally by Public Safety Canada (PS), funding was provided to ECCC for the following two specific initiatives:

• Floodplain mapping ($1.5M^{Footnote 1} ): To modernize best practices for the promotion of consistent floodplain mapping through the improvement of the pan-Canadian Federal Flood Mapping Guideline Series. A national engagement framework for floodplain mapping is also to be established to facilitate common national best practices and increase the sharing and use of flood hazard information.
• Predicting and Alerting for Coastal Flooding (PACF) ($5M): To develop a national prediction system with the capability to generate forecasts and alerts for coastal flooding. This initiative aims to cover all five basins (Great Lakes, Saint Lawrence, Atlantic, Pacific, and Arctic coasts) to support public safety, resilient coastal communities, and safer near-shore marine navigation. The goal is to establish predictions based on a range of potential water levels to enable emergency management communities to better prepare for possible impacts via a thorough understanding of the range of uncertainties and potential effects of a given coastal flooding event.

1.2 Governance and organization

Accountability for the initiatives rests principally with the Assistant Deputy Minister (ADM), Meteorological Service of Canada (MSC) as per the Departmental Results Framework. The initiatives fall under the Core Responsibility “Predicting Weather and Environmental Conditions”, with linkages to both program inventories of Hydrological Services (Floodplain Mapping) and Weather and Environmental Observations, Forecasts and Warnings (PACF). Within the MSC, four Directorates have important roles in the delivery of the initiatives. The roles and responsibilities of each directorate are as follows:

• Monitoring and Data Services Directorate is the lead for the floodplain mapping initiative. It also is the lead for atmospheric and hydrometric data collection, management, and quality assurance, as well as network design and operation. Monitoring and observation of environmental and hydro-meteorological conditions over Canada’s vast and diverse landmass and shoreline is enabled through the MSC’s extensive monitoring and observation networks. Data are also received from international sources and through partnerships with provincial and territorial (P-T) agencies.
• Prediction Services Directorate is the lead for PACF initiative. It is responsible for developing weather related environmental services and information dissemination to meet the needs of a variety of audiences, notably providing weather warnings, forecasts and information to the general public, 24 hours per day, 365 days per year.
• Canadian Centre for Meteorological and Environmental Prediction Directorate is the operational hub where numerical weather and environmental prediction models are developed, tested, and operated, and observation data is ingested into the numerical models to provide the foundation for Canada’s weather forecasts and warnings.
• Policy, Planning and Partnerships Directorate captures the evolving weather and environmental service needs of Canadians and integrates these into MSC’s program planning, strategic directions, and partnerships.

Partners and enablers

To deliver on the PACF initiative, additional support is provided by two ECCC branches: Science and Technology Branch, which leads with original research supporting prediction modelling; and Corporate Services and Finance Branch, which provides expertise in the development of the IT solution to enable program officials to use multiple data sources to generate predictions.

Other federal departments contribute to the PACF initiative, including NRCan (geomatics leadership), Department of Fisheries and Oceans (DFO) (tidal gauge information), Indigenous Services Canada (consultations with Indigenous groups) and PS (overall leadership for EMS).

Finally, provinces and territories are key partners in the development of these products to ensure they meet their needs and the needs of the population they serve. Furthermore, the PACF initiative relies on these partners for key data that are required to properly assess coastal flooding risk. Figure 1 on the following page provides a representation of this.

Figure 1. Representation of MSC, EMS, the initiatives, and the partners

1.3 Resources

Budget 2019 announced funding for Ensuring Better Disaster Management Preparation and Response, for which ECCC received $6.5M for five years, sunsetting on March 31st, 2024.

Annual expenditures managed for the EMS averaged $1.1M between 2019-2020 and 2022-2023. Additional information is provided in section 2.3 of the report.

1.4 About the evaluation

The evaluation of ECCC’s contribution to the EMS is part of ECCC’s Audit and Evaluation Plan 2022 to 2027. This evaluation covers the 4-year period from 2019-2020 to 2022-2023 under the following themes: relevance; governance; program design and delivery; effectiveness and results achieved; and use of performance information.

The evaluation matrix can be found in Annex A.

Multiple lines of evidence were used, including:

• Document and file review: The primary focus of the document review was ECCC’s program specific documents and Government of Canada policy and program documents, complemented by other publicly available sources.
• Analysis of administrative, performance and financial data: Project and performance data on program outputs and outcomes was reviewed.
• Key informant interviews: 15 interviews were conducted with ECCC program representatives that play a role in the delivery of the initiatives.
• Partner interviews: Interviews/meetings were held with three other federal government department representatives, as well as four provinces to further confirm elements that were raised in the consultation notes.
• Participation in face-to-face retreat: Evaluators participated in a three-day retreat organized to advance, review, and demonstrate the latest developments of the PACF initiative. The retreat, which took place in late March 2023, included the 20-plus ECCC members of the project team, and covered all components of the initiative, such as consultations in each PT, IT developments, modelling, training, product testing and implementation, and work planning.

Findings

2.1 Relevance

Climate change is expected to impact flooding in Canada in a number of ways:

• Coastal flooding is expected to increase along most of Canada’s coasts, as local sea-level rise associated with climate change leads to a greater frequency and magnitude of extreme high water-level events.
• Rain-generated flooding in urban areas is expected to become more likely in some areas, as extreme precipitation events are expected to occur more frequently.
• Changes in the frequency and magnitude of flooding related to rivers and streams are difficult to predict, as these floods can be caused by multiple factors, many of which will be impacted by climate change in different ways. (Canada’s Changing Climate Report)

In the context of these changes, there is a need for up-to-date, accurate floodplain mapping. Floodplain mapping is important for adaptation to climate change. It is an essential tool for local decision-making related to flood mitigation, land use planning, and emergency management. It also helps to develop public awareness of flood risk. (ICLR Flood mapping in Canada) In Canada, however, many flood maps are outdated and do not reflect urban development or the impacts of the changing climate. (Flood Risk Mapping in Canada - Policy Brief no.141, National Risk Profile) The use of out-of-date floodplain mapping can constitute a real barrier to understanding and addressing flood risks. It can also lead to continued development and an unfounded sense of security in areas that will be increasingly likely to flood as the climate changes (Building a Resilient Canada).

Additionally, as climate change leads to increased coastal flooding risk, there will be a growing need for ECCC’s flood predictions, forecasts, and alerts. Effective flood warnings can save lives, as highlighted through the World Meteorological Organization “Early Warnings for All" initiative, which aims to ensure that everyone on Earth is protected from hazardous weather, water, or climate events through life-saving early warning systems. Predicting, forecasting, and alerting is an essential part of living with and adapting to floods. However, there is a significant gap between the existing and desired state of Canada's current capabilities related to monitoring and alerting for flooding. (National Risk Profile). For both initiatives, the evaluation found that the integration of new climate change data is a key component in the development of innovative and modern products.

The continued need for ECCC’s involvement to support floodplain mapping and forecasting and alerting for costal flooding is clear. Funding received in 2021 and 2023 to support ECCC’s work in the Flood Hazard Identification and Mapping Program (FHIMP) reflects that. Both Departmental and external interviewees made it clear that ECCC's expertise in hydrology and hydraulics was essential for the floodplain mapping initiative, and weather, climate, and water for the flood forecasting initiative. This expertise and leadership are welcomed by partners, as they allow for the establishment of communities of practice, the sharing of information, and the application of common standards and best practices across Canada. Further, ECCC can supplement areas where a PT does not have the necessary resources or the expertise.

Where jurisdictional concerns were raised by PTs (i.e., inland waters), ECCC was willing to adjust its offerings in accordance with the partner’s needs and preferences, for example by being complementary to what is currently produced. However, this has led to implementation delays (see section 2.3). Finally, the need to ensure clarity related to emergency management roles and responsibilities and ensuring a single authority in the event of an emergency situation was clearly expressed, understood and implemented by all.

2.2 Governance

The EMS, led by PS, has an overall governance structure, with key bodies like the Federal, Provincial and Territorial (FPT) Senior Officials Responsible for Emergency Management (SOREM). While these serve a specific broad purpose, they are outside of the scope of the evaluation.

The evaluation noted a willingness for both initiatives to use existing governance structures and relationships to be as efficient as possible, with the understanding that some new governance and approaches were required to advance the work in certain areas.

For the Floodplain Mapping initiative, the work was led by ECCC staff with extensive experience in administering the National Hydrological Program, who understood how each PT operated from their experiences working in forums like the National Administrators’ Table. Given the intersection of work, it was quickly recognized that the existing committees did not include proper PT memberships to support the advancement of the floodplain mapping best practices. As a result, the program established the Community of Practice for Flood Modelling and Mapping in Canada (CoP). This CoP, initiated in 2021, is expected to meet quarterly and is comprised of members from each of the PTs; ECCC, NRCan, and PS also participate as observers. It aims to facilitate and coordinate knowledge exchanges and collaboration opportunities for the hydrological and hydraulic components of flood mapping studies to meet the needs of governments and Canadians.

For the Predicting and Alerting for Coastal Flooding initiative, the evaluation found that a mix of existing and newly created governance structures were being used. At the onset of the project, a Project Board (essentially a director general decision-making body) and a Coastal Flooding Working Group (essentially a technical group) were established to deliver on the initiative. Both have memberships from the same branches, albeit the former at the DG/Director level, and the latter at the manager/analyst level. Membership of these groups aligns to the ECCC branches and divisions elaborated on in section 1.2, with additional membership from NRCan and DFO.

With the creation of a national prediction system, there is a requirement to use the existing Operational and Parallel Runs Committee, which manages technology transfer changes to meteorological and environmental diagnostics, quality control, verification, analysis and forecasting systems. This ensures robust testing and review prior to implementation.

The governance for the PACF with regards to engaging with the PT partners was different. There was a willingness to leverage existing relationships that ECCC staff had built over the years, whether they were from the emergency management or the meteorological side. This resulted in efforts to reach out to all PTs with important coasts (i.e., only Alberta and Saskatchewan were excluded), to engage and facilitate their input in the process. The outcomes of this approach have been varied, in part due to organizational differences across jurisdictions (centralized structures such as in Prince Edward Island (PEI) appear to have facilitated obtaining information easier than in larger, decentralized structures), as well as varying needs and capacity across PTs. We have also seen and heard about the reluctance by some provinces that perceive the PACF product as a federal intrusion in fields of their authority over internal waters and/or over the issuance of warnings. Efforts are being made to find approaches that will be satisfactory for both parties. Possible approaches include directly providing numerical guidance, value-added products and alerting that is consistent and coherent with existing provincial or territorial practices.

The integration of the Indigenous lens has not been a primary driver of the engagement effort, similar to the approach taken with the PTs. Nevertheless, coastal First Nation groups engaged with program representatives from the BC ECCC regional office. Additionally, Indigenous Services Canada (ISC) has been supporting engagement with some First Nations in the Atlantic region from an emergency management standpoint.

Program representatives indicated that the PACF program design, as initiated in 2019, had commitments that were established in a framework that reflected differing priorities as per standards of the time. Evolving priorities and increased focus on indigenous engagement has yielded some initial opportunities to extend the reach of PACF to indigenous communities.

The recent Evaluation of the National Hydrological Services noted the need to improve engagement and collaboration with Indigenous groups in the context of reconciliation. The current evaluation also found a need to look into giving more consideration for Indigenous coastal communities that could have an interest in the products being developed.

For both initiatives, it was viewed that roles and responsibilities were generally clearly defined and contributed to the achievement of results. The establishment of clear leadership for both initiatives was said to have contributed to the completion of the project.

For the PACF, it was further felt that all key partners in the various divisions and branches of the Department knew what they needed to do to contribute, with regular, well-structured meetings in place to keep everyone informed and accountable to deliverables and sharing of documentations through internal shared spaces. Moreover, it was felt that the accountabilities were clear; and where challenges arose, there was an understanding of what actions to take to advance the project, whether at the working group or at the project board level.

2.3 Program design and delivery

Design

We found that the design of both initiatives followed vastly different paths, even if they both stemmed from a willingness to improve upon practices already in place. In the case of the Floodplains Mapping initiative, the design was simply to increase knowledge of the availability of best practices in Canada. This resulted in significant improvements to the majority of chapters of the existing guideline document “Federal Hydrologic and Hydraulic Procedures for Flood Hazard Delineation”. There was also the need to create a community of practice, as described previously, with ECCC staff identifying proper representatives and voices in its establishment and roles, showing flexibility in the process, for example in ensuring wording respected jurisdictional responsibilities.

The design of the PACF originated from an earlier initiative that was in place in the Atlantic region and that had received funding from Budget 2018. Building upon this product using capabilities already available within MSC and ECCC, the PACF’s goal was to add an impact-based layer to the product (i.e., higher risk levels where certain thresholds are met).

The evaluation noted that the FPT governments worked together toward the broad orientations in the Emergency Management Strategy in Canada: Toward a Resilient 2030. There are clear linkages for both initiatives in the strategy. However, beyond that, the evaluation did not find any evidence that the program had consulted the PTs prior to the launch of the PACF initiative. Rather it was assumed that there would be a high likelihood of interest across Canada, with the understanding that the incremental cost and resources needed to build the system for all water basins (as opposed to only in the Atlantic region) was not significantly important. However, four years later, there remain obstacles in securing buy-in from three provinces and territories: Quebec, Ontario, and Nunavut. Reasons include similar products already in place for the St-Lawrence and Great Lakes, some concerns over jurisdictional authorities, and a lack of resources to support the initiative. Efforts through various communication channels were ongoing.

Further to that, there was a recognized reliance on the need for impact-based data to be provided by the PTs and municipalities. These data are essential in determining local impact thresholds (i.e., at a certain water level, infrastructure would be damaged), but securing it is proving very challenging, as there is a lack of uniformity between PTs and municipalities in how they are organized for mapping. In some instances, the data have not been produced and are not available to share. We’ve heard of efforts being made in several areas to obtain the data, but some partners interviewed recognized that this is a systemic issue they are also facing.

To note, in the exchanges we had with partners, they all noted the excellent collaboration of ECCC, with a willingness to adjust to the PTs’ preferences, for instance in working on areas of specific interest or risk, or in adjusting products and processes to meet specific needs and/or timelines.

We therefore conclude that as a best practice, in the future, ECCC should ensure to properly engage stakeholdersbefore launching wide-ranging initiatives like the PACF, to define the needs of end users of products and services and to promote their optimal utilisation. Part of these best practices include continued efforts to identify and confirm stakeholder needs and requirements, co-development and design of products and services, and clearly communicated vision and desired outcomes of the project.

Delivery

We found that to deliver on both initiatives, the Department has maximized the use of existing knowledge and resources to make advancements in the projects, leveraging past experiences gained through the development of weather models, involvement in weather preparedness, administration of the national hydrological program, consultations with partners, and much more. When asked, ECCC staff felt that the project was delivered efficiently.

To deliver on both initiatives, the Department was allocated the sum of $6.5M over the five-year period of 2019-2020 to 2023-2024, of which an estimated allocation of 9.2 full time equivalents (FTEs). The majority of the resources went to the PACF initiative, which then used the allocation to mostly fund positions in different divisions with different expertise (i.e., project manager, product development specialist, software developer, meteorologists). While this baseline funding was provided, MSC also allocated resources beyond this envelope to support the project, as was evident at the Halifax Face-to-Face discussions, where over 25 individuals from the Department actively participated and shared their expertise, whether it was from modelling, software and systems, training, emergency preparedness, partner consultations, or meteorological forecasting lens. Leveraging these in-house resources was important but additional funding dedicated to this initiative could have expedited the progress, as several staff members had competing priorities that inherently took away time from the PACF initiative. Dedicated funding to that end could have resulted in timelier work.

It was noted by one interviewee that the MSC has been very efficient in securing funding to advance the two initiatives under the EMS umbrella, building on the funding from Budget 2018 to transform the NHS. The EMS funding was also essential in making progress and completing important steps leading to the FHIMP, for which ECCC received funding. This program should help further understand climate change impacts and mitigation with regards to flooding.

In interviews, ECCC personnel indicated there was flexibility built into the initiatives to allow the advancing of the projects. The five-year timeline was viewed as appropriate, with some buffer for events like COVID, challenges associated with hiring highly specialized staff, and the inherent challenges of consultations. Furthermore, the fact that MSC teams from different divisions and teams were used to collaborate on projects was an important benefit. Finally, when challenges arose, issues and decisions could be made through the DG-lead project board or the working group.

One aspect that was identified as challenging for the PACF was the number of project documentation requirements to complete to comply with the Departmental Project Management Framework, especially as the process evolved when the project was occurring. This required the completion of several lengthy documents (project charters, risk log, project performance record, etc.). Conversely, the project management requirements were viewed positively by some and ensured all the key project information is in once place. However, having an additional resource to respond to these requirements would have been ideal because it diverted time and resources away from developing the forecasting and alerting products.

Overall, the PACF initiative offers a promising product to help flood forecasting and alerting in coastal areas, with models nearing completion that should help Canadians and emergency organizations provide more lead time in flooding scenarios. However, with uptake and involvement that has varied significantly by PTs, there is very low likelihood that the complete, impact-based flood product will be available for all Canadian coasts, as initially targeted. Nonetheless, the Department expects to have the baseline system in place, with the capability to add the impact data as it becomes available, and support partners in using it as they wish.

With regards to the usage of the funds, the evaluation found that most of the money allocated to the initiative was used over the period. As table 1 shows:

Table 1. Planned and actual spending by initiative from 2019-2020 to 2022-2023

($)	2019-2020	2020-2021	2021-2022	2022-2023
Floodplain Mapping
Planned	93,243	359,789	359,789	359,789
Actual	82,729	142,837	202,156	64,552
Variance	10,514	216,952	157,633	295,237
Variance %	11.2	60.3	43.8	82.1
PACF
Planned	696,790	1,047,430	1,047,430	1,122,430
Actual	689,516	998,135	1,077,095	1,150,636
Variance	7,274	49,295	-29,665	-28,206
Variance %	1.0	4.7	-2.8	-2.5
Total EMS
Planned	790,033	1,407,219	1,407,219	1,482,188
Actual	772,245	1,140,972	1,279,251	1,215,188
Variance	17,788	266,247	127,968	267,031
Variance %	2.3	18.9	9.1	18.0

Excluding the amounts allocated for corporate services, which represented on average 27% of the expended funds, the split was roughly 92% dedicated to salary, and the remaining 8% for Operations and Maintenance (O&M), a fair representation of what we heard. Once the funding sunsets on March 31st, 2024, the expectation is that the PACF will shift into the regular programming of the MSC.

2.4 Effectiveness and results achieved

As stated earlier in the introduction, funding was provided to ECCC under the EMS to support two different projects: 1) Floodplain Mapping ($1.5M); 2) Predicting and Alerting for Coastal Flooding (PACF) ($5M).

Results for floodplain mapping

The funding was set aside to allow ECCC to work in partnership with PTs and various other stakeholders to develop best practices for hydrologic and hydraulic methods in support of flood mapping across Canada.

Concretely, this resulted in a complete overhaul and update to the Federal Hydrologic and Hydraulic Procedures for Flood Hazard Delineation (the H&H guidelines). These are part of the Federal Flood Mapping Guideline Series, which consists of all the components of the flood mapping process, from flood hazard identification and priority setting to the implementation of flood mitigation efforts via land use planning. The H&H guidelines are written for municipal and PT agencies and Indigenous communities working to produce flood hazard maps.

The specific objectives are to:

1. Describe the process that should be expected from practitioners providing technical flood hazard delineation services, including quality management and technical review.
2. Describe different types of flooding that occur in Canada.
3. Provide guidance for practitioners to conduct hydrologic and hydraulic analyses as part of the flood mapping process.
4. Provide guidance on approaches and considerations for incorporating climate change into flood hazard studies.

The update of the H&H guidelines and its 13 chapters improves the presentation of the concepts by using explanatory flow charts, reorganizing some of the contents, providing more information on how to account for climate change impacts, and adding a section on reporting requirements. In discussing with staff, it was indicated that noteworthy additions were made to the majority of chapters, with readability and useability in mind, especially with the addition of the Data Requirements chapter.

The H&H guidelines were reviewed by both internal and external experts in different fields, before being provided to PT members of the CoP for their comments. As of July 2023, the final version is complete. ECCC is awaiting guidance from NRCan, who is the lead of the Federal Flood Mapping Guidelines Series, regarding publishing the document. It is expected to be published in the Fall of 2023. With this new knowledge, the program has already identified areas that could be subject of updates in future iterations, such as expanding on climate change and uncertainty in floodplain management, urban stormwater management, geohazards, and catastrophic events.

Beyond the review of the guidelines, another important component for this initiative was the establishment of the Community of Practice for Flood Modelling and Mapping in Canada (CoP), presented in section 2.2.

Results for predicting and alerting for coastal flooding

The funding for the PACF is to develop a national prediction system with the capability to generate forecasts and alerts for coastal flooding nationally, expanding on the existing program from the Atlantic region, to the remaining basins (Great Lakes, Saint Lawrence, Pacific and Arctic coasts). Furthermore, there is a willingness to include a better understanding of coastal flooding risk in the most vulnerable areas to provide better impact-based warnings. Figure 2 below provides a good snapshot of a prototype of a PACF communication product.

Figure 2. Prototype of PACF communication product

We found that over the evaluation period, work has advanced significantly in the development of the prediction of coastal flooding. In March 2023, the system was being tested and showed promise. A demonstration and exchange on the system was conducted at the face-to-face meeting, allowing staff to discuss scenarios, challenges, and opportunities.

At the current juncture, the initial roll-out is expected to first focus on basic coastal flood forecasting and alerting. The added layer of impact-based forecasting, where thresholds are established to measure impact such as damage to infrastructure or property, disruption to travel, and/or danger to life, is currently not available across all coasts due to challenges in obtaining the necessary data from partners, as noted previously. The task is quite important, as all coastlines need to be properly mapped to understand the elevation data for each infrastructure and an understanding of its function to give it a proper risk threshold.

The province of PEI has been working closely with ECCC to get the required data that will allow for the generation of impact-based coastal flood warnings. Elsewhere in Canada, efforts have been made to collect the information, but it has proved challenging, with less complete datasets. This was explained by both ECCC and external interviewees as a systemic data challenge.

Further to that, as explained earlier, there has been less buy-in from the provinces responsible for the St-Lawrence and the Great Lakes, due to factors that include existing provincial programs that are viewed as responding to their needs, as well as some ongoing jurisdictional concerns. Relationships are still viewed as positive, but advancement on this file is slower than expected, and impact-based coastal flood warnings will not be ready in time for this basin.

For the Arctic basin, some efforts have been made in specific areas, and engagement is necessary with one territory to determine where needs reside. For the Pacific coast, work is ongoing with BC, municipalities and certain Indigenous groups to advance the collection of their data. In the Atlantic region, other than PEI, work is ongoing to gather data, and different strategies are being undertaken, such as giving priority to certain high-risk areas as directed by provincial partners, or where certain regional municipalities have indicated they have a willingness to allocate resources to that end.

2.5 Use of performance information

The evaluation found that initiatives are supported by quality performance information. At no time during the evaluation was the lack of information a challenge, with mechanisms and data sources available to support analysis. As noted in section 2.3, while Departmental Project Management Framework requires a lot of work from staff, it supports the generation of performance information of value for management.

In interviews, senior management indicated that they felt like they have the information they need to support decision-making to advance in their initiatives, with multiple methods to get a pulse of the initiatives, such as the performance dashboards, connecting with the various members on the project, checking against timelines, usage of shared document platforms, and regular meetings of project board and working groups.

In working on the Floodplain Mapping initiative (H&H guidelines), the staff was diligent about ensuring that a proper review process was in place, by first securing an internal review process of scientific experts of the guidelines, followed by a review of the CoP members.

For the PACF, we found that the initiative is built upon the broad expertise and high-performance computing infrastructure of the MSC, which is built upon robust and established processes that support integrity of information. Further to that, some of these processes use ISO standards. Finally, in adding the PACF product, a review by the Operational and Parallel Runs Committee was done, which is the gatekeeper for all major changes to the MSC forecasting products.

We observed that the initial indicator selected for the flood plain mapping initiative was based on a limited understanding and was a misleading representation of the nature of the work (5 best practices by December 2023). Regardless, the update to the H&H guidelines, and to its individual chapters, was performed based on the spirit of the initial indicator. As such, the assessment of performance towards the indicator has been measured as the number of individual technical components that were updated in the overall guidelines rather than the number of individual best practices.

Finally, as the initiatives’ funding will be nearing completion, surveys are expected to be carried out to measure success by 1) asking the CoP about feedback about the updated floodplains guidelines; and 2) survey of users of warnings who will have used the information.

Conclusions

3.1 Conclusions

Relevance

There is a continued need for ECCC’s involvement in floodplain mapping and predicting and alerting for coastal flooding. The science is clear that impacts due to flooding are expected to increase as a result of climate change. ECCC’s expertise in hydrology, hydraulics, weather, and climate allows for leadership in advancing these initiatives and potentially lessening the impacts of these catastrophic events. Both Departmental and external interviewees made it clear that ECCC's expertise and leadership is welcomed by partners across Canada.

Governance

The Department was strategic in using a mix of existing governance structures where possible, and creating new ones where these facilitated the advancement of initiatives. For the floodplain mapping initiative, a new Community of Practice for Flood Modelling and Mapping in Canada (CoP) was established with membership from PTs and other federal departments. For the PACF initiative, new structures like a project board and working group were established to make decisions and support the initiative. With regards to PACF consultations with PTs, reliance on existing relationships built by ECCC was favored, with varying results based on individual PT circumstances. Likewise, Indigenous consultations have not been a primary focus for the initiative, with only some groups involved.

Design and delivery

The design of both initiatives stemmed from earlier work the Department had initiated, in the first case improving an already established guideline and filling a gap by establishing a new CoP, and in the second case expanding and improving coastal flooding predictions and alerts for all. However, it was found that for some of the partners of the PACF, there was limited interest in what the Department is currently proposing, which could have been mitigated by earlier consultations. For the PTs that have embarked in advancing the PACF products, there remains challenges in securing impact-based data typically housed by provinces, territories, and municipalities, which is a systemic issue across Canada. With that said, exchanges between the Department and partners are viewed as very good, with excellent collaboration and flexibility.

With regards to the delivery of both initiatives, the Department has maximized the use of its existing knowledge and resources, on top of the funding received. It built upon the varied expertise of the Department, and its relationships with partners. It also strategically leveraged resources of other budgetary initiatives to advance these files. Finally, at this juncture, there is recognition that there is low likelihood that the complete, impact-based flood product will be available for all Canadian coasts by the end of April 2024, as initially targeted. Nonetheless, the baseline product will be built and is expected to offer a promising product to help flood forecasting and alerting in coastal areas that should help Canadians and emergency management organizations provide more lead time in flooding scenarios.

Effectiveness and results achieved

Both initiatives have so far delivered promising results. For the Floodplain Mapping initiative, a complete overhaul and update to the Federal Hydrologic and Hydraulic Procedures for Flood Hazard Delineation has been undertaken, with partners’ input being integrated and a target publication scheduled for Fall 2023. Furthermore, the Community of Practice for Flood Modelling and Mapping in Canada (CoP) was established as part of this initiative.

For the Predicting and Alerting for Coastal Flooding initiative, work has advanced significantly in the development of the prediction of coastal flooding, with testing taking place, and roll-out expected to first focus on basic coastal flood forecasting. As of July 2023, only the province of PEI had managed to provide the detailed information to allow for impact-based coastal flood warning. Work is ongoing in the other PTs to obtain this information, based on individual PT priority areas.

Use of performance information

Overall, both initiatives were supported by quality information that enabled decision-making. Furthermore, management indicated they had all of the information required to advance their work on the initiative. There were challenges in selecting a proper indicator for one initiative, but this did not hamper progress and the work conducted was in line with the spirit of the initiatives.

Annex A – Evaluation matrix

Themes	Questions	Indicators	Sources/Methods
Relevance	1. To what extent is there a continued need for ECCC’s involvement in a) floodplain mapping, and b) predicting, forecasting and alerting for flooding in Canada?	Evidence of need addressed by the initiatives. Evidence of gaps fulfilled by the federal government. Evidence of linkages to the integrated climate change lens.	Document and File Review Interviews
Governance	2. Has the governance of both initiatives supported the achievement of results?	Evidence of existence of an effective governance structure, roles, internal coordination and roles and responsibilities. Evidence of clear accountability and processes to make decisions. Internal perspectives on clarity of roles and responsibilities, efficiency of internal collaboration in delivery. Evidence of defined governance for ECCC to interact with federal, provincial, territorial and/or Indigenous partners.	Document and File Review Interviews
Program design and delivery	3.1. Is the design of both initiatives appropriate for achieving their intended outcomes? 3.2. Is the delivery of both initiatives conducted efficiently?	Evidence that relevant federal, provincial, and/or territorial partners have been appropriately involved in the design of the initiatives. Evidence that lessons from past initiatives / programs were taken into account in the design of this program. Evidence of flexibilities to adjust the program design and delivery to correct course when unexpected elements occur. Evidence of efficiency in the delivery of the initiatives.	Document and File Review Interviews
Effectiveness/ Results achieved	4. What results have been achieved by the initiatives?	Evidence of achievement (or progress towards) of the expected outcomes from the broader programs’ logic models. Evidence of results achieved (or progress towards) by the initiatives as committed in the TB submission.	Document and File Review Interviews
Use of performance information	5. To what extent do the initiatives have quality performance information (accessible, sufficient and reliable) available to decision-makers and was it adequate to support this evaluation?	Evidence that performance information is accessible, sufficient and reliable for decision-makers and evaluators.	Document and File Review Interviews

Annex B – Reference List

Bush, E. and Lemmen, D.S., editors (2019) Canada’s Changing Climate Report; Government of Canada, Ottawa, ON

Council of Canadian Academies, (2022). Building a Resilient Canada, The Expert Panel on Disaster Resilience in a Changing Climate, Council of Canadian Academies, Ottawa (ON).

Institute for Catastrophic Loss Reduction (2019), Focus on Flood mapping in Canada, Toronto, ON.

Public Safety Canada (2019) Emergency Management Strategy in Canada: Toward a Resilient 2030

Public Safety Canada (2023) National Risk Profile – A national emergency preparedness and awareness tool, Ottawa, ON.

Warren, F. and Lulham, N., editors (2021). Canada in a Changing Climate: National Issues Report; Government of Canada, Ottawa, ON

World Meteorological Organization (2019). Strategic Plan (2020-2023)