Language selection

Government of Canada / Gouvernement du Canada

Search

Evaluation of the Weather Observations, Forecasts and Warnings Program

Evaluation of the Weather Observations, Forecasts and Warnings Program
[PDF - 579 KB]

List of tables

Acknowledgements

The Evaluation Project Team would like to thank all the individuals who contributed to this evaluation. In particular, we would like to acknowledge the contribution of participants and program representatives who provided input and comments to inform the findings and recommendations of this evaluation.

This report was prepared by the Evaluation Division of the Audit and Evaluation Branch, Environment and Climate Change Canada.

The Deputy Ministers approved this report and management action plan on November 10, 2021.

List of acronyms and abbreviations

CCG
Canadian Coast Guard
CSFB
Corporate Services and Finance Branch
DND
Department of National Defence
DRR
Departmental Results Report
ECCC
Environment and Climate Change Canada
FY
Fiscal Year
GBA+
Gender-Based Analysis Plus
IM/IT
Information Management/Information Technology
MSC
Meteorological Service of Canada
N
Number
NPAS
National Public Alerting System
P/T
Provincial/Territorial
QMS
Quality Management System
SSC
Shared Services Canada
STB
Science and Technology Branch
WMO
World Meteorological Organization

Executive summary

Context

This report presents the findings of an evaluation of the Weather Observations, Forecasts and Warnings Program (the Weather Program). The program is delivered by the Meteorological Service of Canada (MSC) of Environment and Climate Change Canada (ECCC), in collaboration with the Science and Technology Branch (STB) and the Corporate Services and Finance Branch (CSFB) of ECCC, as well as Shared Services Canada (SSC). In 2021, ECCC celebrated its 50th year of operation and the MSC celebrated its 150th anniversary of providing weather-related services to Canadians.

The evaluation covered the period between fiscal year (FY) 2015 to 2016 and FY 2018 to 2019. The following program activities were within the scope of the evaluation:

The following program activities were not covered by the evaluation: Meteorological Areas, Air Quality and Health Related Services, and Water and Climate Services.

Due to delays in the start of data collection as a result of the COVID-19 pandemic, some information concerning FY 2019 to 2020 was also captured in this report. The objectives of the evaluation were to examine issues of relevance and performance (effectiveness and efficiency) as per the 2016 Treasury Board Policy on Results, and to identify recommendations for possible improvements in the future.

Findings and conclusions

Relevance

There is a clear ongoing need for the Weather Program. National meteorological services such as the Weather Program play a critical role in supporting the vital functions of governments as well as decision making by individuals, households and businesses, producing a range of societal benefits. Evolving technology, changing consumer preferences, the growth of the private sector, and a shift away from forecasting weather to forecasting impacts in the context of climate change is requiring the Weather Program to adapt in order to remain relevant to its target audiences.

The Weather Program aligns with federal roles and responsibilities, as well as federal priorities relating to climate change, public safety, economic growth and prosperity, science and innovation, and open and transparent government. There is widespread support among internal and external stakeholders to improve services for vulnerable and diverse populations and this could improve program alignment with the federal government’s gender-based analysis plus (GBA+) objectives.

Effectiveness

Key accomplishments of the Weather Program since FY 2015 to 2016 include:

Achievement of outcomes

The Weather Program expects that target audiences will access its weather and environmental services and that they will receive the meteorological information they need. Ultimately, the program expects that Canadians will use weather and related information to make decisions about their health and safety.

There is clear evidence that target audiences are accessing ECCC weather and environmental services. Annual visits to weather.gc.ca increased from 524 million to 585 million between FY 2015 to 2016 and FY 2018 to 2019, while the number of Twitter weather alert followers increased from 83,000 to 190,000 over that same period. Furthermore, there has been a high level of uptake of the WeatherCAN app, which as of March 2020 was the second-ranked weather app in Canada. ECCC has also made an increasing volume of weather and environmental data available through its Geomart and Datamart platforms.

Target audiences arereceiving information they need and are using it to inform their decision making. External key informants―primarily power users―agreed that the meteorological information they receive from the Weather Program is critical to their organizations. They described using this information in various ways, including to prepare hazard and risk reports for local municipalities; inform P/T and municipal emergency management operations; prepare media briefs related to weather events; inform decisions concerning supply chains and critical infrastructure; and develop and implement procedures for assisting communities impacted by weather conditions and events. For the general public, the program has plans to collect information on the use of weather information more regularly in the future, for example, through post-event surveys. With regard to targeted clients (Canadian Coast Guard, Department of National Defence, and NAV Canada), available indicators tracked by the program show a high level of satisfaction with the services received.

Efficiency

The design of the Weather Program is appropriate. Its core activities are consistent with those of other leading national weather services. The MSC’s International Organization for Standardization certification and Quality Management System (QMS) were also highlighted as important factors contributing to efficiency and continuous improvement. Perceived strengths of the Program include:

Although the Program is perceived as having responded effectively to emerging issues and challenges, internal and external key informants emphasized the need for it to keep pace with developments and to continue to adapt its activities in order to remain relevant. Areas for ongoing effort include enhancements to observation, modelling and data management capacity; adaptation of delivery platforms to suit changing consumer preferences; and engagement and collaboration with external stakeholders. In addition, stakeholders identified a need for the program to enhance services for vulnerable and diverse populations through, for example:

The fact that key enabling functions are housed outside of the MSC is widely perceived by internal key informants as posing significant risks and challenges to the program’s adaptability and efficiency. The program’s reliance on special funding in a context of declining permanent funds for operations and maintenance was widely perceived as jeopardizing its ability to sustain its activities, including carrying out lifecycle management of observation networks and keeping up with information technology, data management and infrastructure needs.

Despite the implementation of various changes to strengthen organizational governance during the period covered by this evaluation, program representatives had mixed views on governance. Half believed it was clear and effective, while the other half viewed it as complex and called for improved coordination. Similar conclusions were reached by a 2018 external review of the MSC governance, which found governance for the branch as a whole to be “solid and robust”, but also observed that the committee structure is complex and not well coordinated.

The Weather Program currently collects and reports limited performance information. For example, this includes limited information related to the outcome: meeting the information needs of its target audiences so that, in turn, target audiences use the weather and environmental information provided by the program to inform their behaviour and decision making. The program has plans to collect this information more regularly in the future through post-event surveys of Canadians.

Recommendations

As the senior departmental official responsible for the Weather Program, 3 recommendations are directed to the Assistant Deputy Minister of the Meteorological Service of Canada.

Recommendation 1: Ensure that the program is positioned to keep up with evolving technology and the changing context.

Evolving technology, changing consumer preferences, the growth of the private sector and a shift away from forecasting weather to forecasting impacts in the context of climate change are requiring the Weather Program to adapt in order to remain relevant to its target audiences. Canadians expect timely and reliable information to make decisions about their health and safety. The Weather Program must keep pace with emerging channels for information delivery and the growing need for data and services, so that it can remain authoritative and relevant to its target audiences and to support timely decision making. For example, to expand its observation capacity, ECCC began exploring new observation technologies, including space-based earth observation.

Recommendation 2: Enhance efforts to identify the needs of, and improve services and supports for, vulnerable and diverse populations.

The evaluation identified the need for the program to enhance services for vulnerable and diverse populations and communities. This includes, but is not limited to remote, Northern and some Indigenous communities, who face different risks from a changing climate. Although the program has taken some steps in this direction, program representatives acknowledged the need to do more. Potential areas for improvement, identified by program representatives and external stakeholders, include:

In addition to better serving vulnerable and diverse populations, such initiatives wouldimprove the program’s alignment with the federal government’s GBA+ policy and objectives.

Recommendation 3: Enhance performance measurement to support reporting on the achievement of the departmental result “Canadians use authoritative weather and related information to make decisions about their health and safety”.

At the time of the evaluation, the Weather Program did not have recent information to report on its final outcome and the departmental result “Canadians use authoritative weather and related information to make decisions about their health and safety”. Considering that this is a key component of ECCC’s public reporting on its core areas of responsibility, the Weather Program should ensure that it collects data to report on progress made in achieving this result.

1. Context

1.1 Overview and context

The objective of the Weather Observations, Forecasts and Warnings Program (Weather Program) was to provide authoritative weather and related environmental information to support Canadians in making decisions about their health and safety and to protect their property.

The program is delivered within Environment and Climate Change Canada (ECCC) by the Meteorological Service of Canada (MSC) in collaboration with the Science and Technology Branch (STB), the Corporate Services and Finance Branch (CSFB), and Shared Services Canada (SSC). In 2021, the MSC has celebrated its 150 years of providing weather-related services to Canadians while ECCC celebrated its 50th year of operation.

In 2013, ECCC developed a vision document entitled: A 10-Year Plan for a Modern and Sustainable Meteorological Service of Canada. Since then, program partners have been engaged in carrying out renewal activities outlined in the plan. Major elements of the renewal initiative include:

1.2 Activities

The Weather Program delivers a comprehensive array of weather and environmental information and services to Canadians. Core program activities are listed below.

Observation

Monitoring and observation of environmental and meteorological conditions over Canada’s vast and diverse landmass and shoreline is enabled through the MSC’s extensive monitoring and observation networks. Data are also received from international sources and through partnerships with provincial and territorial (P/T) agencies. The MSC’s Monitoring and Data Services Directorate is responsible for data collection, management and quality assurance, as well as network design and operation.

Modelling and prediction

Through this activity, numerical weather and environmental prediction models are created and tested, and observation data is placed into the models to provide the foundation for Canada’s weather forecasts and warnings. Reliant upon a sophisticated supercomputing infrastructure and a partnership with SSC, and science and technology research from STB numerical weather prediction and modelling underlie day-to-day forecasting and specialized services in support of emergencies. The Canadian Centre for Meteorological and Environmental Prediction is the operational hub for these activities.

Weather forecasts, warnings and data

The MSC’s Prediction Services Directorate is responsible for developing weather and related environmental services and information dissemination to meet the needs of a variety of audiences. It provides weather warnings, forecasts and information to the general public, 24 hours per day, 365 days per year. This includes providing warnings and alerts to inform Canadians of impending high impact meteorological situations or events, as well as promoting warning preparedness and adaptation to the risks posed by changes in the natural environment. These services are provided by 7 regionally based prediction centres.

In addition, forecasts are provided to targeted clients on a cost-recovery basis, including NAV Canada, the Department of National Defence (DND), and the Canadian Coast Guard (CCG), who depend on weather and environmental information for their security, surveillance, and emergency operations. Finally, specialized support is provided to power users – such as private meteorologists; emergency managers for P/T and municipal regions; private sector scientists; and other federal departments – who use MSC data to develop their own forecasts.

Science and technology research

Supporting research assists the Weather Program to continually improve its observations, modelling and predictions, and services. This research includes assessments of new observing technologies, numerical weather and environmental prediction improvements, advancements in data assimilation techniques, and research on cloud physics and severe weather, as well as research on water and air quality, climate, and ice services. Research activities are carried out by STB, including in collaboration with academia.

International engagement

The Weather Program collaborates with international meteorological organizations and networks, facilitated by its membership in the World Meteorological Organization (WMO), and other national meteorological services to obtain and provide weather data, set international policy standards for observation networks and data sharing, and undertake and share meteorological research. The free and open exchange of data between WMO members is critical to enabling robust forecasts beyond 2 days in outlook. The Policy, Planning and Partnerships Directorate within the MSC plays a central role in international engagement strategy and activities.

Information management/information technology (IM/IT)

IM/IT support involves ensuring appropriate computing infrastructure is in place to consume observational data and information obtained from partners and manipulate and analyze those data in forecast models. It also involves data management strategies that ensure they are in the appropriate format and quality for internal processing and provision to external stakeholders through open data sharing platforms. Application development expertise and services also support and continuously improve program functions and services. The responsibilities for the provision of these services are divided between SSC and CSFB.

1.3 Resources

ECCC resources for the Weather Program consist of permanentFootnote 1  and temporary fundsFootnote 2  to support renewal initiatives.

As shown in Table 1, actual spending for the Weather Program from all ECCC branches totalled $906 million between FY 2015 to 2016 and FY 2019 to 2020.

Table 1: Weather Program actual spending, by ECCC branch and fiscal year (in dollars)
Branch FY 2015 to 2016 FY 2016 to 2017 FY 2017 to 2018 FY 2018 to 2019 FY 2019 to 2020 Total
MSC 111,472,929 113,161,618 119,445,757 173,712,709 193,327,832 711,120,845
CSFB 17,722,585 17,262,203 21,470,839 26,729,736 31,065,227 114,250,590
STB 14,930,573 17,054,669 15,129,112 17,054,976 14,586,498 78,755,828
Other branches 74,755 416,614 511,683 542,617 585,184 2,130,853
Total 144,200,842 147,895,104 156,557,391 218,040,038 239,564,741 906,258,116

Source: ECCC financial information

Note: Prior to FY 2018 to 2019 the Vote Netted Revenue (VNR) spending supporting the services to Nav Canada, DND, and CCG were coded under the program 2.2. in the former program alignment architecture, while spending supporting the weather program were captured under program 2.1. Starting in FY 2018 to 2019, these VNR spending are all captured under the new weather program, created under the Departmental Results Framerowk. Hence, there is a large jump in expenditures, from $156 million to $218 million.

1.4 About the evaluation

The evaluation covered the period between FY 2015 to 2016 and FY 2018 to 2019Footnote 3 and included all program activities, with the exception of Meteorological Areas, Air Quality and Health Related Services and Water and Climate Services.

The objectives of the evaluation were to examine issues of relevance and performance (effectiveness and efficiency), as per the 2016 Treasury Board Policy on Results, and to identify recommendations for possible improvements.

Multiple lines of evidence were used in the evaluation. These included:

Except where otherwise noted, reporting on interview findings focuses on common themes, perspectives and areas of agreement across and/or within key informant categories. Appendix A provides a more detailed description of the evaluation approach.

2. Findings: relevance

2.1 Needs addressed by the program

Findings: There is a clear ongoing need for the Weather Program. National meteorological services such as the Weather Program play a critical role in supporting the vital functions of governments as well as decision making by individuals, households, and businesses. This in turn produces a range of societal benefits. Evolving technology, changing consumer preferences and the growth of the private sector are requiring the Weather Program to adapt in order to remain relevant to its target audiences. To respond to climate change, the program is also shifting away from forecasting weather and informing impacts of weather-related events.

The Weather Program is Canada’s national meteorological service and a fundamental part of Canada’s national infrastructure. Evidence from documents, literature and interviews confirm a continued need for the Weather Program. In particular, all key informants and program documentation reviewed noted that the Weather Program delivers “mission critical” services. According to the WMO, national meteorological services like the Weather Program play an important role in supporting the vital functions of governments, including protection of life and property; safeguarding the environment, economic planning and development; and sustainable exploitation and management of natural resources, among other things. In addition, national meteorological services support the decision making of individuals, households, and businesses, producing a range of societal benefits, such as reduced impacts of natural hazards, enhanced safety, increased business profitability, and improved productivity.

The ongoing need for a strong national weather service such as the Weather Program is clear in the context of climate change. The effects of climate change are increasing the risk of severe weather events, drought, wildfires, inland and coastal flooding, and storm surges. In turn, these are projected to produce damage to infrastructure, food insecurity, loss of property and livelihoods, and population displacement. Another projected consequence of these events is an increased mortality and morbidity due to factors such as heat, flooding and storms. According to data reported by ECCC, extreme weather caused an estimated $1.9 billion in property damage in Canada in 2018.

While everyone is at risk from the impacts of climate change, certain populations and groups may be more vulnerable. In the Pan-Canadian Framework on Clean Growth and Climate Change, the Government of Canada noted that Indigenous peoples, as well as northern, remote, and coastal regions, are particularly vulnerable and disproportionately affected by the impacts of climate change.

In the context of climate change, the role of the Weather Program, like that of all national weather services, is shifting away from forecasting weather to forecasting weather and impacts. The Weather Program is increasingly expected to provide timely information to support emergency response and risk mitigation decisions, as well as longer term and localized forecasting to support land use and urban planning decisions. This new role is challenging the program to develop new modelling and observational tools, forecast guidance products, and methods of information and warning dissemination.

At the same time, the Weather Program, like other national meteorological services, is also having to respond to other trends and pressures, including:

Overall, it is clear that the program will need to continue to evolve in response to these developments, in order to remain relevant to its target audiences.

2.2 Alignment with federal roles, responsibilities and priorities

Findings: The Weather Program aligns with federal roles and responsibilities, as well as federal priorities related to climate change, economic growth and prosperity, science and innovation, and open and transparent government. Increased focus on services for vulnerable populations could improve alignment with the federal government’s GBA+ objectives.

The Weather Program is aligned with federal roles and responsibilities. Legal and statutory responsibility for the program resides in the Department of the Environment Act (2018), which gives the Minister of ECCC responsibility for overseeing activities related to preservation and enhancement of the environment, water, and meteorology. The Weather Program assists in the discharge of these duties through the provision of meteorological forecasts and observation data on water and climate indicators. In addition, the program provides support to other departments and agencies acting under the Canadian Weather Modification Act (2005), the Emergency Management Act (2007), the Canada Water Act (2014), and the Impact Assessment Act (2019).

At the international level, ECCC is responsible for supporting Canada’s commitment to the Convention of the WMO, including Canada’s participation in international weather, water, climate and air quality initiatives. A key responsibility met through the Weather Program is the global sharing of meteorological data. In particular, ECCC shares data with other participating countries and is also able to access data from those countries and integrate these data into its forecast models.

In addition, ECCC partners with other federal departments, the P/Ts, and the private sector in relation to weather services. For example, ECCC partners with other federal departments (led by Public Safety Canada), the P/Ts, and the private sector for emergency preparedness, mitigation and response management related to weather-related threats. As another example, the MSC has recently begun establishing partnerships with the P/Ts to access data collected through the P/Ts’ observation infrastructures. These partnerships are described in detail elsewhere in this report.

The Weather Program aligns well with federal priorities related to climate change, economic growth and prosperity, science and innovation, and open and transparent government. That said, evaluation evidence indicates a need to improve alignment with the federal government’s gender-based analysis plus (GBA+) policy and objectives. Program representatives noted during interviews that more work could be done to increase the use of GBA+ within the program. They acknowledged the importance of GBA+ and the need to improve program alignment with government objectives by, for example, considering the different impacts of weather events on vulnerable populations; and increasing efforts to tailor information and alerts for different sub-populations (see further details in Section 4).

3. Findings: effectiveness

3.1 Activities and accomplishments

Findings: The Weather Program had a number of accomplishments since FY 2015 to 2016. It has been leading the ongoing renewal and replacement of ECCC’s existing observation infrastructure. It continued a shift towards digital dissemination of weather forecasts and warnings, including the launch of the WeatherCAN app. It implemented the Alert Ready system for weather warnings, in partnership with other federal departments, P/T governments, and the private sector. Furthermore, the program put an increased emphasis on providing specialized information and advice, including impact forecasting, to support decision making by emergency management organizations and other power users. Finally, over the period, the MSC’s new high-performance computer became fully operational, enhancing data integration, processing and modelling capacity and enabling more accurate forecasts.

Observation network

ECCC continued work to renew and/or replace the infrastructure that supports its direct observation networks. As of FY 2019 to 2020, ECCC reported it was on track to replace all 32 of its aging radar stations with new state-of-the-art radars by 2023. Upgrades were made to the Surface Observing Network, the Upper Air Network, the Marine Observing Network, the Canadian Lightning Detection Network, and the Satellite Reception Network.  

To expand its observation capacity, ECCC made progress in establishing joint observation networks with the P/Ts and in formalizing agreements for joint data access. Memoranda of Understanding have been signed with 6 jurisdictions to date. Planning and coordination activities are underway with several federal departments, including Crown-Indigenous Relations and Northern Affairs Canada, Department of Fisheries and Oceans, and Parks Canada. ECCC also began exploring new observation technologies, including space-based earth observation. It continues to collaborate, through the WMO and other partnerships, to expand observation capacity and access data from international sources.

Furthermore, ECCC currently has a number of contribution agreements in place that support Indigenous communities building observation capacity for weather monitoring. For example, the Centre for Indigenous Environmental Resources, Inc was funded to develop a strategy for involving Indigenous communities in weather observation data collection.

Progress on observation network renewal, as of fiscal year 2019 to 2020

Surface Observing Network: The core of the Weather Program’s observation infrastructure, this network consists of approximately 1,000 weather stations (550 automated and 450 human observation) across the country which collect data on temperature, humidity, wind and barometric pressure.

Canadian Doppler Radar Network: This consists of 32 stations across the country that provide images based on the reflection of water droplets and snow to identify areas of precipitation. The images allow forecasters to observe speed and direction of precipitation systems allowing the MSC to alert Canadians to severe thunderstorms, damaging winds, freezing precipitation and tornadoes.

Upper Air Network: This network measures atmospheric conditions at various altitude levels through the release of radiosonde devices carried on weather balloons and through equipment placed on commercial aircraft.

Marine Observing Network. This network collects meteorological and oceanographic data in ocean regions through weather buoys (drifting and moored) as well as equipment placed on a network of ships and reported by volunteers.

Canadian Lightning Detection Network: This network collects lightning strike information to monitor potential storm and forest fire threats.

Satellite Reception Network: This network consists of 15 stations that process satellite imagery from foreign weather satellites. The images received show cloud patterns, allowing meteorologists to monitor clouds, fog and storm movement patterns, as well as sea ice, forest fires and volcanic ash dispersion.

To expand its observation capacity, ECCC made progress in establishing joint observation networks with the P/Ts and in formalizing agreements for joint data access. Memoranda of Understanding have been signed with 6 jurisdictions to date. Planning and coordination activities are underway with several federal departments, including Crown-Indigenous Relations and Northern Affairs Canada, Department of Fisheries and Oceans, and Parks Canada. ECCC also began exploring new observation technologies, including space-based earth observation. It continues to collaborate, through the WMO and other partnerships, to expand observation capacity and access data from international sources.

Furthermore, ECCC currently has a number of contribution agreements in place that support Indigenous communities building observation capacity for weather monitoring. For example, the Centre for Indigenous Environmental Resources, Inc was funded to develop a strategy for involving Indigenous communities in weather observation data collection.

Progress on observation network renewal, as of fiscal year 2019 to 2020

Surface Observing Network: The core of the Weather Program’s observation infrastructure, this network consists of approximately 1,000 weather stations (550 automated and 450 human observation) across the country which collect data on temperature, humidity, wind and barometric pressure.

Canadian Doppler Radar Network: This consists of 32 stations across the country that provide images based on the reflection of water droplets and snow to identify areas of precipitation. The images allow forecasters to observe speed and direction of precipitation systems allowing the MSC to alert Canadians to severe thunderstorms, damaging winds, freezing precipitation and tornadoes.

Upper Air Network: This network measures atmospheric conditions at various altitude levels through the release of radiosonde devices carried on weather balloons and through equipment placed on commercial aircraft.

Marine Observing Network. This network collects meteorological and oceanographic data in ocean regions through weather buoys (drifting and moored) as well as equipment placed on a network of ships and reported by volunteers.

Canadian Lightning Detection Network: This network collects lightning strike information to monitor potential storm and forest fire threats.

Satellite Reception Network: This network consists of 15 stations that process satellite imagery from foreign weather satellites. The images received show cloud patterns, allowing meteorologists to monitor clouds, fog and storm movement patterns, as well as sea ice, forest fires and volcanic ash dispersion.

Modelling and prediction

Case study: Canada’s National Public Alerting System (NPAS)

The NPAS is a collaborative initiative between federal and P/T governments, Pelmorex Corp, and other private sector actors. It delivers critical alerts of various types to Canadians following an established process that includes:

  • Authorized government issuers including ECCC and P/T organizations and emergency management officials create alert messages and submit them to the System operated by Pelmorex
  • Pelmorex systems validate the information  and push the alert message to distributors, including television, radio, cable/satellite, and wireless service providers
  • Alert distributors deliver the alert messages to Canadians over their communications platforms
  • The public receives the alert messages, containing potentially life-saving information

Weather-related alerts and warnings are the largest portion of public alerts distributed through the NPAS, making ECCC the biggest system user.

ECCC continued to develop new approaches to mathematical modelling to produce results more efficiently, leading to more effective use of computing power. Several new or improved models were developed, including a High Resolution Prediction System for weather warnings, which allows for more localized observation and warnings to be created due to improved resolution. As another example, ECCC delivered the first regional scale ensemble environmental system model for the Great Lakes region, which combines ice, wind and wave observation data to improve wave forecasts, contributing to improved safety and efficiency of marine transportation. As a final example, ECCC collaborated with Mercator Ocean to develop the Global Ice-Ocean Prediction Systems, which combines observations from satellite, sea and air data to enable real-time, three-dimensional images of the state of oceans in support of coastal navigation and isolated northern communities whose hunting and transportation options are impacted by ice flows.

Weather forecasts, warnings and data

ECCC has made several improvements to its forecasts and warnings for the general public. Recognizing that changing consumer preferences and technological changes demanded a shift to mobile digital presentation and dissemination of weather information, ECCC launched the WeatherCAN application in February 2019. The application provides current, hourly, and 7-day weather and air quality forecasts in English and French for over 10,000 Canadian locations.

ECCC modernized its weather warning and forecast systems through the deployment of new software; an improved weather bulletin that includes descriptions of weather impact and suggested protective actions and colour-coded warnings to communicate risk levels; and delivery of warnings through digital means. In July 2015, Alert Me email weather alerts and twitter weather alerts were launched for over 830 communities across Canada. ECCC also promotes general public safety by providing advice on the Get Prepared website on how to prepare and respond to various weather-related emergencies.

ECCC partnered with Pelmorex Corporation and media broadcasters in improving Canada’s National Public Alerting System (NPAS) (see text box above). The system, known publicly as “Alert Ready”, was implemented in 2015 for automated television and radio interruptions for alerts, and expanded to mobile channels in April 2018. These changes improved the reach and geographic specificity of alerts.

In addition to forecasts and warnings for the general public, ECCC provided specialized forecasts and support services to power users such as emergency management organizations. It adopted a vigilance approach, focused on providing advanced warnings of weather events and associated risks, as well as interpretation and advice to support decision making. To help support the delivery of these services, ECCC created decision support desks staffed with Weather Preparedness Meteorologists, who serve as its primary support interface with emergency management organizations and the media. Some examples of the specialized services delivered by ECCC include:

ECCC also continued to provide forecasts to targeted clients on a cost-recovery basis, including aviation forecasts to NAV Canada, and marine and ice forecasts to the DND and the CCG. Long-term service agreements were renewed with DND and the CCG. Contract renewal negotiations with NAV Canada were ongoing at the end of the period covered by this evaluation.

International engagement and collaboration

ECCC participates in a range of international partnerships and collaborations. As an active member of the WMO since 1950 and current member of its Executive Council, Canada plays a lead role in WMO activities relating to co-creation and access to international meteorological policy, research and data. As a result of these activities, Canada has been able to influence future plans and objectives. Currently, Canada is leading the development of a new strategy for the WMO, based on an earth systems approach.

During the period covered by this evaluation, Canada provided funding to the WMO for various projects, including $10 million to build the Climate Risk Early Warning Systems, multi-hazard early warning systems in Small Island Developing States and Least Developed Countries, including countries in southeast Asia and the Caribbean. MSC funding to the WMO also supported projects related to polar observations, predictions and climate services; and the development of global aircraft meteorological data relay.

The MSC continues to collaborate with a number of other international meteorological services, including the United States National Oceanic and Atmospheric Administration, the United Kingdom Met Office, the Australian Bureau of Meteorology, the Ninjo Consortium (involving German, Swiss, and Danish weather services) and the China Meteorological Association. It also collaborated with the European Centre for Medium Range Weather Forecasting, EUMETSAT, the Group on Earth Observations, Mercator Ocean International, and the North American Climate Services Partnership, among many others, on a range of activities, including data collection, sharing, modelling, and research.

Science and technology research

ECCC undertook scientific and technology research and collaborated with other organizations through the provision of funding to universities, P/T government departments, and non-governmental organizations, in order to expand its knowledge and improve its observation networks, modelling and service capacity. A few examples of funded projects, based on the file review, are listed below.

Information technology and information management

In FY 2018 to 2019, ECCC’s new high-performance computer became fully operational. The increased computing capacity supports the integration and processing of data collected from new infrastructure and partner observations and allows the data to be processed in improved models to produce more accurate forecasts. This brings Canada’s capacity in line with other leading meteorological services.

ECCC also undertook a variety of activities to improve data acquisition, processing and transmission, and data quality and efficiency. For example, ECCC improved its Numerical Environmental and Weather Prediction System for forecasts beyond 7 days by bringing over 13.5 million observations into the system each day. ECCC also calibrated its national alert system to reflect data from the multiple observation networks described above, which is expected to result in more accurate and timely public weather alerts.

3.2 Achievement of outcomes

Findings: Target audiences are accessing ECCC weather and environmental services. External key informants – primarily power users – agreed that the meteorological information they receive from the Weather Program is critical to their organizations and informs their decision making. With regard to targeted clients, available indicators tracked by the program show a high level of satisfaction with the services received.

The Weather Program expects that target audiences will access its weather and environmental services and that they will receive the meteorological information they need. Ultimately, the program expects that Canadians will use weather and related information to make decisions about their health and safety.

Target audiences access to ECCC weather and environmental services

Evidence indicates that target audiences are accessing ECCC weather and environmental services. Available data, reported in ECCC’s DRRs, show that:

In addition, some of the projects funded by ECCC were intended to increase public awareness of, and access to, weather information. For example, ECCC funding supported:

Case study: Delivery of weather services for the 2015 Pan American and Parapan American Games

ECCC’s role in the 2015 Pan American and Parapan American Games was to provide enhanced weather monitoring and forecasting and local-level preparedness activities, and to support environmental assessments of projects relating to the Games.

To deliver these services, a new temporary atmospheric monitoring network was designed, known as the Mesonet. The Mesonet added approximately 60 new automated land and marine stations to the existing weather monitoring system to cover all venue locations and transportation routes between them. It provided enhanced atmospheric monitoring data and gave the specially selected and trained bilingual MSC weather briefing teams the ability to tailor observational data and provide sport-specific forecasts or alerts.

Data captured from the Mesonet network was used by ECCC forecasters to produce forecasts and alerts for organizers, volunteers, athletes and security teams in support of safety and security, including:

  • 463 venue-specific public and marine forecasts
  • 37 venue-specific weather warnings, watches and advisories
  • 69 Air Quality Health Index forecasts

ECCC scientists used Mesonet data to work on improving severe weather detection and early warning notification capabilities to enhance general public forecasts and weather alerts.

Almost all external key informants reported accessing a wide variety of Weather Program products and services, including vigilance reports; alerts (windchill, heat, cold weather, etc.) and automated email updates; Storm Prediction Centre services including convective outlooks and the 1-800 consultation line; and quarterly seasonal regional forecasts. Many external key informants also reported having direct interactions with particular ECCC staff members, who were able to accommodate specific requests for information tailored to their needs. Having ECCC personnel embedded within P/T emergency management organizations was highlighted as particularly useful for improving access.

Both internal and external key informants believed the program’s investments in new modes of information dissemination have improved the accessibility of weather information. However, some suggested that public access and open sourcing of data could be further enhanced, for example, by developing ECCC’s website as a “one-stop shop” for weather information and data including observations, long-range forecasts, archived data, metadata, and other information.

Clients receive the meteorological data they need

External key informants―primarily power users―agreed that the Weather Program provides critical information to their organizations, and that without it, there would be significant gaps in needed weather and climate information.

ECCC’s provision of weather services for the 2015 Pan American and Parapan American Games in Toronto illustrates how the Weather Program works to deliver specialized meteorological information to clients (see text box above).

As shown in Table 2, the Weather Program has established service standards and uses a variety of indicators to measure forecast accuracy and the satisfaction of targeted clients.

Table 2: Weather Program indicators of forecast accuracy and client satisfaction, by fiscal year
Indicator FY 2015 to 2016 FY 2016 to 2017 FY 2017 to 2018 FY 2018 to 2019 FY 2019 to 2020
Accuracy of 3-day forecasts within 3 degrees 85% 86% 86% 86% 87%
Accuracy of 5-day forecasts within 3 degrees 75% 77% 75% 77% 77%
Index of timeliness and accuracy of severe weather forecasts 8.3 8.4 8.4 8.6 8.8
NAV Canada client satisfaction 7.4 8.1 8.1 8.3 8.0
DND client satisfaction 8.5 8.8 8.5 8.9 8.8
CCG client satisfaction 9.1 8.8 7.6 8.8 9.2
Average satisfaction of targeted clients 8.3 8.6 8.1 8.7 8.7

Beyond this, the file review indicated that many of the projects funded by the MSC during the period covered by this evaluation were intended to expand the amount of data available and improve the quality of data, models and accessibility to data and services. This should indirectly improve the extent to which clients receive the meteorological data that they need.

There is no recent information on the extent to which the Canadian public is satisfied with the weather information they receive from ECCC, since public opinion research exploring this topic was not conducted during the period covered by this evaluation.

Canadians use authoritative weather and related information to make decisions about their health and safety

The ultimate departmental result for the Weather Program is that Canadians use authoritative weather and related information to make decisions about their health and safety. According to ECCC’s DRR for FY 2017 to 2018, in that year, 53% of the population saw or heard a weather warning and took action in response, compared to 45% in FY 2015 to 2016. In FY 2018 to 2019, however, the program began using the severe weather warning index as the indicator for this outcome.

Some anecdotal information is available illustrating how the public uses weather information to inform its decision making. One example, widely reported in the media, occurred during the September 2018 tornados that struck the Ottawa/Gatineau region. Emergency alerts disseminated through the NPAS were credited by media with saving lives, since warnings were received by residents in time to allow them to take shelter and avoid injury even if their homes were in the path of the tornado. Notably, mobile alerting had only been implemented a few months earlier in April 2018. Key informants who participated in the NPAS case study believed that the NPAS is succeeding in providing timely and immediate access to critical weather information to the maximum number of people.

External key informants (primarily power users) provided numerous examples of how they use ECCC weather-related information to support decision making to:

As another example, ECCC reported that the targeted recipients of the forecasts and alerts provided during the 2015 Pan American and Parapan American Games were “extremely appreciative” to have received this information, which they considered valuable in planning and carrying out their work related to the Games.

MSC representatives acknowledged the need to better track and report on the extent to which the general public accesses and uses the information provided by the Weather Program to make decisions about their health and safety. Some steps are being taken to address this gap. For example:

Other outcomes

ECCC has identified 2 other expected outcomes for the Weather Program, these are related to data access and meteorological research.

Continued access to meteorological data from other countries

Many key informants, both internal and external, observed that the MSC’s involvement and leadership in various international organizations and bilateral collaboration with other countries has led to 2-way data-sharing with other countries. The MSC’s involvement in the WMO, in particular, was credited with bringing access to “a huge amount of real-time data” shared globally on a daily basis. According to internal documentation, on average, more than 99% of observational data used in ECCC numerical weather prediction models were obtained from international sources each year between FY 2016 to 2017 and FY 2018 to 2019.

Meteorological research addresses issues affecting Canadian interests and priorities

As already reported, the MSC undertakes meteorological research internally and by funding external organizations. According to internal documents, 58% of MSC grants and contributions funding was dedicated to research projects in FY 2019 to 2020.Footnote 4 This exceeded the target of 30%. A review of funded project files indicated that many of the MSC’s contribution agreements are for research projects relevant to the program and its core responsibility areas. The extent to which they address issues affecting Canadian interests and priorities could not be assessed, given that the indicators focus only on whether a funded project is a research project and not on whether it meets Canadian interests and priorities.

4. Findings: efficiency

4.1 Program design

Findings: The Weather Program’s core activities are consistent with those of other leading national weather services. Moreover, the program is perceived as having responded effectively to emerging issues and challenges. Key informants emphasized the need for the program to keep pace with developments and continue to adapt its activities in order to remain relevant. Areas for ongoing effort include enhancements to observation, modelling and data management capacity; adaptation of delivery platforms to suit changing consumer preferences; services for vulnerable populations; and collaboration with external stakeholders.

There is widespread agreement among internal and external key informants that the design of the Weather Program is appropriate and that its core activities are consistent with those of other leading national weather services. Many internal and international key informants noted that Canada’s weather service is 1 of the best in the world. Perceived strengths included:

Internal and external key informants believe the Weather Program is responding quite effectively to emerging challenges and pressures facing all national weather services. As described in Section 2, these pressures include evolving technology, changing consumer preferences, the growth of the private weather sector, and the shift to impact forecasting in the context of climate change. As evidence, key informants cited the program’s efforts to expand observation capacity, enhance modelling, adopt digital platforms for information dissemination, and deliver value-added products and services focused on impact forecasting and support for decision making.

Both internal and external key informants also identified a need for the Weather Program―like national weather services in other countries―to keep pace with developments and continue to adapt its activities in order to remain relevant. Areas for further effort and improvement highlighted included:

Case study: Ongoing evolution of the National Public Alerting System

The launch of the Alert Ready public awareness campaign in 2015 and the introduction of wireless alerting in 2018 are credited with enhancing public awareness of the NPAS, and improving the reach and geographic specificity of alerts. However, limitations remain. Unreliable broadband coverage in rural and remote areas, particularly in Northern Canada, means that alerts in these areas — which are received primarily via radio or television — cover larger areas and may be less relevant to local populations. Vulnerable populations in any geographic location may similarly lack access to wireless communications technology.

Case study participants identified several potential future enhancements to the NPAS, including:

  • tailoring alerts for vulnerable sub-populations, to increase the relevance of alerts to those most affected
  • targeting alerts to specific geographic areas more precisely, to reduce instances where people receive alerts that do not apply to them
  • proactively anticipating technological changes and engaging with manufacturers of technology to ensure that the NPAS can take advantage of new dissemination mechanisms

As a key system user, case study participants noted that ECCC is well placed to promote greater consistency in how the NPAS is used, and to guide further enhancements to address identified limitations.

4.2 Efficiency

Findings: The Weather Program has made numerous enhancements to equipment, technology, services and products, and on the whole, is seen to be operating efficiently to deliver services to Canadians. However, having key enabling functions housed outside of the MSC is widely perceived by internal key informants as posing significant risks and challenges to the program’s efficiency and adaptability.

Overall, internal key informants believed that the Weather Program is operating efficiently to deliver weather and environmental services to Canadians, pointing to the improvements to equipment, technology, services, and products that have been made during the period covered by the evaluation. The MSC’s International Organization for Standardization certification and Quality Management System (QMS) were also highlighted as important factors contributing to efficiency and continuous improvement.

Nevertheless, many internal key informants believed that the fact that key enabling functions (for example, staff providing IT services) are housed outside of the MSC (CSFB and SSC) challenges the efficient delivery of the Weather Program, as well as its adaptability and flexibility to respond to emerging issues and challenges. They noted the following:

Overall, internal key informants noted that the program’s interactions with enabling functions require a significant amount of time and administrative resources and have high transaction costs. In turn, resources devoted to these interactions are not available for service delivery. Key informants noted that in contrast to Canada, other national meteorological services, including those in the United States and the United Kingdom, have enabling functions embedded within them. While acknowledging that the centralized model has largely been implemented government-wide, internal key informants believed that this model creates challenges from the perspective of program efficiency.

4.3 Governance

Findings: Despite the implementation of various changes to strengthen organizational governance, program representatives had mixed views on governance. Half believed it was clear and effective, while the other half viewed it as complex and called for improved coordination. Similar conclusions were reached by a 2018 external review of the MSC governance.

An external review of the MSC’s governance structure, completed in July 2018, found governance for the branch as a whole to be “solid and robust”, but also observed that the governance committee structure is complex and not well coordinated, and lacks consistent documentation. Recommendations focused on clarifying the roles and responsibilities of the committees involved. Recommendations were not implemented and data collected in this evaluation do not allow to conclude whether the challenges noted in the review continue to persist. As previously indicated, key informants interviewed had mixed views about the clarity and effectiveness of the structure.

Program representatives reported that during the period covered by this evaluation, several changes were implemented to strengthen organizational governance:

Key governance committees are currently in place specifically for the Weather Program. Numerous other committees exist and play an oversight or advisory role with respect to specific components or activities of the Weather Program. For example, a Monitoring Directors Committee has been implemented and meets regularly. Items discussed at those meetings included renewal, among other things. As well, other departmental level committees may make decisions affecting Weather Program activities but these were not considered in the evaluation as their mandate is broader.

Key governance committees for the Weather Program

Notwithstanding improvements made to strengthen organizational governance, internal key informants were divided in their views on the clarity and effectiveness of the current governance structure. Some internal key informants reported that there were communication issues. For example, they perceived that the governance had become less clear since the 2015 re‑organization.

Those who believed it was working well indicated that:

Case study: The role of strong governance and collaboration in effective service delivery

Collaboration between STB and the MSC was essential to ECCC’s successful delivery of meteorological services during 2015 Pan American and Parapan American Games. Guided by a senior management directive that any new technology used during the Games required operational applicability, collaboration allowed STB and the MSC to take advantage of the unique opportunities provided by the Games to leverage state-of-the-art technologies and enhance services in the future. Outcomes included:

  • investment in local weather monitoring infrastructure to ensure the region had accurate and high quality observations and forecasting data
  • the transfer of high-resolution monitoring and numerical modelling data by STB to MSC forecasters in real time, which were planned to be utilized for future projects post-Games
  • broader system-wide service improvements, such as the later development of the WeatherCAN app, evolved out of an app developed for the Games
  • nationwide rollout of the one-hour Air Quality Health Index forecast program in 2016 and improvements to air quality monitoring services

ECCC key informants noted that the type of research, technology development and demonstration, and model validation that occurred during the Games would have been too costly for STB to undertake on its own. Leveraging the opportunity to undertake this work in concert with service delivery for the Games allowed the necessary resources to be made available. Furthermore, collaboration between research and operations branches facilitated the generation of new ideas that will help ECCC imagine what the future of weather programing could look like and how to better serve Canadians. Program representatives reported that 3 other national meteorological services adopted, or will adopt, MSC’s approach during Winter and Summer Olympic Games, including in Seoul in 2018; Beijing in 2020; and Paris in 2024.

Many internal key informants also emphasized that clear governance structures and strong relationships across the program partners and components were essential to the program’s success in sustaining critical services during the COVID-19 pandemic, and to the successful delivery of meteorological services during the 2015 Pan American and Parapan American Games. Strong governance was also important in leveraging the large-scale environment of the Pan American and Parapan American Games to undertake research and development activities in a cost-effective manner in concert with service delivery, in order to support future capabilities (see text box).

During interviews for this evaluation, some internal key informants mentioned similar insights and perceived the governance structure as complicated, with responsibilities for program components divided among and shared across multiple entities. These key informants believed that components of the program operated largely in a siloed manner, and identified a need for better communication and coordination among the various entities involved. A need to continue to set clear priorities for the Weather Program, and to effectively communicate these across the program, was identified.

4.4 Resources

Findings: The Weather Program spent nearly all planned funds (99.7%) during the period covered by this evaluation. Internal key informants indicated that the program’s ability to sustain its activities is jeopardized by its reliance on temporary funding in a context of declining permanent funds for operations and maintenance.

As shown in Table 3, during the period covered by this evaluation, the Weather Program’s actual expenditures were 99.7% of planned spending. The MSC and STB slightly exceeded planned spending, while CSFB underspent (due to resources being reallocated to other programs).

Table 3: Weather Program planned and actual spending for fiscal year 2015 to 2016 and fiscal year 2019 to 2020, by branch
Branch Planned spending Actual spending Amount of variance Actual/planned
MSC $707,060,860 $711,120,845 $4,059,985 101%
CSFB $125,148,202 $114,250,590 -$10,897,612 91%
STB $75,167,930 $78,755,828 $3,587,898 105%
Other $1,998,635 $2,130,853 $132,218 107%
Total $909,375,627 $906,258,116 -$3,117,511 99.7%

Note: Expenditures in the “other” category consist primarily of funds held in reserve which were not ultimately expended, accounting for the large discrepancy between planned and actual spending in this category.

Almost all internal and external key informants believed that the Weather Program is under-resourced and indicated that the program’s ability to sustain its activities is jeopardized by its reliance on temporary funding in a context of declining permanent funds for operations and maintenance.

Many program representatives commented specifically on the balance between permanent and temporary funding to support program activities. In particular, they expressed concern about the program’s ability to sustain its activities over the longer term, given its reliance on temporary funding. While internal key informants consider temporary funding to be adequate at present, they also noted that permanent funding has declined over time and is currently lower than what the program requires for operations and maintenance. This has put pressure on the program’s ability to carry out lifecycle management of observation networks, to the extent that some stations currently do not meet international standards; and more generally, to keep up with information technology, data management and infrastructure needs.

Other resource challenges included:

Internal QMS documents identify severe financial risks associated with the pending end of Renewal II funding in 2023. Documents note that although some cost savings have been realized in salaries and operations, these savings were due to inability to fill vacancies and to restrictive maintenance policies, respectively, and were not sustainable in the longer term. The documents add that the failure to fill staff positions and properly maintain equipment risks program failures, potentially impacting clients. The MSC has attempted to mitigate risks by seeking lower-cost operations and maintenance contracts.

4.5 Performance measurement

Findings: At the time of the evaluation, the Weather Program did not have recent performance data on its final outcome and departmental result: “Canadian use authoritative information to make decision about their health and safety”.

The Weather Program’s approach to performance measurement has evolved over the period covered by this evaluation. After the introduction of the Policy on Results, in FY 2018 to 2019, ECCC began using a Performance Information Profile and a Departmental Results Framework to plan objectives and indicators and report on results of its programs including the Weather Program.

Despite this transition, pre-existing outcomes related to access and use of weather information by targeted audiences, and their corresponding indicators, were maintained or slightly adjusted. At the time of the evaluation, however, program representatives acknowledged that limited information was being collected on the extent to which the Weather Program is meeting the information needs of target audiences, and the extent to which audiences are using the information provided to inform their behaviour and to support decision making. As shown in Table 3, the Weather Program has collected data on the satisfaction of key partners but it does not have data on the extent to which the Canadian public is satisfied with the weather information they receive from ECCC.

Program representatives identified the program’s International Organization for Standardization certification and QMS framework as key elements supporting performance measurement, since these guide the setting of performance standards; ensure that performance and risks analyzed are twice per year; and provide information that supports decision making.

5. Conclusions, recommendations and management response

5.1 Conclusions

There is a clear ongoing need for the Weather Program. National meteorological services such as the Weather Program play a critical role in supporting the vital functions of governments as well as decision making by individuals, households, and businesses, producing a range of societal benefits. Evolving technology, changing consumer preferences, the growth of the private sector, and a shift away from forecasting weather to forecasting impacts in the context of climate change are requiring the Weather Program to adapt in order to remain relevant to its target audiences. There is widespread support among internal and external stakeholders to improve services for vulnerable and diverse populations, and this could improve program alignment with the federal government’s gender-based analysis plus (GBA+) objectives.

Key program accomplishments during the period covered by this evaluation included:

While there is clear evidence that target audiences are accessing ECCC weather and environmental services. The program has plans to collect information on the use of weather information more regularly in the future, for example, through post-event surveys.

The Weather Program core activities are consistent with those of other leading national weather services. There is a need for the program to keep pace with developments and continue to adapt its activities in order to remain relevant. Areas for ongoing effort include enhancements to observation, modelling and data management capacity; adaptation of delivery platforms to suit changing consumer preferences; and engagement and collaboration with external stakeholders. In addition, there is a need for the program to enhance services for vulnerable and diverse populations through, for example:

The housing of key enabling functions outside of the MSC is widely perceived by internal key informants as posing significant risks and challenges to the program’s adaptability and efficiency. The program’s reliance on special funding in a context of declining permanent funds for operations and maintenance was widely perceived as jeopardizing its ability to sustain its activities, including carrying out lifecycle management of observation networks and keeping up with information technology, data management and infrastructure needs.

Despite the implementation of various changes to strengthen organizational governance during the period covered by this evaluation, program representatives had mixed views on governance. Half believed it was clear and effective, while the other half viewed it as complex and called for improved coordination. Similar conclusions were reached by a 2018 external review of the MSC governance, which found governance for the branch as a whole to be “solid and robust”, but also observed that the committee structure is complex and not well coordinated.

5.2 Recommendations and management response

As the senior departmental official responsible for the Weather Program, 3 recommendations are directed to the Assistant Deputy Minister of the Meteorological Service of Canada.

Recommendation 1

Ensure that the program is positioned to keep up with evolving technology and the changing context.

Discussion: Evolving technology, changing consumer preferences, the growth of the private sector and a shift away from forecasting weather to forecasting impacts in the context of climate change are requiring the Weather Program to adapt in order to remain relevant to its target audiences. Canadians expect timely and reliable information to make decisions about their health and safety. The Weather Program must keep pace with emerging channels for information delivery and the growing need for data and services, so that it can remain authoritative and relevant to its target audiences and to support timely decision making. For example, to expand its observation capacity, ECCC began exploring new observation technologies, including space-based earth observation.

Statement of agreement or disagreement: The Assistant Deputy Minister of the Meteorological Services of Canada agrees with the recommendation.

Management response: Canada’s Weather Observations, Forecasts and Warnings Program will continue evolving its operations and supporting science to ensure that Canadians are better prepared for unprecedented weather and more resilient to climate change. ECCC’s MSC will prepare a forward-looking business plan (2023 to 2033) that will span the hydro meteorological value chain, from observations to services. This plan will guide the continued transformation of observation networks, including testing and implementing new technologies and approaches, as well as enhance the delivery of impact-based, fit-for-purpose information and services to Canadians, public authorities, and other specialized clients. The importance of leveraging international collaboration in science and collaboration data exchange will continue to be a key driver. Therefore, the MSC will also continue supporting Canadian leadership and participation in the World Meteorological Organization’s governance bodies to access global data, stay abreast of major technology shifts, and leverage other countries’ know-how through collaboration and actively engage in specific areas of science and technology evaluation of direct importance to our future business plan.

Deliverable(s):

  1. 10-year business plan
    1. Timeline: December 2022
    2. Responsible party: DG, Policy, Planning and Partnerships Directorate
  2. Canada will support leadership and participation in the transformation of the WMO governance bodies
    1. Timeline: December 2023
    2. Responsible party: DG, Policy, Planning and Partnerships Directorate
  3. Canada will maintain a participation of experts at the international stage, including in the WMO bodies.
    1. Timeline: Ongoing
    2. Responsible party: DG, Policy, Planning and Partnerships Directorate

Recommendation 2

Enhance efforts to identify the needs of, and improve services and supports for, vulnerable and diverse populations.

Discussion: The evaluation identified the need for the program to enhance services for vulnerable and diverse populations and communities. This includes, but is not limited to remote, Northern and some Indigenous communities, who face different risks from a changing climate. Although the program has taken some steps in this direction, program representatives acknowledged the need to do more. Potential areas for improvement, identified by program representatives and external stakeholders, include:

In addition to better serving vulnerable and diverse populations, such initiatives would improve the program’s alignment with the federal government’s GBA+ policy and objectives.

Statement of agreement or disagreement: The Assistant Deputy Minister of the Meteorological Services of Canada agrees with the recommendation.

Management response: ECCC’s Weather Program provides foundational data and services that help Canadians make decisions about their health and safety, and support public authorities prepare and respond to weather-related emergencies. Historically, the Weather Program has provided services broadly targeting all Canadians and some targeted services to specialized clients. Going forward, the Weather Program will add a focus on vulnerable Canadians and communities, particularly in the context of Canada’s changing climate and associated unprecedented weather. The Air Quality Health Index program, which provides targeted health messages for at-risk individuals who may be more sensitive to air pollution, is an example that the MSC can build on to develop other services streams for vulnerable and diverse Canadians and communities.

Deliverable(s):

  1. As part of its business plan, the MSC will develop services streams that target vulnerable and diverse Canadians and communities.
    1. Timeline: December 2022
    2. Responsible party: DG, Prediction and Services Directorate
  2. The MSC will expand its efforts to gather input from specific vulnerable groups in order to design more effective services.
    1. Timeline: December 2022
    2. Responsible party: DG, Prediction and Services Directorate

Recommendation 3

Enhance performance measurement to support reporting on the achievement of the departmental result “Canadians use authoritative weather and related information to make decisions about their health and safety”.

Discussion: At the time of the evaluation, the Weather Program did not have recent information to report on its final outcome and the departmental result “Canadians use authoritative weather and related information to make decisions about their health and safety”. Considering that this is a key component of ECCC’s public reporting on its core areas of responsibility, the Weather Program should ensure that it collects data to report on progress made in achieving this result.

Statement of agreement or disagreement: The Assistant Deputy Minister of the Meteorological Services of Canada agrees with the recommendation.

Management response: The Weather Program has been managed with a robust performance and verification program, since the 1980s. Since 2007, the organization has regularly maintained ISO certification on all elements of the operational programs and some elements of the enabling functions. As part of this Quality Management System (QMS), the organization conducts regular mid-year and year-end reviews, which include client feedback and benchmarking. The MSC has been and will continue to be committed to continuous improvement on key performance metrics and mechanisms for client feedback. In addition to performance and verification through the QMS, the Weather Program collects and reports on numerous performance indicators at various levels, including those that assess progress toward wide-ranging outcomes through the Departmental Performance Report, the Federal Sustainable Development Strategy, the Canadian Environmental Sustainability Indicators program and departmental integrated planning, which includes reporting on risks and performance.

The MSC will continue to use omnibus surveys every 2 years to seek client feedback. Considering the evolving context in which the Program operates, the MSC is exploring the use of targeted surveys as a potential mechanism to assess the extent to which Canadians had access to specific weather warnings and took action or modified their behaviour in response to them.

Deliverable(s):

  1. Omnibus Survey
    1. Timeline: December 2023
    2. Responsible party: DG, Policy, Planning and Partnerships Directorate
  2. Complete at least 5 post event surveys.
    1. Timeline: June 2022
    2. Responsible party: DG, Prediction and Services Directorate
  3. The MSC will continue to present and discuss client feedback at mid-year and year end QMS reviews with the full QMS steering committee
    1. Timeline: Ongoing
    2. Responsible party: DG, Policy, Planning and Partnerships Directorate

Appendix A: Evaluation strategy

Evaluation issues and questions

The following questions related to the relevance and performance of the Weather Program were examined in the evaluation.

Relevance

Continued need for the program

Alignment with government priorities

Alignment with federal roles and responsibilities

Performance

Program efficiency

Effectiveness – Achievement of expected results

Evaluation approach and methodology

Several data collection methodologies were used to address the evaluation issues and questions. Evidence drawn from these methods informed the findings and conclusions.

Document, literature and data review

The document, literature and data review served to develop a thorough understanding of the Weather Program and to contribute as a line of evidence to address all evaluation questions. Examples of the types of materials reviewed included key policy setting documents, program planning and operational documents, departmental plans and results reports, previous evaluation and audit reports, performance measurement data, financial information, internal communications, peer-reviewed and grey literature, and other materials.

File review

A total of 28 project files were reviewed for projects funded by the MSC. For each project, the documents reviewed included the funding application, the contribution agreement, and progress and final reports. A standardized template was used to collect consistent information across all of the projects and to facilitate the analysis. The file review provided information to support an understanding of ECCC’s weather-related research activities, as well as its financial contributions to the WMO.

Key informant interviews

Key informant interviews were used to solicit informed opinions and observations on the evaluation questions from various stakeholders involved in or familiar with the Weather Program. A total of 44 key informants were interviewed by telephone, including:

Case studies

The evaluation team completed 2 case studies. One focused on how public alerting and dissemination of weather warnings has evolved since the NPAS was implemented in 2015. The other examined how effectively and efficiently the ECCC scientific research team and the MSC service delivery team worked together to deliver services for the 2015 Pan Am Games. Each case study consisted of a review of documents and key informant interviews. Interview questions relating to the case studies were integrated into the key informant interview guides, and all key informants were given the option of addressing these questions.

Limitations and mitigation strategies

The evaluation team encountered 2 main limitations while conducting the evaluation and put in place strategies to mitigate their impact.

Limitation 1: Potential key informants were identified through purposive sampling, and self-selected into the interview process once invited to participate. As such, the perspectives of key informants not represent the views of all relevant partners and stakeholders.

Limitation 2: Performance information was limited to support firm conclusions on the extent to which the Weather Program has achieved or made progress toward its outcomes.

Appendix B: References

CBC. (2018, September 23). Dunrobin families credit emergency warning system for saving lives. CBC News.

ECCC. (2015). Pan and Parapan American Games Project Client Satisfaction Evaluation.

ECCC. (2016a). Toronto 2015 Pan and Parapan American Games Experience. Environment Canada.

ECCC. (2016b). Departmental Performance Report 2015 to 2016, Environment and Climate Change Canada. Environment Canada.

ECCC. (2017). Compendium of Canada’s Engagement in International Environmental Agreements and Instruments: Convention of the World Meteorological Organization. Environment Canada.

ECCC. (2018a). Environment and Climate Change Canada Departmental Results Report 2017-18. Environment Canada.

ECCC. (2018b). Departmental Plan, 2018-19. Environment Canada.

ECCC. (2019). 2019-20 Departmental Plan. Environment Canada.

GOC. (2016). Pan-Canadian Framework on Clean Growth and Climate Change.

IIPCC. (2014). Climate Change 2014: Summary for Policymakers. Impacts, Adaptation, and Vulnerability.

IPCC. (2018). Summary for Policymakers. In: Global Warming of 1.5°C. An IPCC Special Report on the impacts of global warming of 1.5°C above pre-industrial levels and related global greenhouse gas emission pathways, in the context of strengthening the global response to the threat of climate change, sustainable development, and efforts to eradicate poverty.

National Research Council. (2010). When Weather Matters: Science and Service to Meet Critical Societal Needs. National Academies Press.

WMO (Ed.). (2007). Fifteenth World Meteorological Congress, Geneva, 7-25 May 2007: Abridged final report with resolutions. World Meteorological Organization.

WMO. (2015a). Role and operation of National Meteorological and Hydrological Services: a statement by the World Meteorlogical Organization for Directors of NMHSs World Meteorological (wmo.int)

WMO. (2015b). Valuing weather and climate: Economic assessment of meteorological and hydrological services. World Meteorological Organization.

Page details

Date modified: