Weather warning index

Access PDF (783 kB)

When severe weather threatens, Environment and Climate Change Canada issues public weather alerts. This allows people in affected areas to take steps to protect themselves and their property from harm. A weather warning alert is an urgent message that severe weather is either occurring or will occur. The index tracks the extent to which the severe weather warning system is providing Canadians with warnings with sufficient lead time.

Results

Key results

  • The overall index score improved from 7.6 to 8.3 between the 3-year periods 2009 to 2011 and 2011 to 2013. It remained at that level until the 2013 to 2015 period.
  • The overall index score declined slightly, to 8.2, for the period 2014 to 2016.

Weather warning index for a 3-year moving average, Canada, periods 2009 to 2011 through 2014 to 2016

Line Chart - See long description below
Long description

The chart shows weather warning index component scores for the 6 3-year moving averages available, by warning type: wind, freezing rain, rain, marine gale, snow and severe thunderstorm. The overall index score for each 3-year moving average is also provided.

Data for this chart
Weather warning index for a 3-year moving average, Canada, periods 2009 to 2011 through 2014 to 2016
Warning type 2009 to 2011
score
2010 to 2012
score
2011 to 2013
score
2012 to 2014
score
2013 to 2015
score
2014 to 2016
score
Wind 8.6 8.8 9.0 9.1 9.1 9.1
Freezing rain 7.7 8.5 8.8 9.0 8.8 8.8
Rain 8.0 8.4 8.7 8.6 8.2 8.0
Marine gale 8.3 8.2 8.4 8.2 8.1 7.9
Snow 7.4 7.2 7.8 7.6 7.8 7.8
Severe thunderstorm 4.9 5.8 6.8 6.5 7.7 7.4
Overall score 7.6 7.9 8.3 8.3 8.3 8.2

Download data file (Excel/CSV; 1.01 kB)

How this indicator was calculated

Note: The index value is 10 if all extreme weather events in targeted areas were preceded by a warning with sufficient lead time, as per the weather warning performance targets. The overall index and its components are expected to exhibit modest fluctuations due to year-to-year changes in the predominant weather patterns.
Source:
Environment and Climate Change Canada (2017) Meteorological Service of Canada – Integrated Planning and Performance Management Division.

More information

Each year, the Meteorological Service of Canada issues 15 000 severe weather warnings, on average. Warnings are usually issued 6 to 24 hours in advance, although some severe weather (such as thunderstorms) can occur rapidly, with less than a half-hour's notice.

About the indicator

About the indicator

What does the indicator measure

The indicator tracks the performance of Environment and Climate Change Canada's severe weather warning system in providing Canadians with warnings with sufficient lead time. The index is calculated based on information from 6 warning types that are representative of Canada's climate. These warning types are severe thunderstorm, rainfall, freezing rain, wind, snowfall and marine gale.

Why is this indicator important

Weather warnings are invaluable for the protection of life and property. They are critical to provincial and municipal emergency response organizations for managing flood control, sewer overflow and stormwater run-off. Weather-sensitive users such as snow removal operators and outdoor recreational enthusiasts also rely on weather warnings.

The indicator is intended to assess the performance of Environment and Climate Change Canada's severe weather warning program. Over time, Canadians will be able to track trends in the program's performance.

Data sources and methods

Data sources and methods

What are the data sources

The data on warnings issued are from Environment and Climate Change Canada. The data on severe weather events are from weather reports submitted by the public and data collected by Environment and Climate Change Canada's observation network.

More information

The indicator includes timeliness performance information for the identified 6 severe weather warning types. The index components are calculated using warning data from a set of selected geographic regions that are representative of Canada's climate and for which Environment and Climate Change Canada has sufficient warning event information.

This information is compiled by comparing warnings issued by Environment and Climate Change Canada against severe weather reports submitted by the public and data collected by Environment and Climate Change Canada's observation network. Data for each calendar year are available by the spring of the following year.

Environment and Climate Change Canada's national weather forecast and warning system relies on several observation networks to detect changes in the atmosphere and the development of threatening conditions. The monitoring infrastructure runs 24 hours a day, 7 days a week, 365 days a year. It includes 31 weather radar stations, over 80 lightning detection sensors, approximately 1 580 surface weather and climate stations, 46 weather buoys, 54 ships equipped with automated observation systems, 31 stations for launching balloon-borne observations of the upper atmosphere, as well as 10 geostationary and 4 polar orbiting weather satellite receiving stations. Hundreds of volunteer weather observers and severe weather watchers from coast to coast also provide extremely valuable information to Environment and Climate Change Canada.

Public forecast regions used in the Weather warning index for rainfall, snowfall, freezing rain and wind warnings

Public forecast regions used in the Weather warning index for rainfall, snowfall, freezing rain and wind warnings
Long description

The map of Canada shows the public forecast regions used in the Weather warning index for rainfall, snowfall, freezing rain and wind warnings. These regions are highlighted on the map. Les Suetes wind warnings are issued only for 1 forecast region in Nova Scotia, and Wreckhouse wind warnings are issued only for 1 region in Newfoundland and Labrador. A complete list of the regions can be found in the Annex.

Note: Data from public forecast regions that were substituted are used in historical data up to the year of substitution. Some regions are not visible at the national scale. See Annex for the complete list.
Source:
Environment and Climate Change Canada (2015) Forecast Regions of Canada. Meteorological Service of Canada – Integrated Planning and Performance Management Division.

Public forecast regions used in the Weather warning index for severe thunderstorm warnings

Public forecast regions used in the Weather warning index for severe thunderstorm warnings
Long description

The map of Canada shows the public forecast regions used for the severe thunderstorm warning component of the Weather warning index. These regions are highlighted on the map. A complete list of the regions can be found in the Annex.

Note: Data from public forecast regions that were substituted are used in historical data up to the year of substitution. Some regions are not visible at the national scale. See Annex for the complete list.
Source:
Environment and Climate Change Canada (2015) Forecast Regions of Canada. Meteorological Service of Canada – Integrated Planning and Performance Management Division.

Marine forecast areas used in the Weather warning index for marine gale warnings

Marine forecast areas used in the Weather warning index for marine gale warnings
Long description

The map of Canada shows all marine forecast areas. The marine forecast areas used for the marine gale warning component of the Weather warning index are highlighted on the map. A complete list of the areas can be found in the Annex.

Note: Data from marine forecast areas that were substituted are used in historical data up to the year of substitution. Some areas are not visible at the national scale. See Annex for the complete list.
Source:
Environment and Climate Change Canada (2015) Canadian Marine Warning Program. Meteorological Service of Canada – Integrated Planning and Performance Management Division.

How is this indicator calculated

The index is calculated based on information from 6 warning types that are representative of Canada's climate. These warning types are severe thunderstorm, rainfall, freezing rain, wind, snowfall and marine gale. For each warning type, a component score is determined based on the warning's accuracy in predicting an actual severe weather event and its timeliness in comparison to the lead times identified within Environment and Climate Change Canada's warning performance targets .

More information

Definitions

Several definitions related to the index are provided below.

Accuracy definitions

An event is an individual instance of a weather or environmental hazard that meets hazard criteria thresholds.

A hit is a warning event that was forecast and occurred.

A miss is a warning event that occurred but for which no advance warning was provided or no alert was issued.

A false alarm occurs when a warning event was forecast but conditions did not reach warning criteria.

A correct negative corresponds to an instance where no warning was issued and no event occurred.

Timeliness definitions

The event time is the time at which the criteria threshold is first met for an event. For alerts where the criteria thresholds are accumulated precipitation, the event time is the time at which the total accumulated precipitation first equals or exceeds the criteria amount.

The issue time is the time at which the alert was transmitted by the Environment and Climate Change Canada forecaster.

The lead time is the difference between the time that an alert is issued and the event time. For example, if an alert is issued at 09:00 and the event time is 09:30, the lead time is 30 minutes.

The target lead time is Environment and Climate Change Canada's performance goal for the timeliness of alerts. The target lead time is intended to provide adequate time for the public to take appropriate action when alerted of a predicted event. Other factors such as the predictability of an event and the ability of the public and media to receive alert bulletins may influence the actual lead time provided.

Methodology

The index is constructed by taking a weighted mean of timeliness and accuracy statistics. To reduce volatility from year to year, a 3-year moving average is used for reporting purposes.

Weighting

A 20 % weight is assigned to the marine gale component of the index. The remaining severe weather warning types (severe thunderstorm, wind, rainfall, snowfall and freezing rain) are assigned a combined 80% weight in the index. Each of these land components is weighted based on its frequency of occurrence during the reporting periods.

Timeliness factors

Timeliness statistics reflect the average lead time and target lead time for each of the 6 warning types, as determined by the warning performance targets described in Table 1.

To calculate the average lead time for each warning component, the warning issue time is subtracted from the time when the warning event occurred. Missed events are assigned zero lead time. The lead times obtained are then averaged over the year to compute the "average lead time." This value is then compared to the target lead time for the specific warning type. The resultant value represents the "timeliness" aspect of the index.

Table 1. Target lead time by severe warning type (warning performance target)
Severe weather warning type Target lead time
Rainfall greater than or equal to 12 hours
Freezing rain greater than or equal to 6 hours
Wind greater than or equal to 12 hours
Snowfall greater than or equal to 18 hours
Severe thunderstorm greater than or equal to 30 minutes
Marine gale greater than or equal to 18 hours
Accuracy factors

Accuracy statistics are expressed in terms of the extremal dependency index, which incorporates the number of successfully forecast events (hits), the number of missed events and the number of false alarms.

Long description

The extremal dependency index is calculated by dividing the difference between the logarithm of the false alarm rate and the logarithm of the hit rate by the sum of the logarithm of the false alarm rate and the logarithm of the hit rate.

The hit rate is calculated using the number of hits and the number of misses. The false alarm rate is based on the number of false alarms and the number of correct negatives. The number of correct negatives is an estimate of the number of times during the reporting period that forecasters had to consider whether or not to issue a warning, and correctly decided against issuing one. Since the extremal dependency index is a ratio of logarithms, it does not matter which base is used.

Long description

The hit rate is calculated by dividing the number of hits by the sum of the number of hits and misses.

Long description

The false alarm rate is calculated by dividing the number of false alarms by the sum of the number of false alarms and correct negatives.

Assumptions
  • All lead times are greater than or equal to zero
  • Individual lead times greater than twice the target lead time are assigned a value of twice the target lead time
Index calculation

The scoring formula for each warning component is as follows:

Case 1

If the average lead time is equal to or greater than the target lead time, the index component score becomes

Long description

If the average lead time is equal to or greater than the target lead time, the index component score can be calculated using the following equation: {extremal dependency index + 0.5 x [(average lead time / target lead time) -1] x (1 - extremal dependency index)} x 10.

Case 2

If the average lead time is less than the target lead time, the index component score becomes

Long description

If the average lead time is less than the target lead time, the index component score can be calculated by taking the product of the extremal dependency index and the ratio of the average lead time over the target lead time, and then multiplying it by 10.

The Weather warning index calculation, timeliness and accuracy statistics and component scores for the 3-year moving average (2014, 2015 and 2016) by warning type: rain, snow, freezing rain, wind, severe thunderstorms and marine gale is presented in Table 2.

Table 2. Weather warning index calculation, timeliness and accuracy statistics, 2014 to 2016
2014 to 2016 Rain Snow Freezing rain Wind Severe thunderstorm Marine gale
Hits 201 188 141 261 99 2 536
Misses 119 73 50 79 47 546
False alarms 139 112 61 143 631 1 018
Correct negatives 6 741 9 752 3 798 19 767 27 573 3 595
Total 7 200 10 125 4 050 20 250 28 350 7 695
Hit rate 0.63 0.72 0.74 0.77 0.68 0.82
False alarm rate 0.02 0.01 0.02 0.01 0.02 0.22
Extremal dependency index 0.787 0.863 0.864 0.898 0.814 0.771
Average lead time (in hours) 12.26 15.97 6.92 14.36 0.37 21.41
Target lead time (in hours) 12 18 6 12 0.5 18
Weight 25 21 15 27 12 100
Weather Warning Index component score 7.95 7.83 8.76 9.10 7.42 7.92
Individual weighted Weather warning index component score 1.62 1.30 1.06 1.97 0.69 1.58
Change relative to individual weighted Weather warning index for 2013 to 2015 -0.02 -0.14 -0.11 0.07 0.14 -0.04

The Weather warning index is the sum of the individual weighted Weather warning index component scores: 8.22

Note: False alarms are not available for severe thunderstorm alerts. A convective warning bias of 5 is used to estimate the number of false alarms, where:

False alarms = convective warning bias x (hits + misses) - hits

What has recently changed

There have been changes in the selection of public forecast regions used to calculate the index. Some regions were added or removed from the indicator while others were substituted.

More information

Historical values of the index have been recalculated using the new regions to allow for comparisons among the different 3-year periods included in this release of the indicator. New index values reflect the fact that data from the removed regions no longer contribute to the index, while values from the added regions are taken into account. For substituted regions, data from the original regions are used in the index calculation up to the year of substitution. Table 3 summarizes the removals, additions and substitutions of forecast regions that have taken place since the last release.

Table 3. Changes to public forecast regions used to calculate the Weather warning index
Warning type
Removals
Additions Substitutions
Public forecast regions for rainfall, snowfall, freezing rain and wind warnings
  • Chibougamau (Quebec)
  • Fermont (Quebec)
  • Eastern Townships (Quebec)
  • Lac-Saint-Jean (Quebec)
  • From 2012 onwards, Goose Bay and vicinity was renamed Upper Lake Melville (Atlantic).
  • From 2012 onwards, St. Georges was replaced by Channel – Port aux Basques and vicinity (only for Wreckhouse wind warnings) (Atlantic).
  • From 2014 onwards, Kapuskasing – Hearst was replaced by Timmins – Cochrane (Ontario).
Public forecast regions for severe thunderstorm warnings n/a n/a
  • From 2012 onwards, Goose Bay and vicinity was renamed Upper Lake Melville (Atlantic).
  • From 2014 onwards, Kapuskasing – Hearst – Smooth Rock Falls was replaced by Timmins – Cochrane – Iroquois Falls (Ontario).
Marine forecast regions for marine gale warnings
  • Northern Lake Huron (Ontario)
  • Southern Lake Huron (Ontario)
n/a
  • From 2013 onwards, Donnacona to L'Isle-aux-Coudres was replaced by Beauport to L'Isle-aux-Coudres (Quebec).
  • From 2013 onwards, Tadoussac to Pointe-des-Monts was replaced by Tadoussac to Pointe à Michel and Pointe à Michel to Pointe-des-Monts (Quebec).
  • From 2015 onwards, Georges Bank was replaced by Browns Bank (Atlantic).

Note: n/a = not applicable.

What are the caveats and limitations

The calculation of the index does not consider when weather warnings are received by Canadians, as this factor varies considerably depending upon how warnings are received (for example, via a media outlet or a website).

The index does not represent all forecast regions in Canada; instead, it represents areas that regularly have sufficient event data against which to compare severe weather warnings.

Resources

Resources

References

Environment and Climate Change Canada (2013) Marine Warnings and Watches Program. Retrieved on May 31, 2017.

Environment and Climate Change Canada (2016) Criteria for public weather alerts. Retrieved on May 31, 2017.

Further reading

Ferro C et al. (2011) Extremal Dependence Indices: Improved Verification Measures for Deterministic Forecasts of Rare Binary Events. Weather and Forecasting 26(5): 699–713.

Wilson L and Giles A (2013) A new index for the verification of accuracy and timeliness of weather warnings. Meteorological Applications 20: 206–216.

Related information

Annex

Annex

Public forecast regions used in the Weather warning index for rainfall, snowfall, freezing rain and wind warnings

Pacific and Yukon

  • Metro Vancouver
  • Fraser Valley – West including Abbotsford
  • East Vancouver Island
  • Greater Victoria
  • North Coast – Inland Sections
  • Central Okanagan – including Kelowna
  • West Kootenay
  • Prince George
  • BC North Peace River
  • Whitehorse

Prairie and Northern

  • City of Winnipeg
  • Thompson – Nelson House – Split Lake
  • City of Regina
  • City of Saskatoon
  • Prince Albert – Shellbrook – Spiritwood – Duck Lake
  • City of Calgary
  • City of Edmonton – St. Albert – Sherwood Park
  • Yellowknife Region
  • Resolute
  • Iqaluit

Ontario

  • Windsor – Essex – Chatham-Kent
  • London – Middlesex
  • City of Hamilton
  • City of Toronto
  • City of Ottawa
  • North Bay – West Nipissing
  • Greater Sudbury and Vicinity
  • Sault Ste. Marie – Superior East
  • City of Thunder Bay
  • 2009 to 2013: Kapuskasing – Hearst
    Since 2014: Timmins – Cochrane

Quebec

  • Metro Montréal – Laval
  • Eastern Townships
  • Quebec City
  • Saguenay
  • Lac-Saint-Jean
  • Rimouski – Mont Joli
  • Forillon National Park – Gaspé – Percé
  • Abitibi
  • Sept-Iles – Port-Cartier
  • Blanc Sablon
  • Kuujjuaq

Atlantic

  • Saint John and County
  • Moncton and Southeast New Brunswick
  • Halifax Metro and Halifax County West
  • Queens County PEI (Charlottetown)
  • Sydney Metro and Cape Breton County
  • Inverness County – Mabou and north (Les Suetes, wind only)
  • 2009 to 2011: St. Georges
    Since 2012: Channel-Port aux Basques and vicinity (Wreckhouse, winds only)
  • Gander and vicinity
  • St. John's and vicinity
  • Upper Lake Melville (known as Goose Bay and vicinity before 2012)
  • Labrador City and Wabush
Public forecast regions used in the Weather warning index for severe thunderstorm warnings

Pacific and Yukon

  • Metro Vancouver
  • Fraser Valley – West including Abbotsford
  • East Vancouver Island
  • Greater Victoria
  • North Coast – Inland Sections
  • Central Okanagan – including Kelowna
  • West Kootenay
  • Prince George
  • BC North Peace River
  • Whitehorse

Prairie and Northern

  • City of Winnipeg
  • City of Brandon
  • R.M. of Dauphin including Sifton and Valley River
  • City of Regina
  • City of Saskatoon
  • City of Prince Albert
  • City of Calgary
  • City of Edmonton – St. Albert – Sherwood Park
  • City of Lethbridge
  • County of Grande Prairie near Grande Prairie and Wembley

Ontario

  • Windsor – Leamington – Essex County
  • London – Parkhill – Eastern Middlesex County
  • City of Hamilton
  • City of Toronto
  • Ottawa North – Kanata – Orleans
  • North Bay – Powassan – Mattawa
  • Greater Sudbury and vicinity
  • Sault Ste. Marie – St. Joseph Island
  • City of Thunder Bay
  • 2009 to 2013: Kapuskasing – Hearst – Smooth Rock Falls
    Since 2014: Timmins – Cochrane – Iroquois Falls

Quebec

  • Montréal Island area
  • Huntingdon area
  • Saint-Hyacinthe – Acton Vale area
  • Rawdon – Joliette area
  • Lachute area
  • Mont-Tremblant – Sainte Agathe area
  • Papineau-Labelle Reserve area
  • Val D'Or – Louvicourt area
  • Granby – Waterloo area
  • Quebec City area
  • Alma – Desbiens area

Atlantic

  • Saint John and County
  • Moncton and Southeast New Brunswick
  • Halifax Metro and Halifax County West
  • Queens County PEI (Charlottetown)
  • Sydney Metro and Cape Breton County
  • Oromocto and Sunbury County
  • Gander and vicinity
  • St. John's and vicinity
  • Upper Lake Melville (known as Goose Bay and vicinity before 2012)
  • Labrador City and Wabush
Marine forecast areas used in the Weather warning index for marine gale warnings

Pacific and Yukon

  • Strait of Georgia – south of Nanaimo
  • Strait of Georgia – north of Nanaimo
  • Juan de Fuca Strait – East Entrance
  • West Vancouver Island North
  • Queen Charlotte Sound – Western half
  • Hecate Strait - Northern Half

Prairie and Northern

  • Great Slave Lake Basin
  • Tuktoyaktuk
  • Frobisher Bay
  • West Brevoort – southern half
  • Barrow
  • Lake Manitoba
  • Lake Winnipeg – South Basin
  • Lake Winnipeg – North Basin

Ontario

  • Western Lake Superior
  • Eastern Lake Superior
  • Western Lake Erie
  • Eastern Lake Erie
  • Western Lake Ontario
  • Eastern Lake Ontario

Quebec

  • 2009 to 2012: Donnacona to Isle-aux-Coudres
    Since 2013: Beauport to L'Isle-aux-Coudres
  • Tadoussac to Pointe à Michel
  • Pointe à Michel to Pointe-des-Monts
  • Pointe-des-Monts to Anticosti – southern half

Atlantic

  • 2009 to 2014: Georges Bank
    Since 2015: Browns Bank
  • Sable
  • Gulf – Magdalen
  • Northeast Gulf
  • Southwest Coast
  • Northeast Coast
  • Northern Grand Banks
Report a problem or mistake on this page
Please select all that apply:

Privacy statement

Thank you for your help!

You will not receive a reply. For enquiries, contact us.

Date modified: