Evaluation of Canada’s meteorological warning services for Arctic Ocean: chapter 1
Executive Summary
Background
This report presents the results of the evaluation of Canada’s Provision of Meteorological Warning Services for Defined Regions of the Arctic Ocean (the METAREA Initiative). The purpose of this evaluation was to assess the relevance and performance of the METAREA Initiative.
The METAREA Initiative is a five-year initiative, funded from 2010-11 to 2014-15 for $26.46 million. The goal of the METAREA Initiative is to strengthen safety and promote environmental protection for marine activity in the Arctic. The expected outputs of this activity are meteorological, sea-state and ice bulletins which will be transmitted to ships traversing METAREAs XVII and XVIII, and others monitoring the broadcasts, such as domestic and international safety and security agencies, to support safe and efficient marine navigation. Ultimately, the activities and outputs are expected to result in direct and immediate outcomes that lead to the following final outcomes:
- Canada demonstrates Arctic sovereignty;
- Reduced risk of marine safety incidents in METAREA XVII and XVIII related to meteorological, sea-state and ice conditions;
- Increased efficiency of marine navigation and economic activities in the North; and
- Reduced risk to human safety and property from hazardous meteorological, sea-state and ice conditions in the North.
The study period for the evaluation is a four-year timeframe from 2010-2011 to 2013-2014. Data collection for the evaluation occurred between November 2013 and February 2014, and involved a review of documentation and literature, an analysis of program performance data, and key informant interviews with internal and external stakeholders.
Findings and Conclusions
The findings of the evaluation are very positive overall, and indicate that the program is highly relevant, well managed and delivered, and making solid progress toward intended outcomes. Program delivery might be further strengthened through the refinement of existing performance measurement systems and by leveraging federal partnerships to expand information dissemination and enhance operations.
Relevance
The evaluation evidence confirms that the METAREA Initiative addresses a need for more robust meteorological maritime safety information (MSI) in the Arctic to accommodate an expected expansion in shipping and resource extraction activities. EC’s METAREA Initiative also complements other initiatives, such as DFO’s NAVAREA Initiative, that together provide both meteorological and navigational MSI in Canada’s assigned METAREAs. There are, however, several new initiatives underway in the North by other government departments that may have implications for the METAREA Initiative or offer opportunities for collaboration and synergies (e.g., Aboriginal and Northern Development Canada’s Canadian High Arctic Research Stations and the Canadian Coast Guard’s emerging strategy to create marine transportation corridors in the North).
The METAREA Initiative supports federal priorities through an enhanced presence and service provision in the Arctic METAREAs, consistent with Canada’s priorities for promoting Arctic sovereignty and supporting the development of potential in the North. The Initiative is aligned with departmental strategic outcomes and priorities related to ensuring that Canadians are equipped to make informed decisions on changing weather, water and climate conditions, and is aligned with federal responsibilities as a participating member state of the IMO. Accepting responsibility for METAREAs also demonstrates Canada’s compliance with international law and policy (e.g. Article 43 of United Nations Convention on the Law of Sea, Footnote 1 or the delivery of safety and weather services emphasized in the International Convention for the Safety of Life at Sea (SOLAS) Footnote 2).
Effectiveness
Evidence suggests that the METAREA Initiative is making appropriate progress toward meeting targets with respect to the quality and accessibility of meteorological MSI, as demonstrated by: the advancement of weather and ice forecast modelling in the Arctic; the program’s success in meeting or exceeding the vast majority of performance targets; progress of meteorological MSI bulletins toward compliance with international standards; and progress toward 100% geographic coverage of the METAREAs by 2015. Although progressing well on all intended outcomes related to the quality and accessibility of forecast information in the METAREAs, some respondents nonetheless expressed a desire for more information on ice edge than that required by international guidelines and better geographic coverage of the METAREAs during peak shipping season. Both of these issues are expected to be addressed before the project is completed in 2015.
Given the early stage of implementation and monitoring, there is insufficient evidence to assess the overall use of the METAREA Initiative information by mariners, or the indirect benefits of the Initiative for economic sectors and Northern residents. MSC is undertaking outreach to potential users to raise awareness and collect feedback to inform the METAREA Initiative.
With respect to final outcomes, the evaluation suggests that the METAREA Initiative reinforces Arctic sovereignty through a growing infrastructure, presence and enhanced capability in the North. Qualitative feedback also confirms the logical conclusion that the METAREA Initiative contributes to reduced risks to human safety and the environment, although there are limited data available and a relatively short program implementation period to demonstrate this trend base on shipping accident data.
Efficiency and Economy
Most of those involved in the METAREA Initiative feel the design and delivery of the METAREA Initiative has been appropriate to achieve intended outcomes (e.g., project management approach and organization of tasks into five components) and few alternatives to the program design were noted. The vast geography, delivery costs and small market would inhibit other potential providers.
The METAREA Initiative is generally being delivered as designed, despite occasional operational and staffing challenges. The METAREA Initiative five-year allocation was approximately $26 million, and actual spending on activities has been relatively consistent with the planned spending. Cost recovery is not viewed as viable and is felt to be at odds with the Initiative’s core mandate (i.e., information for maritime safety).
While Canada currently meets internationally prescribed standards regarding broadcasts, a potential design enhancement concerns additional (Internet-based) dissemination channels to allow for more detailed and on-demand information linked to geo-spatial capabilities.
Governance and roles and responsibilities are clear and appropriate. The Prince2 project management approach assists by clearly identifying roles and responsibilities, including decision-making authority of a Project Board and Executive.
The Initiative is viewed as efficient, in large part due to leveraging of existing core weather and ice production systems and collaborations. Detracting from efficiency are paperwork and approval burdens associated with the project management system, time required to secure multi-agency approvals for infrastructure installations, and difficulties in recruitment of personnel.
A performance measurement strategy has been approved for the program, and annual reports are issued based on a collection of almost 30 performance indicators, many of which have targets established. While considerable data collection is occurring through the Prince2 project management system, this primarily tracks output-related information. There is less client-centred information to assess achievement of intended outcomes regarding use and usefulness of the METAREA Initiative products and services.
Recommendations
The following recommendations are based on the findings and conclusions of the evaluation. The evaluation recommendations are directed to the Assistant Deputy Minister, Meteorological Services of Canada (ADM MSC), in view of the ADM’s responsibility for the overall management of the METAREA Initiative.
- Continue efforts to engage users (alongside the DFO NAVAREA Initiative) to better determine and prioritize their needs and options for disseminating meteorological (and navigational) MSI in the future.
- Revisit the program’s logic and performance measurement strategy to ensure that intended outcomes are sensible and measurable, and that the performance indicators are streamlined to meaningfully address program performance.
- Engage with domestic partners that are active in the Arctic to explore whether opportunities exist to collaborate on new and emerging initiatives.
The ADM MSC agrees with the recommendations and has developed a management response that appropriately addresses each of the recommendations. The full management response can be found in Section 6 of the report.