Evaluation of Canada’s meteorological warning services for Arctic Ocean: chapter 6

4.0 Findings

This section presents the findings of this evaluation by evaluation issue (relevance and performance) and by the related evaluation questions. For each evaluation question, a rating is provided based on a judgment of the evaluation findings.  The rating statements and their significance are outlined below in Table 2.  A summary of ratings for the evaluation issues and questions is presented in Annex 3: Summary of Findings.

4.1    Relevance

4.1.1 Continued Need for Program

Declining sea ice in the Arctic has prompted the international community to implement five new Arctic METAREAs to provide meteorological Maritime Safety Information in recognition of the anticipated increase in shipping and other activity in the Arctic.  There is no problematic duplication between the METAREA Initiative and other activities, although a number of complementary initiatives in the North were mentioned.

• The review of documentation indicates that interest in Arctic shipping is anticipated to increase due to declines in the extent and thickness of sea ice. Satellite data indicate that, since 1979, winter Arctic ice has decreased nearly four per cent each decade.^Footnote 5 By the summer of 2007, satellite imaging captured the opening of the Northwest arctic. In 2013, the Danish Nordic Orion was the first commercial bulk carrier to complete this voyage.
• The Arctic Council’s 2009 assessment of Arctic marine use acknowledged that the level of vessel activity is relatively small - representing less than two per cent of the world’s registered fleet of large ocean-going vessels. However, the report also noted that “the number of ships operating today in the Arctic is significant in the context of both the unique aspects of the Arctic environment and the insufficient infrastructure and emergency response in many parts of the region, relative to southern waters”.^Footnote 6
• Interviewees across all respondent groups confirm the need for the METAREA Initiative. They note that weather, sea and ice information were previously very limited in Arctic regions. Vessels accessed information on an ad hoc basis and in non-standard formats. At the same time, key informants noted that expected reductions in sea ice will lead to increases in Arctic shipping (commercial shipping traffic, resource development, scientific research or recreational travel) and to increased demand for information to assist vessels to safely navigate these waters.
• Most interviewees state that the Initiative does not duplicate other activities, though there are recommended redundancies to ensure continuity of information and safety of mariners (e.g., broadcasting information related to icebergs is considered both meteorological and navigational MSI).
• Key informants mentioned several new initiatives underway in the North by other government departments that may offer opportunities for collaboration. These include initiatives such as the Canadian High Arctic Research Station (CHARS) (funded by Aboriginal Affairs and Northern Development Canada (AANDC)), expansion of the CCG’s Automatic Identification System (AIS) that will support implementation of the e-Navigation concept, and CCG’s emerging strategy to create marine transportation corridors in the North.

4.1.2 Alignment with Federal Government Priorities

The METAREA Initiative is aligned with the federal government priority on the North, including promoting Arctic sovereignty and supporting potential development in the North. The Initiative represents an expansion of EC’s existing meteorological services, which support the strategic outcome related to “Canadians are equipped to make informed decisions on changing weather, water and climate conditions”.

• The review of documentation identifies the Arctic region as a priority of the federal government. In the 2010 Speech from the Throne^Footnote 7, the Government of Canada announced the establishment of “the Northern Strategy … to increase marine safety and reduce pollution from shipping and other maritime traffic.” The Northern Strategy^Footnote 8 was developed with a focus on four priority areas: exercising Arctic sovereignty; promoting social and economic development; protecting the North’s environmental heritage; and improving and devolving northern governance. In the 2013 Speech from the Throne^Footnote 9, Canada’s Northern sovereignty continued to be identified as a priority of the government.
• In support of this priority, Budget 2010 announced new funding for the METAREA Initiative. In addition, federal funding was also allocated to DFO for the parallel NAVAREAs and to the Canadian Space Agency for its RADARSAT Constellation Mission, which is an important data source for ice information.
• Most internal program interviewees confirmed the close alignment between the METAREA Initiative and Canada’s Northern Strategy. The Initiative supports the strategy by providing meteorological MSI to support safe navigation and economic activities in the North. Canada’s responsibility for the Arctic METAREAs and presence in the Arctic were perceived to reinforce Arctic sovereignty objectives.
• Evidence from documentation and EC interview respondents indicate that the Initiative is aligned with EC`s departmental strategic outcome to ensure “Canadians are equipped to make informed decisions on changing weather, water and climate conditions”.

4.1.3 Consistency with Federal Roles and Responsibilities

The METAREA Initiative is consistent with the federal responsibility for, and EC’s mandate to provide, meteorological information, including for marine safety and security. The METAREA Initiative fulfills Canada’s commitment as the Issuing Service for METAREAs XVII and XVIII. The federal level has the expertise, capacity, forecasting knowledge and resources to provide services in national and international waters. The Initiative utilizes the existing infrastructure and expertise of EC’s Weather and Environmental Prediction and Services program.

• The Constitution Act, 1867assigns legislative powers on Navigation and Shipping, and Sea Coasts and Inland Fisheries, to the Parliament of Canada.
• The Department of the Environment Act^Footnote 10 assigns jurisdiction over meteorology issues and responses to the Minister of the Environment, including the provision of environmental information to Canadians.
• According to most internal manager and staff interviewees, the Initiative addresses Canada’s international commitments as the Issuing Service for METAREAs XVII and XVIII which primarily involve Canadian waters but also waters outside Canada’s jurisdiction. The federal government has the expertise, installation capacity, forecasting knowledge and resources to provide services Canadian waters and waters adjacent to it.
• Accepting responsibility for METAREAs also demonstrates Canada’s compliance with international law and policy (e.g. Article 43 of the United Nations Convention on the Law of Sea (UNCLOS)^Footnote 11 or the delivery of safety and weather services emphasized in the International Convention for the Safety of Life at Sea(SOLAS) ^Footnote 12). SOLAS emphasizes areas within the boundaries of Canada’s exclusive economic zone and METAREAs extends Canada’s participation to areas adjacent to Canadian arctic waters.

4.2 Performance

4.2.1 Achievement of Intended Outcomes

To simplify reporting given the limited availability of end-user information at this stage, the findings are presented in three general areas: direct outcomes associated with the quality and accessibility of information; intermediate outcomes associated with the use of information by various groups; and final outcomes related to arctic sovereignty, marine safety, navigation and economic activity, and human safety and safety of property.

Evidence suggests that the METAREA Initiative is making appropriate progress toward meeting targets with respect to the quality and accessibility of information given the stage of implementation of the project. The meteorological MSI bulletins are expected to be in compliance with international standards this year.

• According to many key informants, a strength of the METAREA Initiative has been the advancement of the sophistication of the weather and ice forecast modelling in the Arctic. A review of the program’s performance measurement data indicates that additional data points provided by the expanded marine and surface weather stations have increased the number and quality of observations available. The forecasts have also been enhanced by the introduction of a wave model to the Arctic domain and a coupled forecasting model that integrates the atmosphere, ocean and ice conditions to reflect the dynamic interplay among these factors for greater accuracy in the models.
• During the peak shipping season, the weather operations desk is a 24/7 operation. The ice operations desk has not yet been fully operational with METAREA funding, though the staffing requirement for 10/7 has been supplemented by internal resources.
• Combined weather and ice bulletins are broadcast twice daily at a prescribed time by satellite (Inmarsat-C) south of 75°N and by CCG high frequency radio North of 75°N. The broadcasts are formatted for each marine area within the METAREAs to include the following information:
  • Warnings;
  • Synopsis; and
  • Weather and Ice Forecasts, which include marine wind, visibility, wave height, freezing spray and ice edge.
• Seasonal coverage of the bulletins will undergo expansion during the METAREA Initiative, with “null” bulletins issued once weekly during winter months.
• The program is working toward WMO compliance of the METAREAs bulletins, which is expected to occur by 2014. While the WMO does not assess compliance of countries that are responsible as Issuing Service for METAREAs in an official manner, Canada has provided progress reports to the WMO METAREA Coordinator and other international bodies (e.g., the Expert Team on Maritime Safety Services). According to the Initiative’s 2012 PM Annual Report, positive feedback has been received on Canada’s progress and manner of implementation with respect to the Arctic METAREAs. Similarly, the consensus among key informants is that Canada’s commitment to the IMO as an Issuing Service in the two METAREAs is being met, as are GMDSS guidelines for the bulletins.^Footnote 13 EC key informants added that international standards and manuals were used as guidelines for format and content.
• MSC monitors the products they disseminate to ensure they are broadcast successfully. Reliability of the transmission of the bulletins is very high - in 2011-12, 99 to 100 per cent of weather and ice forecasts in the north and south regions of the METAREAs XVII and XVIIII were issued on time and were successfully transmitted the first time.
• Mariners who receive the bulletins may contact the MSC directly to ask questions, make suggestions or request additional information. No complaints were received from mariners in 2011-12 related to access. However, there were some special requests for information (an indicator of the adequacy of the information provided in the bulletins). In total, 16 requests for additional information were received in 2011-12, the majority of these for additional information related to ice. The analysis presented in the Performance Measurement (PM) Annual Report indicated that ice edge information provided in the merged bulletin is not sufficient to help mariners operate safely in ice infested waters. A separate ice concentration bulletin is scheduled to begin in Summer 2014.
• Although appropriate progress is being made toward the achievement of intended objectives related to the quality or reliability of forecast information in the METAREAs, some suggestions for improvement had to do with:
  • a desire expressed by a few key informants for more information in the bulletins on ice beyond the information on ice edge that is required by international guidelines. This confirms the performance data above which noted a number of special requests for additional ice information from mariners in 2011-12. As mentioned, a separate ice concentration bulletin is scheduled to begin in Summer 2014; and
  • remaining service gaps in some geographical areas within the METAREAs. Access to meteorological MSI is not available in all marine areas in METAREAs XVII and XVIII during peak shipping season, however, this is expected to have been addressed by March 2015 at the completion of the Initiative.

Given the early stage of implementation and monitoring, there is very limited evidence to assess the overall use of the METAREA Initiative information by mariners, or the indirect benefits of the Initiative for economic sectors and Northern residents. While evaluation evidence suggests that mariners are using meteorological, sea-state and ice information in their operations, this evidence is anecdotal. MSC is undertaking outreach to potential users to raise awareness and collect feedback.

• The intended intermediate outcomes of the METAREA Initiative refer to use of meteorological, sea-state and ice information by:
  • Canadian and international safety and security agencies to support security operations and search and rescue response; and
  • Mariners to facilitate safe and efficient navigation.^Footnote 14
• In addition, as an ancillary benefit of investments in monitoring and research and development, the domestic services provided to Northerners and economic sectors operating in the Canadian North will be enhanced.
• The 2012 PM Annual Report indicated that public opinion research (POR) data expected from the Marine Weather Service Survey (2013) and from the 2012 CIS website user survey would help to assess achievement of direct and intermediate intended outcomes (access and use of the METAREA information). However, the Marine Weather Service Survey did not contain cross-tabulations for Arctic water mariners or Inmarsat information users, and therefore did not render findings attributable to the METAREA Initiative. Similarly, the CIS website user survey focused largely on satisfaction with the CIS website and therefore also did not produce findings relevant to the METAREA Initiative.
• Most key informant interviewees for this evaluation agreed that the METAREA Initiative has provided access to sufficient meteorological, sea-state and ice information to facilitate safe and efficient navigation. In addition, a few key informants emphasized the improvements from previously available formats. However, key informants stated that there has been little systematic feedback from mariners and confirmation of use is ad-hoc, for example, through professional contacts with mariners and industry groups at industry meetings or through the follow-up requests that are received from users for supplementary products. A few key informants noted that there has been more outreach to promote the services and to obtain feedback from users through Canadian Marine Advisory Councils in the North. Based on interviews and program documentation, MSC representatives plan on presenting at these councils and will attempt to survey these groups, though no survey results were available at the time of evaluation.

Canada’s active presence in the North through the METAREA Initiative was viewed by most as reinforcing Arctic sovereignty. Other intended outcomes related to reduced risk and improved efficiency of operations in the North are challenging to measure, particularly given the limited activity in the Arctic at this time. However, key informant evidence suggests a logical link between improved meteorological MSI and these intended outcomes.

Canada demonstrates Arctic sovereignty

• Most key informants, including international stakeholders, stated that Canada's presence and contribution through the METAREA Initiative reinforces Arctic sovereignty. Canada’s knowledge and capability in the North is perceived to be translating into leadership among international partners. A small number of key informants felt Arctic sovereignty could more properly be seen as an ancillary benefit of the METAREA Initiative.

Reduced risk of marine safety incidents in METAREAs XVII and XVIII related to meteorological, sea-state and ice conditions; Reduced risk to human safety and property from hazardous meteorological, sea-state and ice conditions in the north.^Footnote 15

• The review of documentation did not identify statistics on reduced risk of marine safety incidents. However, the 2012 PM Annual Report indicated that data are being collected by the CCG Marine Communications and Traffic Services (MCTS) and Transportation Safety Board (TSB) which will be a source of data in the future to determine associated risks and number of incidents with increases in traffic.
• Key informants were largely in agreement that improved meteorological and ice information contributes to reduction of risk. Most respondents feel that the METAREA bulletins and improvements in domestic products are providing better information (e.g., ice leads, ice spray) to support navigational decisions to improve safety, particularly in Canadian waters (less so in the high Arctic where coverage is not yet complete). However, there are measurement challenges given the absence of a baseline measure and fluctuations in traffic in the Arctic.

Increased efficiency of marine navigation and economic activities in the North

• The 2012 PM Annual Report indicates that weather and ice information can contribute to efficiencies of marine navigation and economic activities in the North. This perception was confirmed by key informants in the evaluation. As an example, for regulatory and safety reasons, mariners and shipping companies must factor weather and ice into their logistics. Improved information allows them greater flexibility in making decisions with respect to timing or routing of voyages in the Arctic during the operating season (e.g. improved timing, less down-time, more efficient routing) which can lead to significant savings.

No evidence of significant unintended outcomes of the METAREA Initiative was observed.

4.2.2 Efficiency and Economy

The design of the METAREA Initiative, including the organization of activities into five components, was held to be appropriate and effective. Few alternatives to the program design were noted, although a potential design enhancement that was identified is additional (internet-based) dissemination channels. Cost recovery is not viewed as viable for METAREA Initiative products and at odds with the core mandate of the Initiative to provide information for maritime safety.

• As described above, the METAREA Initiative has been organized into five components. Most internal key informants believe the design and organization of the METAREA Initiative is appropriate for achievement of intended outcomes. These interviewees noted that the components allow managers and leads to focus on their area of specialization.
• Specifically, the components feature:
  • investments in on-land, on-ice and in-the-water monitoring infrastructure to improve the number and quality of observations available to inform forecast models given the geographic expansion and extension to year-round service delivery. There was agreement among key informants that investments in monitoring were required given the data-sparse Arctic environment and, in fact, several key informants felt that additional infrastructure to improve the density of the observations would be beneficial in the future;
  • science support and technology transfer with a northern-focus was undertaken to expand research and development related to data assimilation and coupled modelling for the atmosphere, ocean and ice. Again, key informants note that MSC understanding of Arctic weather interactions such as the impact of ice coverage on the atmospheric conditions is underdeveloped compared to the South. Research and development work was funded to improve weather and ice forecasting in Arctic conditions;
  • service and platform design involved the establishment of a system to produce weather and ice forecasts for METAREAs XVII and XVIII. Coverage of the METAREAs has been approached in a step-wise fashion. In 2014, expansion is to include areas near the North Pole and North of Greenland, with complete geographic coverage expected by March 2015. Level of marine forecast and ice service provided for METAREAs marine forecast zones is based on navigable waters and marine activity within those zones. There is no service to ice-bound METAREAs marine forecast zones if there is no known marine activity. While the design of the program has situated the CIS under the service and platform design component, the role of ice was noted to be a critical factor influencing other components of the Initiative (e.g., operations, dissemination). To date, the bulletins have ice edge information, with plans in 2014 to include more robust information on ice concentration for improved MSI;
  • staffing of operational desks (one for weather, one for ice) produce the meteorological bulletins for Canada’s METAREAs; and
  • product dissemination using methods that are consistent with international standards and dictated by systems availability in domestic and High Arctic waters within the METAREAs. While the program has leveraged the DFO HF radio system for dissemination North of 75°N, the METAREA Initiative disseminates bulletins using the Inmarsat-C satellite South of 75°N which is also utilized by the CCG MCTS for navigational bulletins.
• There were few program alternatives to the METAREA Initiative identified by key informants. Several respondents noted that the vast geography of the METAREAs and the small market for the information would inhibit other providers such as the private sector from assuming a role in meteorological or ice forecasting in the Arctic of the kind delivered by the METAREA Initiative. The review of international literature also did not reveal any alternative, more effective models. While other countries have assumed responsibility for the other new METAREAs (i.e., Russia and Norway), there has been limited exchange and coordination to date with these countries (international meetings are planned for 2014). According to program managers and confirmed by international bodies, Canada’s progress on the implementation of service in the new METAREAs is similar to or more advanced than other countries.
• One potential design enhancement mentioned by a few program and international respondents is to consider additional methods of dissemination of forecast bulletins in the future. While Canada and other countries meet international (WMO) prescribed standards regarding broadcasts (CCG HF radio telex and Inmarsat-C satellite), future channels could likely include web-based methods. In the Arctic, there are currently technological limitations in broadband width for these kinds of transmissions, however, as there are technological opportunities to do so, web-based transmissions would allow for more detailed and on-demand information linked to geo-spatial capabilities. The evolution of dissemination strategies for marine weather and ice information was a key issue that was raised in the 2011 Evaluation of the Services to Marine Transportation Sub-activity, and which resulted in the establishment of a strategy to manage new approaches to disseminations based on client needs. Likewise, the CCG 2013 Commanding Officer Survey identified suggestions for colour weather charts and more frequent updates of satellite images.^Footnote 16
• Internal key informants were probed about opportunities to pursue cost-recovery within the METAREA Initiative. No respondents felt cost-recovery is a viable option for METAREA Initiative products and most key informants felt it was inappropriate given that the intent of the Initiative is core to the federal mandate and responsibility (see Section 4.1.3). A few key informants also noted that while it may be feasible to create enhanced, custom forecast products for mariners, the market is very small and such activities are now undertaken by commercial providers.

The roles and responsibilities of the METAREA Initiative stakeholders are clearly defined and understood. The governance structure of the Initiative is appropriate and efficient according to most internal interviewees. The Prince2 project management approach assists by clearly identifying roles and responsibilities, including decision-making authority of a Project Board and Executive.

• While there are many stakeholders involved in the Initiative, overall, most staff interviewees indicate that the roles and responsibilities of the various METAREA Initiative stakeholders are clearly defined and understood. The governance structure, consisting of a METAREA project manager responsible for overall coordination and leads for each of the 5 program components, is felt to be appropriate and efficient according to most internal interviewees. A few interviewees indicate that the Prince2 project management model^Footnote 17and Project Board decision-making authority contributes to clear governance by defining the roles and responsibilities of participants, elaborating the authority of the Project Board and Executive,^Footnote 18and documenting meeting outcomes.
• While governance of the Initiative was perceived to be positive overall, a few key informants noted some potential to improve: 1) the level of coordination across the components (resourcing and task allocation more closely linked to clearly defined objective); 2) coordination with DFO's regionalized Science structure for research and development work requiring additional time and effort; 3) funding for Component Lead positions;^Footnote 19 and 4) clarity in allocating responsibility for certain Initiative budget items to the individual components.

The METAREA Initiative is being delivered as designed and intended. Operational challenges, often related to geographic and climate conditions in the Arctic, have not compromised most targets for deliverables. Adequate staffing (due to limited resources or recruitment difficulties) has been problematic for one component.

• Program performance data and internal key informants confirm that the METAREA Initiative is being delivered as designed and intended. The most recent 2012 PM Annual Report for the Initiative indicates that the implementation of the METAREA Initiative is on track to be completed by the end of March 2015 as planned.^Footnote 20 Most key informants confirm that the METAREA Initiative has been implemented as intended (all major milestones have been met), owing in part to the disciplined application of the project management system according to some.
• According to key informants, challenges in implementation have largely been of an operational nature and are often related to the remote and massive geography of the METAREAs and the harsh Arctic climate which restricts the window of opportunity to conduct installation work. Other implementation challenges that were mentioned by some key informants included: need for multi-agency approvals to install monitoring equipment at some sites and consequent delays; organizational changes external to the Initiative that created uncertainty about roles or delays (e.g., creation of SSC and associated delays in IT management); and difficulties in recruitment and staffing the operational forecast desks. With respect to the latter, the ice services desk, for example, has not been fully operational with METAREA Initiative funding due to the staggered allocation of FTEs across the fiscal years of the Initiative for the positions required to staff the desk, as well as recruitment challenges. To address staffing challenges of the operational desks, internal MSC and CIS resources have been utilized. Finally, a few key informants noted that an expansion to become the Issuing Service in Hudson’s Bay (included within a much larger Northern Atlantic METAREA IV, which is a US responsibility) was not in the original plan for the Initiative, though has not reportedly created significant difficulties for the program. 

Program inputs are appropriate to achieve intended outcomes. Analyses of available financial data show an increasingly close relationship between allocated and actual expenditures for each of the five components of the Initiative, following lapses in the inaugural year due to late receipt of funds.

• Table 3 summarizes annual allocated and actual spending on the METAREA Initiative components for the first four years of the Initiative. In the first year of the Initiative, there was underspending on the components (about 46 per cent considering all components) due to late receipt of funds in the fiscal year. For the remaining years of the Initiative, there is a closer and increasing alignment between allocated and expended funds. Internal key informants were generally satisfied with the allocation of funds to the Initiative and across the components.

The project management approach uses existing procedures such as PRINCE2 and Prince2-based EC CSB Project management templates. In addition, reports and documents are posted in an existing, shared access, ECollab site.^Footnote 22 Key informants view the Initiative as efficient, in large part because it is an expansion of existing services and has leveraged existing core weather and ice production systems, as well as collaborations.

• According to key informants, factors supporting efficiency (mentioned by at least two key informants) include:
  • leveraging of capacity and expertise within the MSC. The integrated Arctic-specific weather and ice forecasting and prediction models are being developed within the context of EC’s existing suite of operational forecast modelling systems at MSC and the CIS. Thus, the expansion of services under the METAREA Initiative has occurred within existing core production systems which are efficient and will benefit over time from other MSC-wide projects seeking to improve the efficiency of the overall weather enterprise; and
  • leveraging of partner contributions, such as in the deployment of buoys (DND), utilization of capabilities/observations of research vessels in the Arctic for monitoring and infrastructure (in exchange for access to data) (University of Manitoba), and use of the CCG FH radio telex transmission equipment for dissemination of METAREA bulletins in the High Arctic.
• Most key informants felt Prince2 contributed to efficiency of the Initiative through its rigorous project management methodology that tracks progress on implementation and early warning when progress or expenditures drift out of established tolerance. A small number noted that aspects of the system detract from efficiency, including excessive paperwork and reporting burden and bureaucratic layering between the Activity Leads and managers. Time required to secure multi-agency approvals for installation of monitoring equipment and difficulties in recruitment of personnel were cited by a small number of key informants as additional challenges to efficient delivery.

The evaluation determined that performance measures are in place and are being tracked using a Performance Strategy and Framework and through the Prince2 project management system. However, while the performance data are satisfactory for capturing progress toward deliverables, measuring outcomes from the user community represents a significant challenge.

• A Performance Strategy for the METAREA Initiative was developed in 2011 and roles and responsibilities with regard to the implementation of the strategy have been established. The responsibility to gather and analyze the performance data rests with each of the five Component Leads, while responsibility to report on the performance data rests with the Executive Director of the National Service Operation through the Signature Project Lead and the METAREA Project Manager. Performance reports are to be submitted to the Director General of Weather and Environmental Prediction and Services who has the overall responsibility of the delivery of the Initiative. Regular updates are also to be provided to the IMO on the status of the test and implementation phases.
• Two Performance Measurement Annual Reports have been produced for the METAREA Initiative (2011 and 2012) and the next Performance Measurement Annual Report (2013) is scheduled to be available by 2014.^Footnote 23 The reports include an analysis of overall performance, and a “Results Place Mat” table reports on 29 performance indicators using qualitative and quantitative data to measure program outputs and intended outcomes. The table reports on the target, reported value, brief analysis, and issues and path forward. Detailed results in narrative form are provided for each indicator. The annual reports are very informative in terms of measuring implementation and outputs of the Initiative, as well as direct outcomes pertaining to the accuracy, reliability and adequacy of the forecast bulletins. As reported in the previous section, there are challenges in measuring and reporting indicators related to the intended intermediate and final outcomes of the Initiative (use and usefulness of the METAREA Initiative information). In addition to the performance measurement framework, the Prince2 project management system has a reporting regime that requires regular updates and progress reporting.
• In addition, End Stage Reports are produced. The purpose of the End Stage Report is to give a summary of progress to date, the overall project situation and sufficient information to ask for Project Board approval to proceed with the next stage of the project. Recommendations of the March 2013 End Stage 2 Report^Footnote 24 are to continue to use Prince2 and hold planning sessions twice per year.
• Internal program management and staff key informants were of the opinion that METAREA Initiative performance data are appropriate and reasonable, particularly in reporting on implementation and capturing progress toward deliverables. The Prince2 documentation effectively monitors task completion for each component through the use of monthly Checkpoint reports and quarterly Highlights progress reports. The Prince2 reports inform decision-making at the Project Board level when implementation or expenditures diverge beyond an established tolerance. Annual performance and periodic briefing materials also contribute to the reporting on performance. The annual performance reports are used to inform Departmental Performance Reports.
• According to key informants, performance reporting has been less effective in capturing achievement of intended intermediate and longer-term outcomes of the METAREA Initiative. Initiative management and staff attributed the lack of outcome data to challenges in measuring the awareness, use and usefulness of the bulletins for mariners who are a small, international and mobile group with whom it is difficult to consult. For example, there is estimated to be fewer than 100 vessels that traverse Canada’s METAREAs. Therefore, obtaining user feedback, such as through surveys, has been difficult. A similar challenge affected this evaluation of the METAREA Initiative and was also identified as an area for improvement in the 2011 Evaluation of the Services to Marine Transportation Sub-activity.