Evaluation at a glance: Horizontal evaluation of the adaptation theme of Canada’s clean air agenda

About the program

• The Clean Air Agenda Adaptation Theme was a five-year (2011-12 to 2015-16) $181.6 million initiative aimed at helping Canadians adapt to climate change and delivering on the federal role as defined by the Federal Adaptation Policy Framework. The Theme was coordinated by Environment and Climate Change Canada.
• The Adaptation Theme was one of five program themes under the Government of Canada’s Clean Air Agenda (CAA), to address climate change and air pollutants. As of December 2016, the government’s climate change activities fall under the Pan-Canadian Framework on Clean Growth and Climate Change.
• Programming under Adaptation was organized into four streams with a total of 10 program elements. Each program element was delivered by one or more of nine federal partners.

Enhance the science foundation to understand and predict climate and assess climate change impacts

• Aquatic Climate Change Adaptation Services Program  –  Department of Fisheries and Oceans
• Climate Change Prediction and Scenarios Program – Environment and Climate Change Canada
• Understanding Climate-Driven Ecological Changes in Canada’s North– Parks Canada Agency

Enhance public health and safety

• Heat Alert and Response Systems – Health Canada
• Preventative Public Health Systems and Adaptation to Climate Change Program – Public Health Agency of Canada

Build resilience in the North and climate-sensitive Aboriginal communities

• Climate Change and Health Adaptation Program for Northern First Nations and Inuit Communities– Health Canada
• Climate Adaptation and Resilience Program for Aboriginals and Northerners – Indigenous and Northern Affairs Canada
• Integrating Adaptation into Codes and Standards for Northern Infrastructure (also known as the Northern Infrastructure Standards Initiative) – Standards Council of Canada and Indigenous and Northern Affairs Canada

Enhance the competitiveness of climate-sensitive economic sectors and systems

• Enhancing Competitiveness in a Changing Climate (also referred to as the NRCan Adaptation Program) – Natural Resources Canada (NRCan)
• Northern Transportation Adaptation Initiative – Transport Canada

What the evaluation found

Relevance

• Evaluation findings indicate that there is a continued environmental, societal, and economic need for initiatives to help Canadians adapt to climate change and the risks and opportunities it creates.
• The Adaptation Theme is aligned with federal roles, responsibilities and priorities related to protecting Canadians, promoting economic growth and development and addressing the sustainability of natural resources, as well as priorities related to the North, including those reported in Canada’s Northern Strategy.

Performance

• A variety of program delivery mechanisms are used by the various program elements to work towards achieving intended outcomes. A majority of key informants indicated that the design of program elements is appropriate for meeting the intended outcomes.
• Knowledge exchange with stakeholders is identified as a critical aspect of the Adaptation Theme’s program design. While some stakeholder engagement is occurring, more could be done to encourage the participation of existing and potential stakeholders, to increase effectiveness.
• The products and activities of the program elements are generally perceived to be of high quality and useful to external stakeholders; however, some reported they had difficulty finding products and that activities were not well advertised.
• A majority of program key informants reported that governance at the program element level is clear and effective. At the Adaptation Theme level, three committees collectively manage and coordinate the delivery of the Theme activities federally and across jurisdictions. Despite this, there was a strong desire for more collaboration and information sharing among program elements and among federal and non-federal program partners.
• Findings suggest that program resources are being used efficiently and economically. The use of information technologies to communicate, collaborate, for outreach and to promote activities and share information is one example of the cost-saving practices used.
• Funding for Adaptation Theme activities includes both new and existing funding. Coordinated horizontal financial reporting occurs for the Adaptation Theme; however, the report only provides details for new funding. Some federal partners were unable to provide budget, expenditure figures for their existing funding or details to sufficiently account for variances between new or existing funding and actual expenditures. As a result, it is not possible to present a comprehensive picture of the total budget and expenditures for Adaptation Theme activities over the period from 2011-12 to 2014-15.
• A logic model and performance measurement framework (PMF) for the Adaptation Theme are in place, however the Theme-level PMF does not adequately capture all relevant intended outcomes for all program elements.
• Overall, program elements are making progress towards achieving the Adaptation Theme immediate outcomes.
  • Progress is being made towards recognition by targeted communities and sectors of the need for adaptation and assessing their risks and opportunities arising from climate change.
  • Adaptation measures are being identified. There is increased awareness of relevant adaptation measures among targeted communities and sectors and collaboration on climate change adaptation.
  • There is early evidence of progress towards achieving intermediate and final outcomes, for example: development of research facilities, planning around food security and implementing ground stability systems as part of adapting to climate change. However, the longer term nature of these outcomes means that available evidence is still somewhat limited. It is still too early to determine whether progress is being made towards reducing the vulnerability of individuals, communities, regions and economic sectors to the effects of climate change.

Recommendations

Recommendations 1, 2 and 3 are directed to senior management of each of the nine federal organizations that have been involved in delivering the Adaptation Theme programming. Recommendation 4 is directed to all departments, with the exception of INAC and TC. No areas of concern were found in reviewing the financial information provided by these two departments.

Recommendation 1:  Review current adaptation program delivery to identify opportunities to expand engagement among new and existing stakeholder groups and increase awareness of information, products or activities.

Recommendation 2:   Review mechanisms for collaboration and information sharing between the Adaptation program elements, as well as with other federal and non-federal program partners, in order to leverage best practices and resources and avoid duplication of effort.

Recommendation 3:  Review the current performance measurement framework and data to improve tracking and reporting on progress toward Adaptation-theme level intended results.

Recommendation 4:  In collaboration with the departments’ respective Chief Financial Officers, review current processes for opportunities to improve program tracking and reporting of financial information for Adaptation program elements.

Due to the number of participating partners and the diversity of program elements under the Adaptation Theme, federal partners will address the recommendations in the manner that best meets the needs of their relevant program.

About the evaluation

The evaluation covers the period from 2011-12 to 2014-15. Although data is collected and analyzed for all 10 program elements, this was an evaluation of the overall CAA Adaptation Theme, rather than an evaluation of individual program elements.  As such, the degree of detail provided at the individual program level is limited.

Data was collected using various means:

• 89 key informant interviews with people internal and external to the initiative
• an online survey of 148 external partners
• four case studies
• a review of documents and performance data

As well, the evaluation took into consideration the results of an independent evaluation of the Northern Transportation Adaptation Initiative conducted by Transport Canada in 2014-15.