Evaluation of the Sustainability Reporting and Indicators Program

Key Dates

Planning phase completed - February 2015
Report sent for management response - May 2016
Management response received - July 2016
Report tabled before the Departmental Evaluation Committee - July 14, 2016
Report approved by the Deputy Minister - September 16, 2016

List of Key Acronyms

AEB

Audit and Evaluation Branch

ADM

Assistant Deputy Minister

DG

Director General

DPR

Departmental Performance Report

DSDS

Departmental Sustainable Development Strategy

CEAA

Canadian Environmental Assessment Agency

CEPA

Canadian Environmental Protection Act

CESD

Commissioner of the Environment and Sustainable Development

CESI

Canadian Environmental Sustainability Indicators

ECCC

Environment and Climate Change Canada

ESB

Environmental Stewardship Branch

FSDA

Federal Sustainable Development Act

FSDS

Federal Sustainable Development Strategy

G&Cs

Grants and Contributions

GGO

Greening Government Operations

MSC

Meteorological Services Canada

OGD

Other Government Departments

PAA

Program Alignment Architecture

PMF

Performance Measurement Framework

PWGSC

Public Works and Government Services Canada

RBAEP

Risk-based Audit and Evaluation Plan

RPP

Report on Plans and Priorities

STB

Science and Technology Branch

SEA

Strategic Environmental Assessment

SDAC

Sustainable Development Advisory Council

SPB

Strategic Policy Branch

TBS

Treasury Board of Canada Secretariat

Acknowledgements

The Evaluation Project Team would like to thank those individuals who contributed to this project, particularly members of the Evaluation Committee as well as all interviewees and survey respondents who provided insights and comments crucial to this evaluation.

The Evaluation Project Team was led by Susan Wharton, under the direction of William Blois, and included Kevin Marple, Lindsey Derrington, Katheryne O'Connor and Science-Metrix Inc.

Version Control

File name: SuRI - Evaluation Report_Final-2016-09-01.trackchanges.docx

Date: September 1, 2016

Executive Summary

Context

A key area of focus of the Sustainability Reporting and Indicators Program involves working with other government departments and agencies to implement the Federal Sustainable Development Act (FSDA), which mandates ECCC to lead the development of, implementation and reporting on the Federal Sustainable Development Strategy (FSDS). The FSDS 2013-2016 brings together goals, targets and implementation strategies under four priority themes: addressing climate change and air quality; maintaining water quality and availability; protecting nature and Canadians; and shrinking the environmental footprint--beginning with government (frequently referred to as “Greening Government Operations”).

The program also helps other federal departments and agencies to develop their individual departmental sustainable development strategies (DSDSs) that outline how their program activities will support the FSDS. In addition to its ongoing interaction with departments when developing the FSDS and triennial reports on progress, the program works with other government departments, through the Canadian Environmental Sustainability Indicators (CESI) program, to provide data and information on the state of the environment and to measure the progress toward the goals and targets of the FSDS.

The program is delivered by the Sustainability Directorate in ECCC's Strategic Policy Branch. In addition, within ECCC, the Science and Technology Branch, Environmental Stewardship Branch^Footnote1 and the Meteorological Service of Canada contribute to the program by providing subject-matter expertise, managing data collection arrangements with partners and supporting the development of the indicators.

The evaluation was undertaken between January and October 2015 and examined the four-year time frame from fiscal year 2010-2011 to 2014-2015. ECCC expenditures for the program were approximately $6.2 million in 2014-2015, or less than 1% of departmental spending.^Footnote2 Of this amount, approximately $2 million is transferred to other branches to support science and monitoring associated with CESI indicators.

Findings and Conclusions

Relevance

There is a continued need to consider environmental sustainability in government decision-making and to provide parliamentarians and Canadians with information on the state of the environment. There is also a need for a transparent, whole-of-government view of what actions are being taken to address climate change and air quality, maintain water quality and availability, and protect nature and Canadians.

The program fulfills a requirement outlined in the FSDA for the development, implementation and reporting of an FSDS, including sustainable development goals and targets in Canada. It also addresses federal roles and responsibilities outlined in the Canadian Environmental Protection Act and the Department of the Environment Act related to information on the environment.

The program is aligned with government priorities related to ensuring a clean and healthy environment for Canadians and responsible development.

Performance - Effectiveness

The program's performance was assessed against four thematic outcomes: increased awareness and access; continuous improvements to environmental indicators and the FSDS; increased integration of environmental sustainability into federal planning and reporting processes; and support for increased integration of environmental sustainability into federal decision-making. The latter of these reflects the program's longer-term outcome.

The program has made progress on the three shorter-term outcome themes. Access to environmental indicators and the FSDS has increased through the development and ongoing improvement of the CESI and FSDS websites. Additionally, the program has recognized the importance of engagement and outreach campaigns and has undertaken initiatives to enhance awareness among the federal government, key stakeholders and the general public. While evidence points to moderate improvement in awareness, the need to continue to raise awareness, including helping programs and stakeholders understand the value and potential impact of the FSDS and CESI, was identified as critical for the program to add greater value and ultimately play a greater role in supporting decision-making.

Since 2010, internal research, as well as both internal and external quality assessment and revision processes, support ongoing improvements that have led to an increase in the overall quality and coverage of environmental indicators and the FSDS. Consistent with the process of continual improvement, additional opportunities for improvement continue to be identified.

The program has met its commitments under the FSDA to deliver an FSDS and to report on progress against the strategy. OGDs reported high levels of usage and were positive about the quality of ECCC's guidance to support the development and presentation of their DSDSs.Over the evaluation time frame, departmental environmental sustainability reporting in federal reports on plans and priorities (RPPs) and departmental performance reports (DPRs) has become more consistent, comparable and aligned with the FSDS.

With respect to the program's longer term outcome, although there is evidence related to support for increased integration of environmental sustainability considerations into decision-making for government operations (Theme IV of the 2013-2016 FSDS), there is limited evidence available regarding support for increased integration of Themes I-III into federal decision-making as a result of the program. At this time, the FSDS is primarily a way to provide an integrated, comprehensive picture of actions across the federal government that contribute to environmental sustainability, but there is limited evidence of how it is being used to support decision-making. The consideration of FSDS goals and targets as part of the Strategic Environmental Assessment (SEA) process is currently identified as one of the main mechanisms in which the program's work contributes to integrating environmental sustainability into federal decision-making. In 2013, the program was recognized by the CESD for its efforts to strengthen its practices in this area. Representatives from other government departments suggested that there may be opportunities to further leverage the role of the program among related program areas to improve horizontal information sharing and facilitate discussions. It was felt that in addition to contributing to further improvements in environmental indicators, this could also lead to important policy discussions and ultimately support increased integration of environmental sustainability into decision-making.

Performance - Efficiency and Economy

Governance for the program is clearly defined and understood. Likewise, the roles and responsibilities of those who contribute to the FSDS and CESI are generally well understood. However, a challenge was identified related to the effectiveness of current governance processes to secure senior management participation at critical points and obtain input of a strategic nature associated with implementing the FSDA.

The program is viewed by stakeholders as employing efficient operational and administrative processes, including making effective use of technology.

The program collects some performance measurement data related to both the FSDS and CESI. There are gaps in the availability of data for the period of this review for half of the performance measures identified in the FSDS performance measurement framework, although most of the measures with missing data are not due to be reported on until 2016. With respect to CESI, the current logic model is outdated, there is no performance measurement framework (although a few indicators are measured) and there are gaps related to measuring the program's contribution to information on the state of the environment, including the program's work to improve the quality of environmental indicators collected by other federal programs.

Recommendations

The following recommendations are based on the findings and conclusions of the evaluation. The recommendations are directed to the senior departmental official responsible for the program, the Assistant Deputy Minister (ADM) of the Strategic Policy Branch (SPB).

Recommendation 1: Conduct more outreach appropriate to the target audiences, to build greater awareness of the FSDS and CESI indicators.

Recommendation 2: Within the context of developing the FSDS, seek opportunities to facilitate horizontal discussions among environmental sustainability stakeholders that will encourage cross-learning and support policy development.

Recommendation 3: Revisit current tactics and communication approaches to ensure senior management engagement at critical stages.

Recommendation 4: Develop and implement a performance measurement strategy that enables the Sustainability Reporting and Indicators Program to tell its performance story.

The ADM, SPB, agrees with the recommendations and has developed a management response that appropriately addresses each of the recommendations. The full management response can be found in section 6 of the report.

1. Introduction

This report presents the results of the Evaluation of the Sustainability Reporting and Indicators Program, which was conducted by Environment Canada's (ECCC) Evaluation Division, Audit and Evaluation Branch. The evaluation was identified in the 2014 Departmental Risk-Based Audit and Evaluation Plan and conducted in order to respond to the requirements of the Financial Administration Act and the Treasury Board of Canada Policy on Evaluation to evaluate all ongoing grants and contributions programs and direct program spending at least once every five years.

2. Background

2.1 Program Profile

The Sustainability Reporting and Indicators Program supports policy related to sustainable development.^Footnote3 The program involves leading the development of, implementation and reporting on the Federal Sustainable Development Strategy (FSDS), helping other federal departments and agencies to develop their individual departmental sustainable development strategies (DSDSs) in support of the FSDS, and working with other government departments through the Canadian Environmental Sustainability Indicators initiative (CESI) to provide data and information to report to Canadians on the state of the environment and to measure the progress toward the goals and targets of the FSDS.

Federal Sustainable Development Strategy (FSDS)

The Federal Sustainable Development Act (FSDA), which was approved by Parliament in June 2008, has as its purpose “to provide the legal framework for developing and implementing a FSDS that will make environmental decision-making more transparent and accountable to Parliament.” It gives the Minister of the Environment the legislated mandate to lead the development and implementation of the FSDS.^Footnote4 The Act requires that the Minister of the Environment table in Parliament a FSDS encompassing the activities of some 26 federal departments and agencies, engage in mandated consultations, and report on progress. In the 2013-2016 FSDS, seven departments and agencies voluntarily participated in the Strategy, bringing the total number to 33.

The FSDS brings together goals, targets and implementation strategies established through the regular course of government decision-making and consultations. In the 2010-13 and 2013-16 FSDS, goals, targets and implementation strategies are organized under four priority themes:^Footnote5

1. Addressing climate change and air quality;
2. Maintaining water quality and availability;
3. Protecting nature and Canadians; and
4. Shrinking the environmental footprint - beginning with government.

The Sustainable Development Office works with OGDs to develop a sustainable development planning and reporting framework that includes goals, targets and government actions. The resulting strategy then establishes the framework for departments to report on and be transparent and accountable for the integration of environmental and sustainability considerations. The FSDS also supports integrated decision-making --a key principle of sustainable development -- by making stronger links to economic and social policy.^Footnote6 One way in which this is done is that, since the establishment of the first FSDS in 2010, SEAS conducted by federal departments and agencies have been required to consider how proposals could affect the achievement of the FSDS goals and targets.^Footnote7

The Guidelines for the Implementation of the Cabinet Directive on the Environmental Assessment of Policy, Plan and Program Proposals, updated in October 2010, includes public and parliamentary reporting requirements and consideration of the impacts of new policy, plan or program proposals on the goals and targets of the FSDS.

In accordance with the Act, federal departments and agencies are to table individual strategies (DSDSs) to reflect how their program activities support the FSDS. The Sustainable Development Office (SDO) within ECCC's Strategic Policy Branch is responsible for developing and maintaining systems and procedures to monitor progress on the implementation of the FSDS. Specifically, at least once every three years it must provide the Minister with a report on the federal government's progress in implementing the FSDS. The FSDS Management Framework 2013-2016 outlines the key systems and procedures required to implement the Act and guidelines for managing and monitoring the FSDS.^Footnote8

Canadian Environmental Sustainability Indicators (CESI)

The CESI initiative was initially launched in 2004 to produce environmental baseline data and report environmental indicators for three key issues: clean air, clean water and greenhouse gas emissions. With the introduction of the FSDS in 2010, the CESI initiative was identified by the government as the key vehicle to measure progress toward the goals and targets of the FSDS. Its set of indicators was expanded in certain areas such as air quality and climate change, water quality and availability, and protecting nature. In addition, through CESI, ECCC partially fulfills its requirements to monitor environmental quality under the Canadian Environmental Protection Act, 1999 (CEPA) and to provide Canadians with environmental information under the Department of the Environment Act. Overall, the CESI initiative now provides data and information to track Canada's performance on key environmental issues of concerns to Canadians.

The indicators are developed and reported on by ECCC with the support of other federal departments such as Health Canada, Statistics Canada, Natural Resources Canada, Agriculture and Agri-Food Canada, and Fisheries and Oceans Canada, as well as provincial and territorial government departments.

ECCC regularly updates the CESI (four to six times per year) as new data become available for release.^Footnote9 Additional indicators are developed, subject to data availability, and existing ones may be adjusted or further refined to cover any revisions to the CESI's scope or to address gaps. The CESI initiative is an authoritative source of evidence to demonstrate progress on the government's environmental agenda and the indicators are regularly incorporated into communication materials such as speeches and media lines. The indicators are a foundational piece of the performance stories found in ECCC's RPP and DPR. The information is also used by researchers, students and others and can be found in numerous reports and analyses.

From a public perspective, the indicators are designed to be a source of information that is readily accessible to the general population and presented through the use of graphics, maps and short explanatory text. Indicators are national in scope, with information and data available at the provincial, regional and local levels for some indicators, through such tools as interactive maps. To promote transparency, the CESI initiative also publishes comprehensive explanations of source data and indicator methodologies and makes indicator data available for download, on both the Environmental Indicators website^Footnote10and the Open Data Portal of the Government of Canada.^Footnote11

2.2 Governance and Management

The Sustainability Reporting and Indicators Program is managed by the Director General of the Sustainability Directorate, within ECCC's Strategic Policy Branch.

There are three divisions within the Sustainability Directorate:

1. Information and Indicators Division: This Division's key mandate is to deliver the CESI initiative. It is responsible for development, quality management and public dissemination of the indicators and related data and information, including ongoing improvements and updates to the CESI website.
2. Sustainable Development Policy Division: This Division's activities include leading the development of the triennial FSDS, maintaining the FSDS Management Framework and its related processes (i.e. facilitating interdepartmental DG and ADM committees), coordinating ongoing engagement and the legislated public consultations, and monitoring progress toward medium and long-term sustainable development targets.
3. Sustainability Reporting and Accountability Secretariat: Key responsibilities for this team include leading the development of triennial reports on the progress of the FSDS, contributing to the Treasury Board of Canada Secretariat guidance for annual departmental reporting on DSDSs, and helping ECCC to respect the requirements of the Cabinet directive concerning SEAs of policy, planning and program proposals.

In addition to the work carried out by the Sustainability Directorate, the following organizations within ECCC also contribute or have contributed to the program:

• Science and Technology Branch (STB), Environment Stewardship Branch (ESB), and Meteorological Services Canada (MSC): Provide subject-matter expertise, manage related data collection arrangements with partners, including provinces and territories, and support the reporting process for the indicators related to their respective branches.
• Corporate Services Branch: Responsible for developing CESI web tools. This development work was completed in 2013-2014.

Governance mechanisms relevant to the program include the:

• Interdepartmental Assistant Deputy Minister Committee: Co-chaired by the ADMs of the Strategic Policy Branch at ECCC and the Policy, Planning and Communications Branch at PWGSC, this committee provides strategic direction, advances thinking, and makes decisions on key issues associated with implementing the FSDA.
• Interdepartmental Director General Committee:Co-chaired by the DGs of the Sustainability Directorate at ECCC and the Office of Greening Government Operations at PWGSC, this committee provides operational direction and guidance.
• Interdepartmental CESI DG Steering Committee:Chaired by the DG Sustainability Directorate at Environment Canada, this is the key governance body that collectively makes decisions related to CESI strategic and operational planning and provides direction on approaches and priorities pertaining to CESI. It is composed of directors general whose organizations are responsible for data and information that comprise the CESI indicators.
• Additionally, as required, other working groups may be established to carry out certain tasks and facilitate inter-departmental engagement.

Stakeholders

Key stakeholders for the program include:

• Other Government Departments/Agencies:Departments and agencies listed in Schedule I of the Financial Administration Act (20 in total) and any others listed in the Schedule of the FSDA (an additional 6 agencies) must prepare a sustainable development strategy containing objectives and plans for their individual department or agency which complies with and contributes to the FSDS.^Footnote12 Departments and agencies that do not fall under the FSDA may volunteer to participate in the FSDS. An additional seven departments and agencies participate in the 2013-2016 FSDS voluntarily, bringing the total number of participating departments and agencies to 33.
• Public Works and Government Services Canada (PWGSC): In the context of the FSDS, the Office of Greening Government Operations (OGGO) at PWGSC leads the development of greening government operations content for each FSDS and associated progress report, including developing GGO targets, implementation strategies and performance measures, providing guidance to departments, and monitoring and compiling GGO results for the progress report.
• Treasury Board of Canada Secretariat (TBS):TBS is responsible for the Government of Canada's annual reporting processes and integrates input on FSDS-related reporting into its guidance for a supplementary information table supporting the Reports on Plans and Priorities (RPPs) and Departmental Performance Reports (DPRs).
• Canadian Environmental Assessment Agency (CEAA): CEAA supports the Minister of the Environment by promoting the application of SEA to federal government policy, planning and program proposals by providing departments and agencies with SEA guidance and training.
• Commissioner of the Environment and Sustainable Development (CESD): The CESD is legally mandated to review the draft FSDS and comment as to whether the targets and implementation strategies can be assessed. The CESD is also required, under the Auditor General Act to review and comment on all FSDS progress reports, focusing particularly on the fairness of performance information. The CESD conducts audits, looking at various aspects of the DSDSs.

Other program stakeholders include the Sustainable Development Advisory Council (SDAC) which is a stakeholder council established and chaired by the Minister of the Environment and including representatives from the provinces, indigenous peoples, environmental non-governmental organizations, business and labour.

2.3 Resource Allocation

ECCC's total program expenditures for the years 2010-2011 through 2014-2015, by branch, are presented in Table 1 below.

The program expenditures identified for the Strategic Policy Branch represent the core program delivery expenditures. As described above, expenditures mapped to the Corporate Services Branch (CSB) represent costs for developing CESI web tools (development was completed in 2013-2014) and expenditures for MSC, STB and ESB represent expenditures related to such areas as the provision of subject-matter expertise, management of data collection arrangements with partners including provinces and territories, and reporting in support of their respective indicators. The increase in program expenditures from 2010-2011 to 2011-2012 primarily reflects CESI funding renewal and new indicator development in support of expanded reporting for the FSDS.

Table 1: ECCC Expenditures for Sub-Program 1.3.1 from 2010-2011 to 2014-2015

Source: Extracted from ECCC's financial system and confirmed with program representatives.

Table 2 provides detail on the expenditures of G&C projects funded by the program during the study period sorted by branch. SPB funds were primarily used to support the Carbon Disclosure Project aimed at building capacity to collect corporate climate change information. For STB, G&C funding was used to enhance the Province of Saskatchewan's capacity to contribute to CESI water quality indicators. ESB funding was primarily directed at issues related to the reporting and tracking of protected areas. The program's G&C projects are administered under the authority of two of ECCC's umbrella contribution terms and conditions: Contributions to Support Sustainable Ecosystems and Contributions in Support of Water Resources.

Table 2: G&C Expenditures for Sub-Program 1.3.1 from 2010-2011 to 2014-2015

Source: Extracted from ECCC's financial system and confirmed with program representatives.

2.4 Intended Outcomes

There is no single logic model that encompasses all activities and intended outcomes for the program. Rather, the program has separate logic models for its two key components, namely the:

• CESI Renewal Logic Model (2009);^Footnote13 and
• Draft FSDS Logic Model (FSDS Management Framework, 2013-2016).

For the purposes of this evaluation, program performance was assessed against four thematic outcomes presented below, which incorporate the concepts included in the program's two component logic models and the PMF 2014-2015. The fourth thematic outcome represents the program's longer term outcome, and is intended to address the concept of use. Consistent with the purpose of the FSDA, the focus of this outcome is on integration of environmental sustainability in federal decision-making, however, any evidence of the program's contribution to decision-making outside the federal context was also considered.

1. Increased awareness of and access to environmental indicators and the FSDS
2. Ongoing improvements to environmental indicators and the FSDS^Footnote14
3. Increased integration of environmental sustainability into federal planning and reporting processes^Footnote15
4. Support for increased integration of environmental sustainability into federal decision-making

The thematic approach was used to avoid repetition for related concepts. The thematic outcome statements were developed by the evaluation team in collaboration with program representatives and endorsed by senior management during the evaluation's planning phase. A mapping of the outcomes from the two component logic models and the PMF expected results to the four thematic outcome statements can be found in Annex B.

3. Evaluation Design

3.1 Scope

The purpose of this evaluation is to assess the relevance and performance (including effectiveness, efficiency and economy) of the Sustainability Reporting and Indicators Program over the four-year timeframe from 2010-2011 to 2014-2015. The focus of the evaluation was on the activities of the Sustainability Directorate related to:

• the delivery of the CESI initiative;
• supporting the implementation of the FSDA (including leading development of the triennial strategy, maintaining the Management Framework, coordinating engagement and consultations, providing guidance and monitoring Greening of Government Operations (GGO); and
• supporting reporting and accountability commitments regarding the FSDS and ECCC's strategic environmental assessments (SEAs).

The assessment of GGO is limited to the role played by the Sustainability Directorate (i.e., providing guidance and monitoring) and does not extend to the implementation of GGO activities by the Department, as this is primarily the responsibility of Corporate Services Branch and is outside the scope of the program.

The involvement of other ECCC branches in the evaluation was limited to their role as program partners which provide input for CESI indicators (MSC, STB, ESB), or as enablers, as is the case for CSB which has provided support for the development of web-enabled tools.

The evaluation was conducted to respond to the requirements of the Financial Administration Act and the Treasury Board of Canada Policy on Evaluation to evaluate all ongoing grants and contributions programs and direct program spending at least once every five years.

3.2 Evaluation Approach and Methodology

The following data collection methodologies were used to address the evaluation issues and questions. For each line of evidence, instruments were developed to capture and organize data by evaluation issue. These data were then analyzed to develop overall findings, conclusions and recommendations.^Footnote16

Document Review: A review was conducted of descriptive program information, departmental and Government of Canada publications, performance and financial information, and other internal planning and operational documents. A review of past evaluations and audits was also conducted to complement the analysis.

Literature Review: A review of existing research and literature related to sustainable development strategies was conducted to identify best practices in the development of sustainable development strategies and the development of objectives, indicators and targets.

Key Informant Interviews: A total of 24 key informant interviews were conducted with program staff, federal partners and external stakeholders in order to obtain a cross-section of views. The distribution of interviews by respondent category is shown below:

• ECCC/SDO program representatives and management (n=5)
• Federal partners involved in developing guidance for federal departmental DSDSs (TBS, PWGSC, CEAA) (n=3)
• Key FSDS OGD partners (among select departments/agencies associated with FSDS targets) (n=5)
• ECCC and OGD stakeholders responsible for providing data on specific CESI indicators (n=7)
• External stakeholders (recipients of G&C funding, provincial, academic, NGO stakeholders that use CESI/FSDS data) (n=4)

On-line Survey of OGD Stakeholders: An on-line survey questionnaire was sent to 177 individuals from 30 federal departments and agencies involved in the program (excluding ECCC). A total of 57 respondents from 22 departments and agencies completed the survey, for a 33% response rate.

The survey questionnaire was designed to gather only respondents' perspectives about their particular FSDS- or CESI-related involvement with ECCC, thus ensuring that each respondent was asked only those questions that were relevant to their particular experiences with the program. The distribution of respondents by type of involvement is presented below (respondents could select more than one answer):

File Review: A file review of the seven G&C projects funded by the program between 2011-2012 and 2013-2014 was conducted. Project documentation was gathered from program representatives and included contribution agreements, project progress reports, financial information and final project reports, as available.

3.3 Challenges and Limitations

Challenges experienced during the evaluation, as well as the related limitations and strategies used to mitigate their impact, are outlined below.

• Obtaining input and synthesizing results from the large group of OGD stakeholders with varied and different levels of involvement with the program in an efficient manner posed a challenge for the evaluation. In order to address this challenge, the evaluation team worked closely with program representatives to clearly identify the different sub-groups of OGD stakeholders and to align data collection tools to the various types and levels of involvement. The online survey was the key method used to gather input from OGD stakeholders. The survey design included separate modules to address the varied OGD stakeholder involvement. To the extent possible and where relevant, findings are presented in a manner that clarifies the sub-population.
• Although the response rate for the survey was consistent with expected response rates for online surveys (33%), with just 57 respondents, the sample size was small, particularly when looking at sub-population of respondents. The findings from this line of evidence, therefore, should be viewed as illustrative and interpreted with caution. To compensate for this, survey evidence was also triangulated with evidence from other methodologies.
• As previously described, the program has separate logic models addressing the two key components of the program (CESI, FSDS) as opposed to one integrated logic model. In total, the two logic models include ten direct outcomes, five intermediate outcomes and two long-term outcomes. In order to simplify, while still maintaining the key concepts included in the program's logic models, the evaluation team worked in collaboration with program representatives to map the outcomes from the component logic models to four broad thematic outcomes which were used to assess performance for the purpose of the evaluation.

4. Findings

This section presents the evaluation findings by evaluation issue (relevance and performance) and by the related evaluation questions.

For each evaluation question, a rating is provided based on a judgment of the evaluation findings. The rating statements and their significance are outlined below in Table 3. A summary of ratings for the evaluation questions is presented in Annex A.

4.1 Relevance

• Public consultations for the draft FSDS 2013-2016 show that stakeholders believe it is important for the federal government to consider sustainable development when making policy and program decisions and to communicate to the public how the government has integrated sustainable development into decision-making in order to make environmental decision-making more transparent and accountable.^Footnote17
• CESI is based on the premise that Canadians need clearly defined indicators --measuring sticks that describe the health of their environment and track results achieved to protect and improve the environment.^Footnote18
• Sustaining natural resources, protecting the health of people and ecosystems and improving quality of life are issues of importance to Canadians. Stakeholders are also of the view that federal environmental decision-making should be more transparent and accountable.^Footnote19
• The majority of interviewees believe there is a continuing need for the work of the program, highlighting the need for:
  • credible environmental information to understand areas of harm or risk and inform decision-making related to the management of natural resources;
  • standardized data that is comparable across the country; and
  • the federal government to play a leading role in the collection of environmental information and to make data readable and accessible to Canadians.

• Through presenting a whole-of-government view of environmental priorities, tracking progress against these priorities, and incorporating the FSDS into the government's core planning and reporting processes, the program helps bring coherence to Canada's policies on sustainable development and broadly supports federal commitments outlined in the 2013 Speech from the Throne, such as responsible resource development, protecting against threats to the environment and local communities, and providing Canadian families with a clean and healthy environment.
• Additionally, ensuring that strategic environmental assessments (SEAs) of new policy, planning and program proposals consider any potential impacts on FSDS goals and targets is consistent with federal priorities outlined in the Budget 2015 announcement such as a commitment to responsible development of natural resources, to “only proceed with projects that are safe for Canadians and the environment” and to ensure “a secure and prosperous Canada for generations to come”.
• Program managers and OGD stakeholders also identified the program's alignment with federal priorities related to cross-cutting environmental issues such as climate change adaptation and mitigation, water quality and environmental conservation. The FSDS includes strategies related to these issues and the FSDS progress reports and CESI provide performance information that is valuable for monitoring progress and supporting program management related to these issues.
• Given the cross-cutting nature of this program, it aligns with all three ECCC departmental strategic outcomes:
  • “Canada's natural environment is conserved and restored for present and future generations”
  • “Canadians are equipped to make informed decisions on changing weather, water and climate conditions”
  • “Threats to Canadians and their environment from pollution are minimized.”

• The program directly fulfills requirements under the following federal legislation:
  • The Federal Sustainable Development Act(FSDA)which requires “the development and implementation of a Federal Sustainable Development Strategy and the development of goals and targets with respect to sustainable development in Canada.”^Footnote20
  • The Canadian Environmental Protection Act(CEPA),section 44(1), which states that the Minister must establish a system for monitoring environmental quality, conduct research on the effects of pollution and human activity on environmental quality, and report on and publish information related to the state of the Canadian environment.
  • The Department of the Environment Act, section (5) which states that the Minister must establish programs to:
    • promote the establishment or adoption of objectives or standards relating to environmental quality, or to control pollution;
    • provide to Canadians environmental information in the public interest; and
    • promote and encourage the institution of practices and conduct leading to the better preservation and enhancement of environmental quality.

4.2 Performance - Effectiveness

As previously noted, program performance was assessed against four broad thematic outcomes which incorporate the concepts included in the program's two component logic models and the program expected results identified in the PMF 2014-2015. The fourth thematic outcome represents the program's longer term outcome. Findings and conclusions related to each of the four themes are presented below.

• Interviewees across all categories were of the opinion that with the development of communication mechanisms such as the CESI and the FSDS websites, which present the various FSDS-related products (e.g., Strategy, Progress Reports, DSDSs, DPR supplementary information tables) in one location, the program has been successful in increasing access to information regarding environmental performance and federal actions in the area of environmental sustainability. OGDs and external interviewees noted that the program has made good progress in terms of the quality of the CESI website, including the presentation of data and development of tools to access data. While many OGD survey respondents were unable to respond, a large majority of those who had an opinion (n=23) felt that user access to CESI indicators and information products and services had improved since 2010 to a moderate (52%), large (22%) or very large (9%) extent.
• In an effort to enhance awareness, the SDO has developed separate stakeholder engagement strategies for both CESI and the FSDS targeted within the federal government and with external target audiences (e.g., provincial and municipal governments, academia, ENGOs, media organizations and the general public). Highlights of these initiatives include:
  • In 2009, the “Engagement Strategy for the Canadian Environmental Sustainability Indicators” was developed with the purpose of identifying target audiences and proposing tactics to reach each group.^Footnote21
  • In 2014, the SDO established an Outreach and Engagement Strategy for the FSDS 2013-2016.^Footnote22Three audience-focused campaigns were undertaken in 2014-2015:
    • A whole-of-government campaign was conducted to increase awareness and use of the FSDS across the federal government. The campaign involved: launching an FSDS GCconnex group to promote discussion on sustainability; presentations at 12 senior management meetings across 7 departments to raise visibility of the FSDS; and publishing a News@ECCC article about the FSDS, as well as four “Did you know” messages on the ECCC intranet homepage.
    • A stakeholder engagement campaign, focusing on external stakeholders^Footnote23 was carried out between August and December 2014. Through this campaign, the SDO conducted 30 meetings and attended, or presented at, several relevant conferences.^Footnote24 These activities were intended to enhance relationships with FSDS stakeholders, improve the next FSDS and Progress Report, seek input on the use of CESI in reporting on FSDS progress and lay the groundwork for the legislated FSDS stakeholder consultation period.
    • A public social media campaign was designed to increase general public awareness. This included the development of: a communications strategy; a draft social media handbook for OGDs; communications via Twitter and Facebook; and 3 video scripts. Implementation of this campaign has been delayed by communications approval processes.
  • Despite these efforts, federal interviewees had mixed views in terms of progress made to increase awareness. While most were of the view that awareness of CESI and the FSDS is on the rise within the federal government as well as among some key external stakeholder groups, the majority noted that this is an area that would benefit from additional efforts. In particular, the view was expressed that more could be done to enhance or customize the manner in which the program conducts outreach to key target audiences. Additionally, it was noted that the program could find ways to promote awareness more broadly among Canadians (as described above, an engagement campaign targeting the general public was recently developed but implementation has been delayed).
  • Both federal and external interviewees referred to FSDS and CESI as having “untapped potential”, or being a “hidden jewel” that not enough people know about. The most common suggestions from interviewees and OGD survey respondents on how to increase the value of the CESI or the FSDS for federal and non-federal audiences related to increasing awareness. One OGD interviewee noted that within his department, work in this area has historically been viewed as another bureaucratic process, but the potential usefulness is only now starting to be recognized. Specific suggestions included:
    • more outreach at the program level (including interdepartmental) to increase awareness and help programs understand the value and potential impact of the FSDS/CESI for their work;
    • targeted outreach to engage with key stakeholder groups, e.g., river keepers, watershed groups, ENGOs, industry, and provincial government; and
    • increased promotion of the FSDS/CESI through the use of targeted channels, e.g., the Health Canada Science Forum, university seminars or advertising.
• It should be noted that, according to program management, additional resources would be required to support any significant expansion of the program's outreach.
• As shown in Table 4, available results from the FSDS Management Framework and CESI PMF indicators show an increase in 2013 for four measures related to use and access, followed by a decline in 2014 (although 2014 results remained higher than the baseline for all but one measure). The 2013 increases in citations can, in part, be attributed to the release of the draft 2013-2016 FSDS in February and the final strategy in November.^Footnote25

• This outcome theme is based in the premise that aspects such as increased relevance and coverage of indicators, and recognition of credible information related to the environment and the quality of the FSDS are necessary to support longer-term outcomes related to increased integration into planning and decision-making.
• A number of processes were identified that support progress in this area. Both the FSDS and the environmental indicators used to measure progress toward sustainable development are subject to CESD review and consultations with stakeholders and the public as a result of legislative requirements under the FSDA. The second cycle of the FSDS (2013-2016) sought to build on the first cycle by reporting on a broader range of government actions that contribute to FSDS targets, enhancing the link to core federal planning and reporting through ongoing alignment between the FSDS and federal department's PAAs, and expanding the CESI suite to better measure progress on FSDS targets.^Footnote27 In reviewing the 2013-2016 FSDS, the CESD noted several areas for possible improvement, most of which were addressed by ECCC in the final version of the Strategy.^Footnote28
• There are also ongoing processes in place, including annual reviews, whereby CESI staff work with individual subject-matter experts within ECCC and OGD program areas to improve the CESI indicators, including identifying ways to improve the quality and presentation of information. Further, the file review demonstrated that the goals of all of the program's funded G&C projects are related to improving the quality of indicators or sustainability reporting.
• Relevant improvements identified in recent ECCC DPRs include:
  • the introduction of two new targets for climate change adaptation and agriculture in 2013-2014.
  • the addition of 3 new indicators and updates to 18 indicators in 2013-2014.
  • the establishment of a “cross-cutting” thematic database of FSDS activities in 2014-2015 to better tailor and group FSDS-related programs and activities; and
  • the addition of 4 new indicators and implementation strategies as well as updates to 25 indicators in 2014-2015.
• Virtually all of the FSDS's 34 targets have defined indicators.^Footnote29Additionally, 93% of 2013-2016 FSDS targets are deemed “SMART”, based on a definition that 3 out of 5 of the SMART criteria are met for the target. Results show that large majorities of the targets were rated as relevant (100%) and specific (89%), while slightly smaller proportions were deemed to be measurable (82%) or time-bound (79%). Targets were least likely to be rated as achievable (71%), although a majority nonetheless met this criterion.^Footnote30
• Findings from a recent AEB internal audit of External Reporting on Performance^Footnote31 were favourable with respect to the quality of reporting on CESI, noting that:
  • CESI's indicator development and update processes are properly documented,
  • review and approvals were properly carried out, and
  • controls were operating effectively to ensure that the external reporting of CESI met required quality standards.
• Interviewees from the program, as well as from other areas of ECCC and OGDs that contributed to the FSDS and CESI were generally of the opinion that with each cycle there has been improvement in the quality of the indicators and information provided. A few interviewees commended ECCC program staff for their work in encouraging departments to think about “SMARTer” targets and to identify indicators that tell a story about environmental sustainability.
• Similarly, OGD survey respondents who were able to comment generally noted that the quality and coverage of performance information in the FSDS and CESI had improved to a moderate extent. Ratings were lower for the degree to which CESI had contributed to a more favourable attitude toward the credibility of information related to the environment.
• Identified areas for continued improvement included the need to address gaps in data for national coverage, addressing the challenge of presenting data in a manner that meets the needs of both scientific users and the general public, and identifying indicators that provide a broader perspective of the environment, (e.g., summarizing the overall state of an ecosystem instead of only one specific issue).

• Progress toward this outcome considered the program's contribution to the planning and reporting requirements outlined in the FSDA. In addition to requirements associated with the FSDS and progress reports, this includes the program's role supporting departments and agencies with the development of their DSDSs and presentation of related performance information in departmental RPPs and DPRs. The program's role in the provision of guidance and the level of adherence to this guidance by departments and agencies were also important considerations when assessing this outcome, since they contribute to the quality and consistency of reporting.
• The program has met its commitments under the FSDA to deliver an FSDS every three years which provides a whole-of-government view of federal actions to contribute to environmental sustainability and to deliver progress reports which report on government-wide results related to the FSDS goals and targets. Further, all 26 departments bound by the FSDA to contribute to the achievement of the FSDS goals, as well as 6 of 7 departments participating voluntarily, have developed DSDSs, which outlines how their department's program activities will support relevant goals and targets from the FSDS. Plans to address these targets and strategies are incorporated in their respective RPPs and progress is reported on in their DPRs.^Footnote32 In addition, four new departments/agencies voluntarily developed a DSDS during the study time frame.
• ECCC's SDO provides guidance to departments on the development of their respective DSDSs, and collaborates with TBS and PWGSC to develop and distribute guidance material to departments and agencies to enhance the consistency and quality of reporting of sustainable development activities within RPPs and DPRs. In addition, the SDO collaborates with the CEAA to facilitate the inclusion of SEA information in departmental DSDSs. Guidelines for the development of RPPs and DPRs established by TBS in collaboration with ECCC prior to tabling the FSDS 2010-2013 included the requirement to:
  • Focus on outcomes and link departmental activities to the FSDS; and
  • Make clear linkages between departmental sustainable development plans, priorities and expected results and the appropriate levels of the PAA.^Footnote33
• The online survey of OGD respondents identified high usage levels of DSDS guidance material prepared by the SDO. Fully 85% of OGD survey respondents who coordinate or provide input into development of a DSDS reported that they refer to the guidance material to a moderate (27%), large (27%) or very large (30%) extent, when preparing their DSDS and reports for RPPs and DPRs.^Footnote34 For the most part, feedback on ECCC's involvement and provision of guidance was very positive and included comments from OGD survey respondents that the Sustainability Directorate team effectively engages departments and the level of support is good, including providing prompt and clear responses to questions. Suggestions were made, however, to provide earlier and more guidance, including sharing examples of best practices. It should be noted, however, that the timing of the release of guidance is dependent on TBS timelines.
• As part of the data collection for the evaluation, an analysis of all participating department's DSDSs was conducted to examine how the implementation of the FSDA has affected their overall content and structure. Although some interviewees expressed the view that information contained in the DPR and RPP is the same information reported by departments prior to the FSDA, the analysis revealed that after implementation of the first FSDS, DSDSs were generally more streamlined, better aligned with the PAA, and more consistent across departments. Further, there was greater detail on the implementation, progress, and planned changes to DSDSs.^Footnote35
• The analysis of DSDSs also identified that the integration of environmental sustainability into federal reporting was particularly high in 2012-13. In this year, departmental programming contributing to the FSDS was explicitly labelled using visual identifiers and paragraph descriptions within the body of the RPPs and DPRs. Furthermore, all departments included a section in the front-end summarizing their contribution to the FSDS.

• As previously noted, this outcome theme closely aligns with the program's long-term outcomes for both FSDS and CESI and addresses the progression from reporting on activities and trends in environmental sustainability to supporting the integration of environmental sustainability into decision-making.
• According to performance data collected by the OGGO, there has been success in incorporating the goals and targets outlined in Theme IV (Greening Government Operations) into federal decision-making.^Footnote36 For example:
  • 14 out of 15 custodial departments have developed a strategic green buildings framework;
  • 25 departments have plans in place to manage e-waste and have adopted a green meeting guide;
  • 16 of 19 departments which engaged in activities to establish a level of high environmental performance at the office completed these projects in 2012-2013; and
  • 6 new refit projects have achieved an industry recognized level of high environmental performance in 2012-2013.
• When assessing progress toward support for integration of environmental sustainability into decision-making from a broader perspective (i.e., for FSDS Themes I - III related to addressing climate change and air quality, maintaining water quality and availability and protecting nature and Canadians), the majority of interviewees indicated that the level of integration of environmental sustainability into federal decision-making remains low. 55% of OGD respondents to the survey also shared this view (24 of 44 respondents who provided a rating), saying that integration of environmental sustainability considerations has not increased or increased only to a small extent. Survey responses relevant to this outcome are presented below:

• Program documentation and interviewees (both program managers and OGD stakeholders) indicate that the consideration of environmental sustainability goals and targets in SEAs is currently identified as the main mechanism for supporting the integration of environmental sustainability into federal decision-making.
• As previously noted, new guidelines for implementing the Cabinet Directive on the Environmental Assessment of Policy, Plan and Program Proposals were published in conjunction with the tabling of the FSDS 2010-2013 with the goal of ensuring that any potential effects of a policy or proposal on the goals and targets of the FSDS are considered at an early stage of the decision-making process. Through ECCC's development of guidance for DSDS reporting, ECCC also supported the CEAA's delivery of its guidance for departments and agencies on annual reporting on SEA (e.g., in DPRs). Key changes were integrated into the Guidelines for the Cabinet Directive, notably the requirement to:
  • Consider a proposal's impact on FSDS goals and targets when undertaking SEAs;
  • Require the reporting on the results of SEAs in DPRs; and
  • Describe the impact (positive or negative) of the proposals on the achievement of the FSDS goals and targets in SEA public statements.
• Additionally, in 2011, ECCC updated its internal departmental SEA template to prompt analysts to consider the FSDS goals and targets at all stages of the SEA process. The CESD's 2013 Fall Report acknowledged the department's efforts in this area, indicating that ECCC showed satisfactory progress in strengthening its practices related to SEAs by updating internal guidance, reporting the results of SEAs and describing the impacts of these initiatives on federal environmental goals and targets.^Footnote37
• Some interviewees noted that, beyond the integration of FSDS goals and targets into the SEA process, the FSDS and CESI do not currently contribute to supporting the integration of environmental sustainability into decision-making. More explicitly, it was noted that, given that the FSDS presents federal plans within existing resources, it remains a way to communicate federal actions related to environmental sustainability, but has not been recognized as being a trigger that supports change. An exception provided was in relation to the Hydrometric Services Program where it was noted that engagement with the CESI coordinator and use of additional CESI data had made a direct contribution to establishing federal priorities for network planning.
• In response to a question asking for suggestions to increase the value and impact of the CESI/FSDS, the second most frequently made suggestion^Footnote38 by interviewees and survey respondents related to leveraging the centralized role of the program to improve information sharing horizontally and facilitate discussions among related program areas. It was suggested that this could result in identification of improvements to targets and indicators, and could also contribute to important policy discussions and ultimately support increased integration of environmental sustainability into decision-making within the federal government.
• As of yet, no baseline data or performance results exist, with the date for the performance indicators from the FSDS Management Framework to track progress on the outcomes related to this theme expected in 2016, at the end of the current FSDS cycle.

External Factors Affecting the Achievement of Outcomes

In 2013-14, as a result of a TBS decision designed to simplify and reduce the reporting burden on departments,^Footnote39 all departmental management and related policy issues, including the departmental commitments to the FSDS, were moved from the main body of the RPPs and DPRs to supplementary information tables.^Footnote40Several federal interviewees (program managers and OGD partners) were of the opinion that the new format has led to reduced integration and visual prominence of environmental sustainability within the federal reporting process.

Unintended Outcomes

Interviewees from one of the implicated program areas within ECCC reported that the centralized coordination role played by CESI had the positive unintended effect of identifying subject-matter experts from other areas of the department.

4.3 Performance -Efficiency and Economy

• Studies conducted by the SDO and academic literature identify several characteristics of effective sustainable development strategies that are considered to be “best practices” for countries, for integrated and pragmatic approaches to sustainability reporting.^Footnote41 Evidence from the document review and key informant interviews demonstrates that the program is designed according to these characteristics, which include the following:
  • Development of a long-term vision and its linkage to short-term action;
  • Support by a high-level government commitment and influential institutions;
  • The presence of horizontal and vertical coordination mechanisms;
  • Broad participation by societal stakeholders;
  • Establishment of targets with clear budgetary priorities;
  • Integration of economic, social and environmental objectives; and
  • Inclusion of monitoring, learning and improvement/adaptation mechanisms.
• Development of a revised FSDS every three years is another program design characteristic in line with best practices as identified in academic literature, namely:
  • The development of an effective strategy requires an iterative process where objectives are set and progress toward their achievement is monitored as a guide to the next round of the learning process.^Footnote42

• As described in Section 2.2, there are a number of governance committees with membership from key federal stakeholders which support the work of the program. The membership, mandates and meeting requirements of these committees are clearly documented, either in the FSDS Management Framework (for FSDS committees) or in committee terms of reference (for CESI committees).
• Additionally, the FSDA outlines the responsibilities of the SDO, the CESD and those departments that are required to participate in the FSDS, including timelines for the development and progress reporting of both the FSDS and departmental DSDSs.
• Further, within ECCC, MOUs are established with the programs responsible for CESI development or reporting. These MOUs distinctly outline the roles, responsibilities and technical specifications for indicator development and reporting.
• Both internal and external interviewees and OGD stakeholder survey respondents were generally of the view that the governance process and accountabilities were clear and well understood. The online survey results showed that 81% of OGD members and participants in the FSDS DG and ADM committees (n=26) somewhat or strongly agree that the mandate and objectives of the committees are clear.
• Some issues were identified, however, related to the effectiveness of current governance processes to secure senior management participation at critical points and obtain input of a strategic nature associated with implementing the FSDA. OGD interviewees and survey respondents noted that working level staff are frequently delegated to attend the DG and ADM interdepartmental meetings in the place of senior representatives. Some respondents perceived this as a lack of senior management engagement, while others said the length and operational focus of the meetings was a hurdle to senior management attendance. The suggestion was made to reconsider the distribution of agenda items between working level and senior management meetings by streamlining senior management meetings to focus on strategic items or decision points and leaving the operational items to working level meetings.
  • This issue may explain the modest numbers (61.5%) of survey respondents who were members or participants in FSDS DG and ADM committees (n=26) who agree that the current structure is effective, as well as feedback from some respondents indicating that there were too many meetings (possibly because the same delegate ends up attending multiple meetings with similar agendas intended for different participants).

• OGD stakeholders who responded to the online survey were generally positive in their assessment of the efficiency of the program's operational and administrative processes.
  • Three-quarters (73%) of FSDS DG and ADM Committee participants (n=26) agree or strongly agree that administrative tasks such as organizing meetings, maintaining records of decisions and sharing materials are conducted in an efficient manner by ECCC.
  • Nearly two-thirds (64%) of those involved in providing data for CESI development or reporting (n=14) agree or strongly agree that processes for indicator development and publication are managed in an efficient manner by ECCC.
  • A majority agree or strongly agree that requests for input and feedback on key FSDS products (74%, n=35) and for the provision of data for CESI (80%, n=15) are clearly communicated; and
  • A slightly smaller majority agreed or strongly agreed that requests allowed an appropriate time to respond for the development of key FSDS products (57%, n=35) and for provision of data for CESI development and reporting (67%, n=15).
• The majority of OGD interviewees and survey respondents were also very positive about the program's use of technological tools to support efficient development of products.
  • Over two-thirds (69%) of survey respondents who coordinate or provide input into key FSDS products (n=35) agree or strongly agree that ECCC's SDO makes good use of tools and technologies to enhance efficiency.
  • Many respondents note that the use of SharePoint allows for better management of documents and results in a more streamlined approach to the development process.
  • The use of GCconnex and GCpedia for sharing information and meeting materials is perceived by the vast majority of interviewees to be efficient, as it keeps e-mail inbox sizes down and is a good platform for document sharing.
• According to program representatives and OGD stakeholders, the program appears to be appropriately resourced to deliver on its current mandate, including providing the SDO support as required by the FSDA. As shown in Table 1 in Section 2.3, resource levels for SPB's Sustainability Directorate have increased by approximately 30% since 2010-2011 (primarily in salary), when the program expanded to reflect CESI funding renewal and new indicator development in support of expanded reporting for the FSDS.
• Funding for ECCC's program partners in MSC, STB and ESB, who are responsible for providing subject-matter expertise and data collection and reporting on CESI indicators, increased significantly in 2011-2012 to reflect the expanded indicator development and reporting requirements for the FSDS and have remained relatively flat since then. Combined expenditures for the three partner branches increased from approximately $200,000 in 2010-2011 to approximately $2 million per year for the period from 2011-2012 to 2014-2015 (see Table 1 for detailed information).
• G&C funding represents a small portion of the program's overall spending and is primarily used to improve the quality of environmental indicators. In 2014-2015, G&C expenditures were $110,000, less than 2% of the program's expenditures (see Table 1). Results of the file review of G&C projects funded between 2011- 2012 and 2013-2014 indicate that $2.15 was leveraged from other sources for every ECCC dollar.^Footnote43

• The program collects performance measurement data related to both the FSDS and CESI. As currently structured, however, the program does not have the performance measurement tools in place that enable it to tell its performance story.
• The FSDS Management Framework 2013-2016 is a comprehensive document developed by the program to support the FSDA and guide the implementation of the FSDS.^Footnote44 It includes a detailed performance measurement framework, aligned with a logic model, that clearly outlines one or more indicators for each outcome, methodological details for each indicator (e.g., data source, definitions, and reporting frequency), as well as some information related to targets and actual values. While it includes relevant measures to assess the program's work related to the FSDS, its scope is broader than the program's contribution to the FSDS. For example, it includes performance measures related to OGD's greening government operations and their use of guidance related to SEAs provided by CEAA.
  • As of the time of the evaluation, there are gaps in the availability of performance information within the FSDS performance measurement framework, with data available for only 10 of the 20 indicators that were relevant to the Sustainability Reporting and Indicators Program's performance.^Footnote45 As the Management Framework addresses the 2013-2016 FSDS, it is the program's intent to populate the data by 2016. There may also be opportunities to simplify the performance management framework and reduce the number of indicators so that reporting is more manageable.
• With respect to CESI, the program has a separate CESI logic model (2009) and collects some data related to access and use of CESI data (e.g., number of CESI web visits). However, there is no supporting performance management framework that identifies indicators for each outcome or outlines plans for reporting against outcomes. Additionally, the logic model is outdated (e.g., refers to annual reports which are no longer published and uses some terminology which is no longer relevant given the introduction of the FSDS in 2010). Further, there are gaps related to the program's contribution to information on the state of the environment,^Footnote46 including the program's work to improve the quality of environmental indicators collected by other federal programs.

5. Conclusions

Relevance

The Sustainability Reporting and Indicator program is relevant. It fulfills a requirement under the FSDA and also addresses federal roles and responsibilities outlined in CEPA and the Department of the Environment Act.

The program contributes to addressing an ongoing need to consider environmental sustainability in government decision-making and provide information on the state of the environment, as well as a transparent, whole-of-government view of sustainable development actions being taken. It is aligned with government priorities related to ensuring a clean and healthy environment for Canadians and a sustainable economy.

Performance - Effectiveness

The program has made progress on its shorter-term intended outcomes. Access to environmental indicators and the FSDS has increased through the development and ongoing improvement of the CESI and FSDS websites. Additionally, outreach and engagement campaigns have been undertaken to enhance awareness among the federal government, key stakeholders and the general public. While evidence shows moderate improvement, continuing efforts to raise awareness and conduct outreach was identified as a key means to increase the program's value.

Internal research and quality assessment, as well as external revision processes, support a process of ongoing improvements that have led to an increase in the overall quality and coverage of environmental indicators and the FSDS since 2010.

Over the evaluation timeframe, departmental environmental sustainability reporting in RPPs and DPRs has become more consistent, comparable and aligned with the FSDS.

Although there is evidence that environmental sustainability considerations are increasingly integrated into decision-making for greening government operations (Theme IV of the FSDS), for Themes I-III (related to climate change and air, water and nature) there was limited evidence of the program's contribution to the integration of environmental sustainability into federal decision-making other than through the consideration of FSDS goals and targets as part of the SEA process. Opportunities were identified to leverage the program's leadership and coordination role to facilitate more horizontal discussions among environmental sustainability stakeholders to encourage cross-learning and support policy development.

Performance - Efficiency and Economy

Governance for the program is clearly defined and understood. Likewise, the roles and responsibilities of those who contribute to the FSDS and CESI are generally well understood. Challenges were identified, however, related to the effectiveness of current governance processes to secure senior management participation at critical points and obtain input of a strategic nature associated with implementing the FSDA.

The program is viewed by stakeholders as employing efficient operational and administrative processes, including making effective use of technology and leveraging G&C funding. Program representatives and OGD stakeholders were of the view that the program was appropriately resourced to meet its current mandate.

The program collects some performance measurement data related to both the FSDS and CESI. However, at the time of this evaluation, there were gaps in the availability of data for half of the performance measures identified in the FSDS performance measurement framework, although the framework covers a three-year period and most of the measures with missing data were only scheduled to be reported on in 2016. With respect to CESI, the current logic model is outdated, there is no existing performance measurement framework and there are gaps related to measuring the program's contribution to information on the state of the environment, including the program's work on improving the quality of environmental indicators collected by other federal programs.

6. Recommendations and Management Responses

The following recommendations are addressed to the ADM Strategic Policy Branch, as the senior departmental official responsible for the management of the program.

Guided by an outreach and engagement strategy, the program has already taken steps to increase awareness of the FSDS and CESI. While evidence suggests that some progress is being made in this area, the importance of building increased awareness, including helping programs and stakeholders understand the value and potential impact of the FSDS and CESI, was identified as a critical requirement for the program to increase its value and play a greater role in support of decision-making.

The ADM of SPB agrees with the recommendation.

The Sustainability Directorate (SD) acknowledges that continuing to build awareness among target audiences is an important and necessary step toward increasing the impact and use of the FSDS and CESI. The SD will work with Communications Branch to ensure that existing outreach plans are fully implemented. In addition, the SD will leverage the 2016-19 FSDS Consultation phase to both continue to build awareness and to seek input for development of an updated outreach and engagement strategy for the 2016-2019 period.

The program is uniquely positioned to identify the key individuals or groups with expertise or activities contributing to the various strategies outlined in the FSDS. An opportunity was identified to leverage the program's leadership and coordination role to improve information sharing horizontally and facilitate discussions among related program areas. This opportunity was identified as a means by which the program could add greater value and contribute to the increased integration of environmental sustainability into decision-making.

The ADM of SPB agrees with the recommendation.

The Sustainability Directorate will facilitate horizontal discussions among stakeholders.

The program is responsible for ensuring engagement of OGD stakeholders for CESI and the FSDS to ensure they are kept informed at critical points as well as to obtain input of a strategic nature associated with implementing the FSDA. This is accomplished through the hosting of meetings at critical milestones, maintenance of a GCConnex forum, and presentation at various OGD executive tables. While feedback from OGD stakeholders was positive in terms of the program's efficient management of these meetings, an issue was identified related to ensuring senior management participation. The operational nature and length of many of the meetings was identified as a potential barrier resulting in senior managers frequently delegating their attendance to a working level representative. This in turn was viewed as limiting the degree to which discussion or decisions of a more strategic nature occur.

The ADM of SPB agrees with the recommendation.

Building on feedbacks from OGD stakeholders and in a view of continuously improving senior management engagement at critical stages, the Sustainability Directorate took actions to revisit its tactics and communication approaches.

In its capacity as the Sustainable Development Office, the Sustainability Directorate will build on established communication with stakeholders to ensure interdepartmental meetings are held only when necessary and only for the length of time required to cover the material adequately.

The program collects performance measurement data related to both the FSDS and CESI. In combination, many of the key components for a performance measurement strategy for the program have already been developed. However, during the period of the evaluation, data is available for only 10 of 20 FSDS indicators relevant to assessing ECCC's performance, although the Management Framework covers a three-year period and most of the missing indicators were only slated to be reported on in 2016. Likewise, while there is a CESI logic model, there is currently no associated performance measurement framework and the current logic model is outdated. As currently structured, the existing performance measurement instruments do not represent the scope of the program's activities, outputs and outcomes and therefore are not adequate for monitoring and reporting on its performance.

The ADM of SPB agrees with the recommendation.

The FSDS and the CESI program have different logic models representing their different roles/responsibilities. The FSDS Management Framework is updated triennially, and contains the performance measurement strategy that is reported on over a three year period, in line with the FSDS cycle, The CESI logic model is updated as appropriate, supporting the FSDS as well as the Department of the Environment Act and the Canadian Environmental Protection Act by providing information on the state of the environment. Any changes will be done in accordance with the department’s implementation of the TBS Policy on Results.

Annex A - Summary of Findings^Footnote47

Annex B - Mapping of Thematic Outcomes to Logic Models