Language selection

Government of Canada / Gouvernement du Canada

Search

Evaluation of the Environmental Technology program

Final Report
Audit and Evaluation Branch
July 9, 2015

PDF (252 KB)

Report Clearance Steps

Planning phase completed - January 2014
Report sent for management response - February 2015
Management response received - March 2015
Report tabled for approval - March 2015
Report approved by Deputy Minister (DM) - July 2015

Acronyms used in the report

ADM
Assistant Deputy Minister
AEB
Audit and Evaluation Branch
CETACs
Canadian Environmental Technology Advancement Centres
DG
Director General
DPR
Departmental Performance Report
EC
Environment Canada
ecoEII
ecoENERGY Innovation Initiative
ETV
Environmental Technology Verification
FCM
Federation of Canadian Municipalities
Gs&Cs
Grants and Contributions
GHGs
Greenhouse Gases
GMF
Green Municipal Fund
GRDI
Genomics Research and Development Initiative
IAB
International Affairs Branch
ISO
International Organization for Standardization
IWG
International Working Group
NGO
Non-governmental Organization
NRC
National Research Council of Canada
NRCan
Natural Resources Canada
OGD
Other Government Department
PAA
Program Alignment Architecture
PERD
Program of Energy Research and Development
PMF
Performance Measurement Framework
R&D
Research and Development
RBAEP
Risk-Based Audit and Evaluation Plan
RPP
Report on Plans and Priorities
S&T
Science and Technology
SCC
Standards Council of Canada
SDTC
Sustainable Development Technology Canada
SGRP
Soil and Groundwater Remediation Project
SMEs
Small and Medium-Sized Enterprises
SPB
Strategic Policy Branch
STAGE
Strategic Technology Applications of Genomics in the Environment
US
United States

Acknowledgments

The Evaluation Project Team would like to thank those individuals who contributed to this project, particularly members of the Evaluation Committee as well as all interviewees who provided insights and comments crucial to this evaluation.

The Evaluation Project Team was led by Michael Callahan, under the direction of the Environment Canada Evaluation Director, William Blois, and included Urszula Adamik, Natalie Janssens and Kevin Marple. Some of the data collection (key informant interviews and international comparison of Environmental Technology Verification programs) was conducted by gg Consulting.

Prepared by the Evaluation Division, Audit and Evaluation Branch

Table of Contents

Executive Summary

Background

The Environmental Technology sub-program delivers environmental science, technology analysis and assessment, and program management in support of the Government of Canada’s clean air and greenhouse gas (GHG) technology investment decisions, policy making and regulations.

This evaluation addresses three areas of activityFootnote 1 under the Environment Technology sub-program: technology analysis and assessment activities, which are designed to deliver comprehensive analysis of the impacts of environmental technologies; the Environmental Technology Verification (ETV) Program, which was established to promote the uptake of environmental technologies in the marketplace through performance verification and development of specific technology protocols; and Canadian Environmental Technology Advancement Centres (CETACs), which are three not-for-profit organizations created by EC to help small- and medium-sized enterprises (SMEs) advance innovative environmental technologies that address the Government of Canada’s environmental priorities.

A more in-depth assessment of the ETV Program was undertaken given that EC continued to be involved in work on the ETV process at the time data collection for the evaluation was conducted. As the contribution agreement with the delivery agent ended on March 31, 2015, EC is no longer involved in managing the domestic ETV Program. However, the Department’s international activities in the development of the ETV ISO standard continue.

A less comprehensive evaluation of the CETACs was conducted because this activity ended in 2013-14 with the discontinuation of EC’s contribution funding ($1.2 million annually) to the CETACs.

The evaluation covers a five-year period from 2009-10 to 2013-14 and involved a review of program documentation and performance data, an international comparison of ETV programs, and key informant interviews.

Findings and Conclusions

Relevance

There is a continuing need to address a range of environmental issues ultimately targeted by the sub-program (e.g., reducing GHGs and minimizing the environmental impacts of pollution). In addition, there is a need to encourage and support innovation in environmental technologies, to promote the uptake of these technologies to help address environmental needs, and to provide EC’s senior management with analysis of the potential environmental impacts of emerging and new technologies. It is important that evidence of a need for intervention in these areas not be construed as evidence of a need for the ETV Program model. That is, the evaluation does not draw conclusions on the continuing need for this specific program as a means of promoting uptake of environmental technologies.

Although the services provided by the CETACs in supporting and advising SMEs for the commercial development and application of innovative environmental technologies continue to be needed, there are other government programs that can help to address these needs (e.g., the National Research Council’s Industrial Research Assistance Program).

The Environmental Technology sub-program is aligned with EC’s strategic outcomes and priorities related to environmental protection and the development/assessment of technologies to support the Department’s scientific work and regulations. It is also compatible with federal government priorities related to technological innovation and is consistent with federal and departmental roles and responsibilities related to protecting the environment and supporting science, technology and innovation.

Efficiency and Economy

Evidence from the document review and interviews suggests that the design of the ETV Program is appropriate, though some key informants suggest that the level of EC funding for the delivery agent limits the rate at which awareness of and demand for ETV verification by both buyers and suppliers can grow. Achievement of the ETV Program’s intended outcomes (e.g., increased vendor credibility, buyer confidence in and uptake of innovative technologies) is widely perceived as being dependent on continuing growth in awareness and demand for ETV, as well as on the finalization and deployment of an international ETV standard under the International Organization for Standardization (ISO).

As a result of resource constraints in recent years (e.g., Budget 2012 reductions), the sub-program now focuses its technology analysis and assessment activities on analysis of the potential environmental impacts of emerging and new technologies (e.g., hydraulic fracturing in shale gas development) rather than assessments of existing technologies.

The available evidence suggests that ETV activities are being delivered as intended and in an efficient manner. For example, EC provides a relatively modest annual contribution of $300,000 to the delivery agent and, unlike ETV programs in some other jurisdictions, the Canadian approach does not subsidize testing and verification activities (the costs of which are borne by the technology vendors). It is not possible to conclude unequivocally on the efficiency of this program area, however, because detailed data on EC’s expenditures and results for domestic and international work on the ETV Program are unavailable.

The CETACs were seen by key informants to have provided a cost-effective and efficient way for EC to support efforts to increase the uptake of innovative environmental technologies and to leverage the funding provided. This latter point is confirmed by an analysis of financial information showing that 68% of funding for the three CETACs came from sources other than EC. In addition, the ratio of the administrative costs to support the CETACs is quite low (0.04) compared to charitable organizations and other EC grants and contributions programs, which suggests efficient delivery and oversight of the contribution agreements.

The recently redesigned technology analysis and assessment function is considered by the sub-program’s managers to be an efficient approach compatible with current resources. In response to Budget 2012 decisions, technology analysis/assessment and oversight of technology programs were streamlined beginning in 2013-14. As clear data on EC’s expenditures for technology analysis and assessment are unavailable, however, conclusions on the efficiency of this area of activity cannot be corroborated.

The governance structures for the three areas of program activity are generally clear, appropriate and efficient, and performance data were collected and reported in annual reports to EC from delivery agents for the ETV Program and the CETACs, and in summary form in EC’s annual Departmental Performance Reports (DPRs). There have been some challenges, however, in collecting and reporting valid performance information for all outputs and outcomes due to limitations in the division’s capacity and methodological difficulties in collecting data and quantifying environmental benefits.

Achievement of Intended Outcomes

There is limited evidence on the achievement of ETV Program outcomes. The consensus among the interviewees able to comment was that progress is being made but there is not yet conclusive evidence on the program’s degree of success. Data on the sales of environmental technology units in Canada (as an indicator of uptake) and the associated environmental impacts of ETV-verified technologies are not currently available.

The CETACs were successful in improving SMEs’ knowledge and skills, and the implementation of technologies supported by the centres resulted in reduced emissions and other beneficial environmental impacts. However, EC managers have some concerns about the reliability of the data provided by the centres on environmental impacts because of inconsistencies in the methods used to calculate emissions reductions.

Analysis and assessment work to date has contributed to senior management’s understanding of environmental technology issues (e.g., biofuels, fracking, and solar photovoltaic technologies), and the sub-program’s managers expect the technology analysis function to continue to contribute to better informed senior management decision making following recent changes in the sub-program’s approach.

No negative unintended outcomes have resulted from the sub-program’s activities. Reported positive unintended outcomes include the use of aspects of the Canadian ETV approach in other jurisdictions.

Recommendations

Given the dearth of performance information on the Canadian ETV Program, the evaluation evidence indicates a need for the development and implementation of a performance measurement strategy for this program, including the estimation of environmental impacts attributable to ETV-verified technologies. However, given a recent decision to no longer provide contribution funding to the ETV delivery agent beyond March 2015, such a recommendation is no longer relevant for EC. Nonetheless, the Department’s international activities in the development of the ETV ISO standard continue.

The following recommendation is directed to the Assistant Deputy Minister, Science and Technology Branch, as the senior departmental official responsible for the management of the Environmental Technology sub-program:

Recommendation 1: Develop and implement strategies to improve awareness and uptake of ETV-verified technologies.

The responsible Assistant Deputy Minister, Science and Technology Branch, agrees with this recommendation and has developed a management response that appropriately addresses the recommendation. The full management response can be found in section 6 of the report.

1.0    Introduction

This report presents the findings of the Evaluation of Environmental Technology (Program Alignment Architecture (PAA) sub-program 3.2.3), which was conducted between January and September 2014 by gg Consulting and Environment Canada’s (EC) Audit and Evaluation Branch (AEB). The evaluation was conducted in order to meet the coverage requirements of the Financial Administration Act (for grants and contributions (Gs&Cs)) and the Treasury Board Policy on Evaluation (for direct program spending), which require that an evaluation of all ongoing Gs&Cs and direct program spending be conducted at least once every five years. The evaluation covers a five-year period from 2009-10 to 2013-14.

2.0    Background

2.1    Program Profile

Through activities under the Environmental Technology sub-program, Environment Canada delivers environmental science, technology analysis and assessment, and program management in support of the Government of Canada’s clean air and greenhouse gas (GHG) technology investment decisions, policy making and regulations. The three areas of activity addressed in this evaluation are as follows:

In addition to these activities, it should be noted that this sub-program includes management and oversight of EC’s participation in six federal technology programs Footnote 2 led by other federal departments and agencies. These activities are excluded from consideration in the current evaluation because separate evaluations of these programs have been or will be undertaken by the lead organization. A more in-depth assessment of the ETV Program was undertaken, given that EC continued to be involved in work on the ETV process at the time data collection for the evaluation was conducted. As the contribution agreement with the delivery agent ended on March 31, 2015, EC is no longer involved in managing the domestic ETV Program. However, the Department’s international activities in the development of the ETV ISO standard continue.

By comparison, a less comprehensive evaluation of the CETACs was conducted because this activity ended in 2013-14 with the discontinuation of Environment Canada funding to the CETACs. An overview of the activities included in the technology analysis and assessment function, ETV Program and CETACs is provided below.

Technology Analysis and Assessment

This activity is designed to deliver comprehensive technology analysis (e.g., in briefing notes, presentation decks) and formal technology assessment reports mainly for use within Environment Canada.Footnote 3These analyses and assessments are based on an integration of the full body of scientific literature on a given technology.

Technology analysis and assessment is provided mainly to internal Environment Canada user groups, for example:

Some of the analysis and assessment work supports Environment Canada's participation in interdepartmental governance bodies overseeing horizontal initiatives (e.g., Assistant Deputy Minister (ADM) Coordinating Committee for the Genomics Research and Development Initiative (GRDI), the Interdepartmental ADM Energy Science and Technology (S&T) Committee),Footnote 4 ,Footnote 5 and some work supports interdepartmental clean technology committees (i.e., not related to specific programs), such as the ADM Wind Energy Hub Committee.Footnote 6

Technology assessment reports were sometimes developed in consultation with external stakeholders such as non-governmental organizations (NGOs) and industry associations.Footnote 7Once completed, some final assessment reports were shared with external stakeholders. Work on producing assessment reports on existing technologies ended in 2012. The program currently focuses on conducting analysis of the potential environmental impacts of emerging and new technologies.

ETV Program

Environment Canada established the Canadian ETV Program in 1997 to support the implementation of innovative environmental technologies in Canada in areas intended to help the Department meet its environmental and regulatory priorities through the provision of: Footnote 8

The ETV Program was designed to promote the uptake of environmental technologies in the marketplace through performance verification and development of specific technology protocols (i.e., sector-based performance criteria). The performance verification component involves working closely with technology innovators and qualified testing organizations. Activities under the ETV umbrella also include ensuring the Canadian approach is well-coordinated with and, where applicable, harmonized with those of government and international organizations, within and outside Canada.

In managing the Canadian ETV Program, Environment Canada staff deliver activities such as: overseeing the third-party delivery agent Footnote 9that delivers the independent, technical component of the program; developing partnerships with provinces (e.g., Quebec) and municipalities to incorporate ETV into their respective management and decision-making process (procurement, legislation, funding programs, etc.); and ensuring that available environmental performance data (specifically, emissions reductions of greenhouse gases (GHGs) and air pollutants associated with ETV-verified technologies) are submitted to the annual departmental Performance Management Framework (PMF). International activities include leading the development of an international ETV standard under the International Organization for Standardization (ISO) by collaborating with the Standards Council of Canada (SCC), co-chairing the International Working Group (IWG) on ETV, participating in co-verification projects with the IWG members (e.g., the European Union, the Philippines and the Republic of Korea), and continuing bilateral cooperation with other countries, such as Japan and China.

Canadian Environmental Technology Advancement Centres (CETACs)

In 1994, Environment Canada created three CETACs as private, not-for-profit organizations to help small- and medium-sized enterprises (SMEs) advance innovative environmental technologies that address the Government of Canada’s environmental priorities. Environment Canada provided funding to the CETACs until 2014. Since 2014, the CETACs have restructured their operations and continue to operate independently of EC funding.

From 1994 to 2014, the three centres assisted an average of 300 SMEs every year. They operated in partnership with provincial governments, industry associations and the private sector, and are regionally structured to include:

While funded by Environment Canada, the CETACs assisted environmental enterprises in increasing the uptake of environmental technologies by providing a wide range of services, including assistance in accessing investment capital, general business development counselling, demonstration projects, technical services, market analysis, and strategic advisory and mentoring services. In the past, the CETACs have assisted companies in obtaining financing and, further down the road, in developing strategies to use these finances to move the project to the next stage of uptake.Footnote 13As noted above, Environment Canada’s annual contribution of $1.2 million to the CETACs ended in 2013-14, but the three centres continue to operate with somewhat different mandates.Footnote 14

2.2    Management and Governance

At Environment Canada, the management of the Environmental Technology sub-program 3.2.3 is provided by the Science Management Division and S&T Policy Division of the S&T Strategies Directorate, S&T Branch. Management and administration of the three areas of activity is as follows:

2.3    Resource Allocation

Sub-program 3.2.3 expenditures for 2009-10 to 2013-14 are presented in Table 1.
Note that sub-program 3.2.3 was impacted by reductions attributable to the Budget 2012 decisions. Changes included:

Table 1: Environmental Technology Sub-Program Expenditures for 2009-10 to 2013-14
S&T Branch 2009-10 2010-11 2011-12 2012-13 2013-14 Total
FTEs 15.4 18.5 13.7 10.2 12.7 -
Salaries $1,363,463 $1,510,822 $1,222,558 $1,133,196 $1,098,792 $6,328,831
O&M $1,160,819 $1,055,105 $1,102,770 $967,157 $1,489,452 $5,775,303
Gs&Cs $1,400,000 $1,400,000 $1,400,000 $1,500,000 $1,500,000 $7,200,000
Total Costs $3,924,282 $3,965,927 $3,725,328 $3,600,353 $4,088,244 $19,304,134

Notes:
1. FTE information extracted from EC’s Salary Management System (SMS). Note that in 2013-14, the SMS indicates a total of 52.7 FTEs for S&T Branch. However, program management estimates that approximately 40 of these FTEs worked in the broader S&T Strategies Directorate, not directly on sub-program 3.2.3. Therefore, 12.7 FTEs are reported for 2013-14.
2. Other data from EC’s financial system, as provided by Finance Branch, July 25, 2014.
3. Program management estimates that total direct costs (salaries and O&M) were $1,630,000 for the ETV Program, $210,000 for the CETACs, and $1,802,000 for technology analysis and assessment. The remaining direct costs ($8,462,134) were for the sub-program’s management and oversight of six federal technology programs led by other federal departments and agencies.
4. Gs&Cs indicate EC’s contribution funding to the ETV Program ($1,200,000) and CETACs ($6,000,000).

2.4    Intended Outcomes

The intended outcomes for each of the three areas assessed in this evaluation were identified in consultation with program management, as logic models for these areas of activity had not been developed. These intended outcomes are as follows:

ETV Program

CETACs

Technology Analysis and Assessment

3.0    Evaluation Design

3.1    Purpose and Scope

The evaluation of Environmental Technology is part of Environment Canada’s 2012 Risk-based Audit and Evaluation Plan, which was approved by the Deputy Minister, and covers the five-year timeframe from 2009-10 to 2013-14.

The purpose of the evaluation is to assess the relevance and performance (including effectiveness, efficiency, and economy) of the Environmental Technology sub-program.  The evaluation focused on Environment Canada-led components, including the related G&C agreements, which have not been recently evaluated. These include:

As noted earlier, a more in-depth assessment of the ETV Program was undertaken, including an international comparison and assessment of options and alternative processes for Canada’s ETV Program. On the other hand, only a limited evaluation of the CETACs was undertaken because Environment Canada’s annual contribution of $1.2 million to this program ended in 2013-14. The evaluation of this program area relied on existing documentation, performance data and a limited number of interviews to assess the evaluation questions.

This evaluation excluded other specific programs under the Environmental Technology sub-program (SD Tech Fund, NextGen Biofuels Fund, PERD, ecoEII, GMF and STAGE/GRDI) because their evaluations have been or will be led by other federal departments or funded organizations.

3.2    Evaluation Approach and Methodology

The methodological approachFootnote 16 consisted of a review of existing documentation and data, an international comparison of ETV programs, and a round of interviews with key informants, as described below.

Review of Documentation and Data: A primary methodological approach was a review of documentation, including federal government and departmental documents (e.g., EC Reports on Plans and Priorities and Departmental Performance Reports, Speeches from the Throne and Federal Budgets) and program documents (e.g., decks presented to senior management, policy and planning documents). In addition, G&C files for ETV and the CETACs (e.g., annual reports submitted to EC), reports on interviews with ETV clients and on surveys of CETAC clients, selected literature and available performance data were reviewed. The document review provided some evidence on all of the evaluation questions. For each evaluation question, the evidence from each relevant document was summarized in a template and a summary was then prepared for that question.

International Comparison of ETV Program: An international comparison and assessment of options for the ETV Program was conducted, including a document and literature review, as well as a small number of key informant interviews. This component of the evaluation included: (1) an overall comparison of the outcomes of the Canadian ETV Program with ETV programs in the European Union, the United States, Japan, the Republic of Korea and the Philippines, based on material published on the relevant organizations’ websites, and (2) an assessment of options or alternative processes for the Canadian ETV Program. The international comparison provided supplementary evidence for the following evaluation questions: continued need, appropriateness of program design, efficiency of activities, lessons learned and achievement of intended outcomes.

Key Informant Interviews: Interviews were conducted to obtain informed opinions and observations on the evaluation questions, particularly those for which there was insufficient documentary evidence. Interview guides consisting of open-ended questions were utilized for these interviews, and a customized guide was developed for each major type of key informant. The interviews were conducted in-person or by telephone, and were an average of 60 to 90 minutes in duration.

A listing of knowledgeable interview candidates was developed in collaboration with program representatives and all of these candidates were invited to participate (though some were unavailable or declined). A total of 25 key informants were interviewed from the following groups:

Key informant interviews yielded important information to help address all of the evaluation questions. For each evaluation question, the interview findings for each relevant key informant group were summarized in a template.   

3.3    Limitations

4.0    Findings

This section presents the findings of this evaluation by evaluation issue (relevance and performance) and by the related evaluation questions.

For each evaluation question, a rating is provided based on a judgment of the evaluation findings. The rating statements and their significance are outlined below in Table2. A summary of the ratings for each evaluation question is presented in Annex 1.

Table 2: Definitions of Standard Rating Statements
Statement Definition
Acceptable The program has demonstrated that it has met the expectations with respect to the issue area.
Opportunity for improvement The program has demonstrated that it has made adequate progress to meet the expectations with respect to the issue area, but continued improvement can still be made.
Attention required The program has not demonstrated that it has made adequate progress to meet the expectations with respect to the issue area and attention is needed on a priority basis.
Not applicable There is no expectation that the program would have addressed the evaluation issue.
Unable to assess Insufficient evidence is available to support a rating.

The reader should note that ratings are assigned individually by evaluation issue and that overall conclusions of the program’s relevance or performance will not necessarily be consistent with every individual issue rating.

4.1    Relevance

4.1.1    Continued Need

Evaluation Issue: Relevance Rating

1. Is there a continued need for the following areas of activity as part of this sub-program?

  • ETV Program
  • CETACs
  • Provision of technology analysis and assessment
 Acceptable

There is a continuing need to address a range of environmental issues targeted by the sub-program. In addition, there is a need to encourage and support innovation in environmental technologies, to promote the uptake of these technologies among consumers, and to provide EC’s senior management with analysis of the environmental impacts of technologies to support policy and regulatory development.

4.1.2    Alignment with Government Priorities

Evaluation Issue: Relevance Rating

2. Are the following areas of activity aligned with federal government priorities?

  • ETV Program
  • Provision of technology analysis and assessment
 Acceptable

The Environmental Technology sub-program is aligned with Environment Canada’s strategic outcomes and priorities related to environmental protection and the development/assessment of technologies to support the Department’s scientific work and regulations. It is also compatible with federal government priorities related to technological innovation.

4.1.3    Consistency with Federal Roles and Responsibilities

Evaluation Issue: Relevance Rating
3. Are the areas of activity consistent with federal roles and responsibilities? Acceptable

The Environmental Technology sub-program is consistent with federal and departmental roles and responsibilities related to protecting the environment and supporting science, technology and innovation.

4.2    Performance

4.2.1   Efficiency and Economy

Evaluation Issue: Performance - Economy and Efficiency Rating

4. Is the design appropriate for achieving intended outcomes of the following areas of activity?

  • ETV Program
  • Provision of technology analysis and assessment
 Acceptable

The evidence indicates that the design of the ETV Program is appropriate, but some key informants suggest that the level of EC funding for the delivery agent ($300K annually) limits the rate at which awareness of and demand for ETV verification by both buyers and suppliers can grow. With respect to technology analysis and assessment, as a result of resource constraints in recent years (e.g.,  Budget 2012 reductions), the division has shifted its approach away from assessing existing technologies to conducting analysis of the potential environmental impacts of emerging and new technologies.

Evaluation Issue: Performance - Economy and Efficiency Rating
5. To what extent is the governance structure clear, appropriate and efficient for achieving expected results? Acceptable

The evaluation evidence indicates that governance structures for the areas of program activity examined in this evaluation are generally clear, appropriate and efficient. As described in sections 2.1 and 2.2, management of sub-program 3.2.3 is the responsibility of the Science Management Division (for the ETV Program and CETACs) and S&T Policy Division (for technology analysis).

Evaluation Issue: Performance - Economy and Efficiency Rating

6. Are activities being undertaken and products being delivered at the lowest possible cost? How could the efficiency of the activities be improved? Are there alternative, more economical, ways of delivering the outputs of the following areas of activity?

  • ETV Program
  • CETACs
  • Provision of technology analysis and assessment
 Unable to Assess

The available evidence suggests that current ETV activities are being delivered as intended and in a reasonably efficient manner. EC provides a relatively modest annual contribution of $300K to the delivery agent and, unlike ETV programs in some other jurisdictions, the Canadian approach does not subsidize testing and verification activities (the costs of which are borne by the technology vendors). However, in the absence of detailed data on EC’s expenditures and results for domestic and international work on the ETV Program, it is not possible to draw a conclusion on the efficiency of program delivery.

Table 3: ETV Activities and Outputs 2009-10 to 2013-14
Time Period (Delivery Agent) Inquiries Inquiries Moved to Pre-screening Application or Requested More Discussion and/or Review of Existing Test Plan and Data  Developed and/or Implemented Test Plan or Verification Plan Verifications Completed Renewals Completed Number of Canadian ETV Licenses Held
March 2009 to March 2012 (OCETA/BLOOM) 129 27 19 6 11 20 licenses held by 14 companies
October 2012 to April 2014 (Globe) 99 20 16 2 5 19 licenses held by 15 companies
Total 228 47 35 8 16 -

Source: ETV annual reports submitted to EC by delivery agents.

The CETACs were seen by key informants to have provided a cost-effective and efficient way for EC to support efforts to increase the uptake of innovative environmental technologies and to leverage the funding provided. An analysis of financial information indicates that, for the five-year evaluation timeframe, EC contributed 32% of funding for the three CETACs, while 68% came from other sources (i.e., other federal departments, other levels of government and the private sector). The administrative ratio for EC’s work on the CETACs is low (0.04), which suggests efficient delivery and oversight of the contribution agreements.

The recently redesigned technology analysis and assessment function (which now focuses on analysis of potential environmental impacts of emerging and new technologies) is considered by the sub-program’s managers to be an efficient approach aligned to current resources. In response to Budget 2012 decisions, technology analysis/assessment and oversight of technology programs were streamlined beginning in 2013-14. As detailed data on EC’s expenditures for technology analysis and assessment are unavailable, however, it is not possible to make a conclusion on the efficiency of this area of activity.

Evaluation Issue: Performance - Economy and Efficiency Rating
7. Are performance data being collected and reported? If so, is this information being used to inform senior management/decision makers? Unable to Assess

Performance data are being collected and reported in annual reports to EC from delivery agents for the ETV Program and CETACs and, in summary form, in the DPRs. Challenges have been encountered in collecting and reporting valid performance information for all outputs and outcomes of interest. This is due to methodological difficulties in collecting data and quantifying environmental benefits and to limitations on the division’s capacity to fulfil performance measurement requirements.

Evaluation Issue: Performance - Economy and Efficiency Rating
8. What lessons have been learned from the areas of activity? Not Applicable

Some notable lessons identified in documentation and interviews are that the development of awareness and demand for verified technologies among both suppliers and potential buyers is key to the success of the ETV Program and that it takes longer than initially anticipated for companies to develop a market and become commercially successful in environmental technology. 

4.2.2   Achievement of Intended Outcomes

Evaluation Issue: Performance - Effectiveness Rating

9. To what extent have intended outcomes been achieved as a result of the following areas of activity?

  • ETV Program
  • CETACs
  • Provision of technology analysis and assessment
 See Below

Environmental Technology Verification (ETV) Program

There is limited evidence on the achievement of ETV Program outcomes. The consensus among the interviewees able to comment was that progress is being made but there is not yet conclusive evidence on the program’s degree of success. Data on the sales of environmental technology units in Canada (as an indicator of uptake) and the environmental impacts of ETV-verified technologies are not currently available.

Canadian Environmental Technology Advancement Centres (CETACs)

The available evidence suggests that the CETACs have been successful in improving SMEs’ knowledge and skills and that the implementation of technologies supported by the centres has resulted in reduced emissions and other beneficial environmental impacts. However, the division’s managers expressed some concern about the validity of the data provided by the centres on environmental impacts due to the difficulties in measuring uptake of clients’ technologies and the associated estimated environmental benefits.

Provision of Technology Analysis and Assessment

Analysis and assessment work to date has contributed to senior management’s understanding of environmental technology issues (e.g., issues related to biofuels, fracking and solar photovoltaic technologies). The sub-program’s managers believe that technology analysis and assessment should contribute to better informed decision-making, but due to recent changes in the S&T Policy Division’s approach, it is premature to draw conclusions on this outcome.

Table 4: Estimated Environmental Impacts of Technologies Supported by CETACs
Environmental Impact 2011-12 2012-13 2013-14
Reduced Emissions of Air Pollutants 2,142 tonnes 16 tonnes -
Reduced Emissions of GHGs 518,204 tonnes 122,792 tonnes 229,873 tonnes
Reduced Water Consumption 926,506 m3 1,043,967 m3 -
Improved Water Quality: Turbidity 400 FTU 475 FTU -
Improved Water Quality: Dissolved Solids 1,000 kg 750 kg -
Improved Water Quality: Biological Oxygen Demand - - -
Improved Water Quality:
Other		 824,170 tonnes 8,964 tonnes -
Reduced Release of Toxic Substances 8,921 tonnes 6,769 tonnes 201 tonnes
Treatment of Contaminated Soil 612 tonnes 100 tonnes 54,000 tonnes
Diversion of Waste 61,539 tonnes 3,178 tonnes -
Reduced Energy Consumption: Natural Gas 2,287,527 m3 4,802,418 m3 9,908,239 m3
Reduced Energy Consumption: Electricity 52,523 MWh 11,435 MWh -
Reduced Energy Consumption: Other 452,263 L - -
Reduced Release of Ozone Depleting Substances 45 kg 45 kg -

Note: Figures are based on the data provided to EC in CETAC annual reports and represent approximations of the total reduced emissions and other environmental impacts for the three CETACs combined. For 2013-14, data were only available for two of the three CETACs (because BLOOM only provided figures for these three years combined, not separately for 2013-14). These figures should be regarded as estimates only due to inconsistencies in the methodologies used by the centres for calculating emissions reductions.

Evaluation Issue: Performance  - Effectiveness Rating
10. Have there been any unintended (positive or negative) outcomes? Acceptable

No negative unintended outcomes have resulted from the sub-program’s activities. Reported positive unintended outcomes include the use of aspects of the Canadian ETV approach in other jurisdictions.

5.0    Conclusions

 Relevance

Efficiency and Economy

Achievement of Intended Outcomes

6.0   Recommendation and Management Response

Given the dearth of performance information on the Canadian ETV Program, the evaluation evidence indicates a need for the development and implementation of a performance measurement strategy for this program, including the estimation of environmental impacts attributable to ETV-verified technologies. However, given a recent decision by EC senior management to no longer provide contribution funding to the ETV delivery agent beyond March 2015, such a recommendation is no longer relevant for EC.

The following recommendation is directed to the Assistant Deputy Minister, Science and Technology Branch, as the senior departmental official responsible for the management of the Environmental Technology sub-program:

Recommendation 1: Develop and implement strategies to improve awareness and uptake of ETV-verified technologies. The evaluation evidence suggests that the success of the ETV process is dependent on continuing growth in awareness and demand for ETV verification in key sectors such as water/wastewater management and mining, as well as finalization and deployment of the ETV ISO standard and recognition of the value of ETV and ISO in international markets for environmental technologies. In addition, efforts to foster inclusion and/or recognition of ETV verification in public procurement policies could facilitate uptake. Development of strategies should take into account factors such as: which sectors offer the best potential for emission reductions and other environmental benefits; which activities have the best potential to generate increases in awareness and interest (e.g., pilot and demonstration projects); the level of funding and human resources necessary to undertake such activities; the rate at which the verification infrastructure should be expanded to respond to a growth in demand; and the extent to which partners and additional resources could be engaged to facilitate the expanded level of ETV activity. 

Statement of Agreement/Disagreement with the Recommendation

The ADM, Science and Technology Branch, agrees with the recommendation.

Management Action

It is understood that the success of the ETV process is closely linked with awareness and related benefits. That said, although our efforts regarding the management of the Canadian ETV Program ended as of March 31, 2015, our efforts in developing and promoting the ISO-ETV standard (ISO 14034) are expanding. It is expected that once ISO 14034 is developed and established in Canada in 2016-17, the responsibility of implementing the ETV process will be market driven.  

  1. The development of the ISO 14034 is well underway.  A Draft International Standard (DIS) was submitted to the ISO Secretariat in Geneva in March 2015. The DIS will now be translated and reviewed by all ISO countries (~ 140 countries potentially). Uptake and knowledge of ISO 14034 by stakeholders in Canada is essential for the longer term success of ETV. We will collaborate with CSA Group (formerly the Canadian Standards Association) as well as the Standards Council of Canada (SCC) to engage Canadian stakeholders to ensure awareness, usage and/or implementation of the ETV process.
  2. Continued collaboration with important programs and organizations, such as Sustainable Development Technology Canada (SDTC) and the Green Municipal Fund (GMF) to implement ETV in their evaluation processes. This action will help foster the inclusion and recognition of ETV verification in public policies.
  3. Building on a recent reciprocity agreement for verification of wastewater and drinking water technologies between the delivery agent and the Bureau de normalisation du Québec (BNQ), a pilot project will be undertaken to increase awareness and interest of ETV in this key sector.
Timeline Deliverable(s) Responsible Party
July 2015 Second year of MOU with the Standards Council of Canada. Main tasks of SCC will be to distribute the DIS to all key Canadian stakeholders and inform them about the upcoming ISO-ETV standard. SCC will also work with technical stakeholders to develop an accreditation framework for ETV. ADM S&T Branch
June 2015 Contribution agreement with CSA Group that will focus on ETV outreach ensuring awareness and knowledge of ETV in Canada. ADM S&T Branch
September 2015 Pilot project in place with BNQ to verify a wastewater or drinking water technology that will test/evaluate the process in place and increase the interest for further verifications in the sector. ADM S&T Branch

Annex 1

Summary of Findings Footnote 64
Evaluation
Question		 Acceptable Opportunity for Improvement Attention Required Not Applicable Unable to Assess
Relevance:  
1. Is there a continued need for the areas of activity as part of this sub-program? Yes No No No No
2. Are the areas of activity aligned with federal government priorities? Yes No No No No
3. Are the areas of activity consistent with federal roles and responsibilities? Yes No No No No
Performance:  
4. Is the design appropriate for achieving intended outcomes of the areas of activity? Yes No No No No
5. To what extent is the governance structure clear, appropriate and efficient for achieving expected results? Yes No No No No
6. Are activities being undertaken and products being delivered at the lowest possible cost?  No No No No Yes
7. Are performance data being collected and reported? No No Yes  No No
8. What lessons have been learned from the areas of activity? No No No Yes No
9. To what extent have intended outcomes been achieved as a result of the areas of activity?
ETV Program
Increased vendor credibility and buyer confidence in Canadian environmental technologies No Yes No No No
Increased uptake of innovative environmental technologies in Canada in areas that help EC meet its environmental and regulatory priorities No No No No Yes
CETACs
Improved knowledge and skills among SMEs for the uptake of environmental technologies Yes No No No No
Increased uptake of environmental technologies developed by SMEs Yes No No No No
Reduced emissions from implementation of environmental technologies supported by the CETACs Yes No No No No
Provision of Technology Analysis and Assessment
Increased understanding of the environmental impacts of new and emerging technologies by EC senior management Yes No No No No
Improved decision-making by senior management, departmental policy and regulatory user groups, and interdepartmental governance bodies No No No No Yes
10. Have there been any unintended (positive or negative) outcomes? Yes No No No No

Page details

Date modified: