Evaluation of the Environmental Technology program

Final Report
Audit and Evaluation Branch
July 9, 2015

PDF (252 KB)

Report Clearance Steps

Planning phase completed - January 2014
Report sent for management response - February 2015
Management response received - March 2015
Report tabled for approval - March 2015
Report approved by Deputy Minister (DM) - July 2015

Acronyms used in the report

ADM
Assistant Deputy Minister
AEB
Audit and Evaluation Branch
CETACs
Canadian Environmental Technology Advancement Centres
DG
Director General
DPR
Departmental Performance Report
EC
Environment Canada
ecoEII
ecoENERGY Innovation Initiative
ETV
Environmental Technology Verification
FCM
Federation of Canadian Municipalities
Gs&Cs
Grants and Contributions
GHGs
Greenhouse Gases
GMF
Green Municipal Fund
GRDI
Genomics Research and Development Initiative
IAB
International Affairs Branch
ISO
International Organization for Standardization
IWG
International Working Group
NGO
Non-governmental Organization
NRC
National Research Council of Canada
NRCan
Natural Resources Canada
OGD
Other Government Department
PAA
Program Alignment Architecture
PERD
Program of Energy Research and Development
PMF
Performance Measurement Framework
R&D
Research and Development
RBAEP
Risk-Based Audit and Evaluation Plan
RPP
Report on Plans and Priorities
S&T
Science and Technology
SCC
Standards Council of Canada
SDTC
Sustainable Development Technology Canada
SGRP
Soil and Groundwater Remediation Project
SMEs
Small and Medium-Sized Enterprises
SPB
Strategic Policy Branch
STAGE
Strategic Technology Applications of Genomics in the Environment
US
United States

Acknowledgments

The Evaluation Project Team would like to thank those individuals who contributed to this project, particularly members of the Evaluation Committee as well as all interviewees who provided insights and comments crucial to this evaluation.

The Evaluation Project Team was led by Michael Callahan, under the direction of the Environment Canada Evaluation Director, William Blois, and included Urszula Adamik, Natalie Janssens and Kevin Marple. Some of the data collection (key informant interviews and international comparison of Environmental Technology Verification programs) was conducted by gg Consulting.

Prepared by the Evaluation Division, Audit and Evaluation Branch

Executive Summary

Background

The Environmental Technology sub-program delivers environmental science, technology analysis and assessment, and program management in support of the Government of Canada’s clean air and greenhouse gas (GHG) technology investment decisions, policy making and regulations.

This evaluation addresses three areas of activityFootnote 1 under the Environment Technology sub-program: technology analysis and assessment activities, which are designed to deliver comprehensive analysis of the impacts of environmental technologies; the Environmental Technology Verification (ETV) Program, which was established to promote the uptake of environmental technologies in the marketplace through performance verification and development of specific technology protocols; and Canadian Environmental Technology Advancement Centres (CETACs), which are three not-for-profit organizations created by EC to help small- and medium-sized enterprises (SMEs) advance innovative environmental technologies that address the Government of Canada’s environmental priorities.

A more in-depth assessment of the ETV Program was undertaken given that EC continued to be involved in work on the ETV process at the time data collection for the evaluation was conducted. As the contribution agreement with the delivery agent ended on March 31, 2015, EC is no longer involved in managing the domestic ETV Program. However, the Department’s international activities in the development of the ETV ISO standard continue.

A less comprehensive evaluation of the CETACs was conducted because this activity ended in 2013-14 with the discontinuation of EC’s contribution funding ($1.2 million annually) to the CETACs.

The evaluation covers a five-year period from 2009-10 to 2013-14 and involved a review of program documentation and performance data, an international comparison of ETV programs, and key informant interviews.

Findings and Conclusions

Relevance

There is a continuing need to address a range of environmental issues ultimately targeted by the sub-program (e.g., reducing GHGs and minimizing the environmental impacts of pollution). In addition, there is a need to encourage and support innovation in environmental technologies, to promote the uptake of these technologies to help address environmental needs, and to provide EC’s senior management with analysis of the potential environmental impacts of emerging and new technologies. It is important that evidence of a need for intervention in these areas not be construed as evidence of a need for the ETV Program model. That is, the evaluation does not draw conclusions on the continuing need for this specific program as a means of promoting uptake of environmental technologies.

Although the services provided by the CETACs in supporting and advising SMEs for the commercial development and application of innovative environmental technologies continue to be needed, there are other government programs that can help to address these needs (e.g., the National Research Council’s Industrial Research Assistance Program).

The Environmental Technology sub-program is aligned with EC’s strategic outcomes and priorities related to environmental protection and the development/assessment of technologies to support the Department’s scientific work and regulations. It is also compatible with federal government priorities related to technological innovation and is consistent with federal and departmental roles and responsibilities related to protecting the environment and supporting science, technology and innovation.

Efficiency and Economy

Evidence from the document review and interviews suggests that the design of the ETV Program is appropriate, though some key informants suggest that the level of EC funding for the delivery agent limits the rate at which awareness of and demand for ETV verification by both buyers and suppliers can grow. Achievement of the ETV Program’s intended outcomes (e.g., increased vendor credibility, buyer confidence in and uptake of innovative technologies) is widely perceived as being dependent on continuing growth in awareness and demand for ETV, as well as on the finalization and deployment of an international ETV standard under the International Organization for Standardization (ISO).

As a result of resource constraints in recent years (e.g., Budget 2012 reductions), the sub-program now focuses its technology analysis and assessment activities on analysis of the potential environmental impacts of emerging and new technologies (e.g., hydraulic fracturing in shale gas development) rather than assessments of existing technologies.

The available evidence suggests that ETV activities are being delivered as intended and in an efficient manner. For example, EC provides a relatively modest annual contribution of $300,000 to the delivery agent and, unlike ETV programs in some other jurisdictions, the Canadian approach does not subsidize testing and verification activities (the costs of which are borne by the technology vendors). It is not possible to conclude unequivocally on the efficiency of this program area, however, because detailed data on EC’s expenditures and results for domestic and international work on the ETV Program are unavailable.

The CETACs were seen by key informants to have provided a cost-effective and efficient way for EC to support efforts to increase the uptake of innovative environmental technologies and to leverage the funding provided. This latter point is confirmed by an analysis of financial information showing that 68% of funding for the three CETACs came from sources other than EC. In addition, the ratio of the administrative costs to support the CETACs is quite low (0.04) compared to charitable organizations and other EC grants and contributions programs, which suggests efficient delivery and oversight of the contribution agreements.

The recently redesigned technology analysis and assessment function is considered by the sub-program’s managers to be an efficient approach compatible with current resources. In response to Budget 2012 decisions, technology analysis/assessment and oversight of technology programs were streamlined beginning in 2013-14. As clear data on EC’s expenditures for technology analysis and assessment are unavailable, however, conclusions on the efficiency of this area of activity cannot be corroborated.

The governance structures for the three areas of program activity are generally clear, appropriate and efficient, and performance data were collected and reported in annual reports to EC from delivery agents for the ETV Program and the CETACs, and in summary form in EC’s annual Departmental Performance Reports (DPRs). There have been some challenges, however, in collecting and reporting valid performance information for all outputs and outcomes due to limitations in the division’s capacity and methodological difficulties in collecting data and quantifying environmental benefits.

Achievement of Intended Outcomes

There is limited evidence on the achievement of ETV Program outcomes. The consensus among the interviewees able to comment was that progress is being made but there is not yet conclusive evidence on the program’s degree of success. Data on the sales of environmental technology units in Canada (as an indicator of uptake) and the associated environmental impacts of ETV-verified technologies are not currently available.

The CETACs were successful in improving SMEs’ knowledge and skills, and the implementation of technologies supported by the centres resulted in reduced emissions and other beneficial environmental impacts. However, EC managers have some concerns about the reliability of the data provided by the centres on environmental impacts because of inconsistencies in the methods used to calculate emissions reductions.

Analysis and assessment work to date has contributed to senior management’s understanding of environmental technology issues (e.g., biofuels, fracking, and solar photovoltaic technologies), and the sub-program’s managers expect the technology analysis function to continue to contribute to better informed senior management decision making following recent changes in the sub-program’s approach.

No negative unintended outcomes have resulted from the sub-program’s activities. Reported positive unintended outcomes include the use of aspects of the Canadian ETV approach in other jurisdictions.

Recommendations

Given the dearth of performance information on the Canadian ETV Program, the evaluation evidence indicates a need for the development and implementation of a performance measurement strategy for this program, including the estimation of environmental impacts attributable to ETV-verified technologies. However, given a recent decision to no longer provide contribution funding to the ETV delivery agent beyond March 2015, such a recommendation is no longer relevant for EC. Nonetheless, the Department’s international activities in the development of the ETV ISO standard continue.

The following recommendation is directed to the Assistant Deputy Minister, Science and Technology Branch, as the senior departmental official responsible for the management of the Environmental Technology sub-program:

Recommendation 1: Develop and implement strategies to improve awareness and uptake of ETV-verified technologies.

The responsible Assistant Deputy Minister, Science and Technology Branch, agrees with this recommendation and has developed a management response that appropriately addresses the recommendation. The full management response can be found in section 6 of the report.

1.0    Introduction

This report presents the findings of the Evaluation of Environmental Technology (Program Alignment Architecture (PAA) sub-program 3.2.3), which was conducted between January and September 2014 by gg Consulting and Environment Canada’s (EC) Audit and Evaluation Branch (AEB). The evaluation was conducted in order to meet the coverage requirements of the Financial Administration Act (for grants and contributions (Gs&Cs)) and the Treasury Board Policy on Evaluation (for direct program spending), which require that an evaluation of all ongoing Gs&Cs and direct program spending be conducted at least once every five years. The evaluation covers a five-year period from 2009-10 to 2013-14.

2.0    Background

2.1    Program Profile

Through activities under the Environmental Technology sub-program, Environment Canada delivers environmental science, technology analysis and assessment, and program management in support of the Government of Canada’s clean air and greenhouse gas (GHG) technology investment decisions, policy making and regulations. The three areas of activity addressed in this evaluation are as follows:

  • technology analysis and assessment activities;
  • Environmental Technology Verification (ETV) program; and
  • Canadian Environmental Technology Advancement Centres (CETACs).

In addition to these activities, it should be noted that this sub-program includes management and oversight of EC’s participation in six federal technology programs Footnote 2 led by other federal departments and agencies. These activities are excluded from consideration in the current evaluation because separate evaluations of these programs have been or will be undertaken by the lead organization. A more in-depth assessment of the ETV Program was undertaken, given that EC continued to be involved in work on the ETV process at the time data collection for the evaluation was conducted. As the contribution agreement with the delivery agent ended on March 31, 2015, EC is no longer involved in managing the domestic ETV Program. However, the Department’s international activities in the development of the ETV ISO standard continue.

By comparison, a less comprehensive evaluation of the CETACs was conducted because this activity ended in 2013-14 with the discontinuation of Environment Canada funding to the CETACs. An overview of the activities included in the technology analysis and assessment function, ETV Program and CETACs is provided below.

Technology Analysis and Assessment

This activity is designed to deliver comprehensive technology analysis (e.g., in briefing notes, presentation decks) and formal technology assessment reports mainly for use within Environment Canada.Footnote 3These analyses and assessments are based on an integration of the full body of scientific literature on a given technology.

Technology analysis and assessment is provided mainly to internal Environment Canada user groups, for example:

  • Environment Canada senior management (e.g., ADM Science and Technology (S&T) Branch, Deputy Minister);
  • Environment Canada user groups who are developing:
    • policy and regulations (Strategic Policy Branch, Environmental Stewardship Branch), and
    • Canada's position on and input to international clean technology partnerships (International Affairs Branch);
  • other Environment Canada user groups involved in related collaboration (e.g., Shale Gas Science-Policy Integration Pilot, Hydrogen Fuel Cells).

Some of the analysis and assessment work supports Environment Canada's participation in interdepartmental governance bodies overseeing horizontal initiatives (e.g., Assistant Deputy Minister (ADM) Coordinating Committee for the Genomics Research and Development Initiative (GRDI), the Interdepartmental ADM Energy Science and Technology (S&T) Committee),Footnote 4 ,Footnote 5 and some work supports interdepartmental clean technology committees (i.e., not related to specific programs), such as the ADM Wind Energy Hub Committee.Footnote 6

Technology assessment reports were sometimes developed in consultation with external stakeholders such as non-governmental organizations (NGOs) and industry associations.Footnote 7Once completed, some final assessment reports were shared with external stakeholders. Work on producing assessment reports on existing technologies ended in 2012. The program currently focuses on conducting analysis of the potential environmental impacts of emerging and new technologies.

ETV Program

Environment Canada established the Canadian ETV Program in 1997 to support the implementation of innovative environmental technologies in Canada in areas intended to help the Department meet its environmental and regulatory priorities through the provision of: Footnote 8

  • a reliable assessment process for verifying the environmental performance claims associated with technologies and technological processes;
  • a mechanism for third-party verification of environmental performance claims of technology to facilitate successful technology commercialization; and
  • assurance that environmental performance claims are valid, credible and supported by high-quality, independent test data and information.

The ETV Program was designed to promote the uptake of environmental technologies in the marketplace through performance verification and development of specific technology protocols (i.e., sector-based performance criteria). The performance verification component involves working closely with technology innovators and qualified testing organizations. Activities under the ETV umbrella also include ensuring the Canadian approach is well-coordinated with and, where applicable, harmonized with those of government and international organizations, within and outside Canada.

In managing the Canadian ETV Program, Environment Canada staff deliver activities such as: overseeing the third-party delivery agent Footnote 9that delivers the independent, technical component of the program; developing partnerships with provinces (e.g., Quebec) and municipalities to incorporate ETV into their respective management and decision-making process (procurement, legislation, funding programs, etc.); and ensuring that available environmental performance data (specifically, emissions reductions of greenhouse gases (GHGs) and air pollutants associated with ETV-verified technologies) are submitted to the annual departmental Performance Management Framework (PMF). International activities include leading the development of an international ETV standard under the International Organization for Standardization (ISO) by collaborating with the Standards Council of Canada (SCC), co-chairing the International Working Group (IWG) on ETV, participating in co-verification projects with the IWG members (e.g., the European Union, the Philippines and the Republic of Korea), and continuing bilateral cooperation with other countries, such as Japan and China.

Canadian Environmental Technology Advancement Centres (CETACs)

In 1994, Environment Canada created three CETACs as private, not-for-profit organizations to help small- and medium-sized enterprises (SMEs) advance innovative environmental technologies that address the Government of Canada’s environmental priorities. Environment Canada provided funding to the CETACs until 2014. Since 2014, the CETACs have restructured their operations and continue to operate independently of EC funding.

From 1994 to 2014, the three centres assisted an average of 300 SMEs every year. They operated in partnership with provincial governments, industry associations and the private sector, and are regionally structured to include:

While funded by Environment Canada, the CETACs assisted environmental enterprises in increasing the uptake of environmental technologies by providing a wide range of services, including assistance in accessing investment capital, general business development counselling, demonstration projects, technical services, market analysis, and strategic advisory and mentoring services. In the past, the CETACs have assisted companies in obtaining financing and, further down the road, in developing strategies to use these finances to move the project to the next stage of uptake.Footnote 13As noted above, Environment Canada’s annual contribution of $1.2 million to the CETACs ended in 2013-14, but the three centres continue to operate with somewhat different mandates.Footnote 14

2.2    Management and Governance

At Environment Canada, the management of the Environmental Technology sub-program 3.2.3 is provided by the Science Management Division and S&T Policy Division of the S&T Strategies Directorate, S&T Branch. Management and administration of the three areas of activity is as follows:

  • Technology analysis and assessment: Responsibility for the technology analysis function lies with the Director, S&T Policy Division.
  • ETV Program: Responsibility for the overall management and administration of the Canadian ETV Program rested with the Director, Science Management Division, until March 2015, when EC’s contribution agreement with the delivery agent ended. The technical component of ETV is delivered externally by Globe Performance Solutions.
  • CETACs: Responsibility for the overall management and administration of CETACs rests with the three centres. When previously funded by EC, the Director, Science Management Division, was responsible for oversight of the CETACs.

2.3    Resource Allocation

Sub-program 3.2.3 expenditures for 2009-10 to 2013-14 are presented in Table 1.
Note that sub-program 3.2.3 was impacted by reductions attributable to the Budget 2012 decisions. Changes included:

  • streamlining of oversight of S&T programs and technology analysis in support of programs under 3.2.3, including a reduction of three FTEs, starting in 2013-14; 
  • the elimination of Environment Canada’s support of $1.2 million annually to the CETACs ($400,000 to each centre) at the end of 2013-14; and
  • the cancellation of the Soil and Groundwater Remediation Project (SGRP), in which EC played a lead role, by NRCan at the end of 2013-14.
Table 1: Environmental Technology Sub-Program Expenditures for 2009-10 to 2013-14
S&T Branch 2009-10 2010-11 2011-12 2012-13 2013-14 Total
FTEs 15.4 18.5 13.7 10.2 12.7 -
Salaries $1,363,463 $1,510,822 $1,222,558 $1,133,196 $1,098,792 $6,328,831
O&M $1,160,819 $1,055,105 $1,102,770 $967,157 $1,489,452 $5,775,303
Gs&Cs $1,400,000 $1,400,000 $1,400,000 $1,500,000 $1,500,000 $7,200,000
Total Costs $3,924,282 $3,965,927 $3,725,328 $3,600,353 $4,088,244 $19,304,134

Notes:
1. FTE information extracted from EC’s Salary Management System (SMS). Note that in 2013-14, the SMS indicates a total of 52.7 FTEs for S&T Branch. However, program management estimates that approximately 40 of these FTEs worked in the broader S&T Strategies Directorate, not directly on sub-program 3.2.3. Therefore, 12.7 FTEs are reported for 2013-14.
2. Other data from EC’s financial system, as provided by Finance Branch, July 25, 2014.
3. Program management estimates that total direct costs (salaries and O&M) were $1,630,000 for the ETV Program, $210,000 for the CETACs, and $1,802,000 for technology analysis and assessment. The remaining direct costs ($8,462,134) were for the sub-program’s management and oversight of six federal technology programs led by other federal departments and agencies.
4. Gs&Cs indicate EC’s contribution funding to the ETV Program ($1,200,000) and CETACs ($6,000,000).

2.4    Intended Outcomes

The intended outcomes for each of the three areas assessed in this evaluation were identified in consultation with program management, as logic models for these areas of activity had not been developed. These intended outcomes are as follows:

ETV Program

  • Increased vendor credibility and buyer confidence in Canadian environmental technologies (by providing the marketplace with assurance that environmental performance claims are valid, credible and supported by high quality, independent test data and information)
  • Increased uptake of innovative environmental technologies in Canada in areas that help EC meet its environmental and regulatory priorities

CETACs

  • Improved knowledge and skills among SMEs for the uptake of environmental technologies
  • Increased uptake of environmental technologies developed by SMEs
  • Reduced emissions from implementation of environmental technologies supported by the CETACs

Technology Analysis and Assessment

  • Increased understanding of the environmental impacts of new and emerging technologies by Environment Canada senior management
  • Improved decision-making by senior management, departmental policy and regulatory user groups, and interdepartmental governance bodies

3.0    Evaluation Design

3.1    Purpose and Scope

The evaluation of Environmental Technology is part of Environment Canada’s 2012 Risk-based Audit and Evaluation Plan, which was approved by the Deputy Minister, and covers the five-year timeframe from 2009-10 to 2013-14.

The purpose of the evaluation is to assess the relevance and performance (including effectiveness, efficiency, and economy) of the Environmental Technology sub-program.  The evaluation focused on Environment Canada-led components, including the related G&C agreements, which have not been recently evaluated. These include:

  • general technology analysis and assessment activities;
  • the Environmental Technology Verification (ETV) program, including Gs&Cs of $300K per yearFootnote 15 funded under Umbrella Contributions to Support Climate Change and Clean Air; and
  • the Canadian Environmental Technology Advancement Centres (CETACs), including Gs&Cs of $1.2 million per year ($0.4 million for each CETAC), also funded under Umbrella Contributions to Support Climate Change and Clean Air.

As noted earlier, a more in-depth assessment of the ETV Program was undertaken, including an international comparison and assessment of options and alternative processes for Canada’s ETV Program. On the other hand, only a limited evaluation of the CETACs was undertaken because Environment Canada’s annual contribution of $1.2 million to this program ended in 2013-14. The evaluation of this program area relied on existing documentation, performance data and a limited number of interviews to assess the evaluation questions.

This evaluation excluded other specific programs under the Environmental Technology sub-program (SD Tech Fund, NextGen Biofuels Fund, PERD, ecoEII, GMF and STAGE/GRDI) because their evaluations have been or will be led by other federal departments or funded organizations.

3.2    Evaluation Approach and Methodology

The methodological approachFootnote 16 consisted of a review of existing documentation and data, an international comparison of ETV programs, and a round of interviews with key informants, as described below.

Review of Documentation and Data: A primary methodological approach was a review of documentation, including federal government and departmental documents (e.g., EC Reports on Plans and Priorities and Departmental Performance Reports, Speeches from the Throne and Federal Budgets) and program documents (e.g., decks presented to senior management, policy and planning documents). In addition, G&C files for ETV and the CETACs (e.g., annual reports submitted to EC), reports on interviews with ETV clients and on surveys of CETAC clients, selected literature and available performance data were reviewed. The document review provided some evidence on all of the evaluation questions. For each evaluation question, the evidence from each relevant document was summarized in a template and a summary was then prepared for that question.

International Comparison of ETV Program: An international comparison and assessment of options for the ETV Program was conducted, including a document and literature review, as well as a small number of key informant interviews. This component of the evaluation included: (1) an overall comparison of the outcomes of the Canadian ETV Program with ETV programs in the European Union, the United States, Japan, the Republic of Korea and the Philippines, based on material published on the relevant organizations’ websites, and (2) an assessment of options or alternative processes for the Canadian ETV Program. The international comparison provided supplementary evidence for the following evaluation questions: continued need, appropriateness of program design, efficiency of activities, lessons learned and achievement of intended outcomes.

Key Informant Interviews: Interviews were conducted to obtain informed opinions and observations on the evaluation questions, particularly those for which there was insufficient documentary evidence. Interview guides consisting of open-ended questions were utilized for these interviews, and a customized guide was developed for each major type of key informant. The interviews were conducted in-person or by telephone, and were an average of 60 to 90 minutes in duration.

A listing of knowledgeable interview candidates was developed in collaboration with program representatives and all of these candidates were invited to participate (though some were unavailable or declined). A total of 25 key informants were interviewed from the following groups:

  • Management of S&T Strategies Directorate and Science Management Division (n=6);
  • Internal partners and stakeholders within Environment Canada - EC researchers receiving program funds, S&T Branch, Strategic Policy Branch and International Affairs Branch (n=7);
  • Federal partners and stakeholders - NRCan and NRC (n=4);
  • External organizations delivering programs - Globe Performance Solutions (ETV) and management of the three CETACs (n=5); and
  • External partners and stakeholders of ETV - provincial governments and NGOs (n=3).

Key informant interviews yielded important information to help address all of the evaluation questions. For each evaluation question, the interview findings for each relevant key informant group were summarized in a template.   

3.3    Limitations

  • Reliance on qualitative evidence from a limited number of interviews: One line of evidence in this evaluation consists of interviews with key informants. Interview findings were most important for evaluation questions for which there was insufficient documentary evidence. Limitations of this approach included the possibility of memory deficits on the part of interviewees and/or bias in the selection of interviewees (i.e., it was not possible to interview every relevant stakeholder group). These limitations were overcome by (1) carefully selecting the sample of interviewees to ensure that all relevant perspectives were adequately covered by knowledgeable respondents from each interviewee group; (2) asking interviewees to provide evidence or concrete examples to support the views they expressed; and (3) corroborating the interview findings with reliable evidence from documentation and data where possible. For instance, the original evaluation plan included plans to solicit feedback from buyers and end users of ETV-verified technologies. However, a list of these stakeholders was unavailable at that time. To address this limitation, some interviewees familiar with industry participants were included and related documents were reviewed (e.g., surveys of CETAC clients, a report on interviews with a sample of ETV clients).
  • Incomplete performance measurement information: As the sub-program does not have a performance measurement strategy with corresponding data collection in place for each of the areas of activity being covered in this evaluation, the information on the achievement of outcomes was incomplete. To the extent possible, gaps in performance information were filled by the available departmental PMF data and by evidence obtained in the document review and interviews.

4.0    Findings

This section presents the findings of this evaluation by evaluation issue (relevance and performance) and by the related evaluation questions.

For each evaluation question, a rating is provided based on a judgment of the evaluation findings. The rating statements and their significance are outlined below in Table2. A summary of the ratings for each evaluation question is presented in Annex 1.

Table 2: Definitions of Standard Rating Statements
Statement Definition
Acceptable The program has demonstrated that it has met the expectations with respect to the issue area.
Opportunity for improvement The program has demonstrated that it has made adequate progress to meet the expectations with respect to the issue area, but continued improvement can still be made.
Attention required The program has not demonstrated that it has made adequate progress to meet the expectations with respect to the issue area and attention is needed on a priority basis.
Not applicable There is no expectation that the program would have addressed the evaluation issue.
Unable to assess Insufficient evidence is available to support a rating.

The reader should note that ratings are assigned individually by evaluation issue and that overall conclusions of the program’s relevance or performance will not necessarily be consistent with every individual issue rating.

4.1    Relevance

4.1.1    Continued Need

Evaluation Issue: Relevance Rating

1. Is there a continued need for the following areas of activity as part of this sub-program?

  • ETV Program
  • CETACs
  • Provision of technology analysis and assessment
Acceptable

There is a continuing need to address a range of environmental issues targeted by the sub-program. In addition, there is a need to encourage and support innovation in environmental technologies, to promote the uptake of these technologies among consumers, and to provide EC’s senior management with analysis of the environmental impacts of technologies to support policy and regulatory development.

  • There is a continuing environmental need in each of the areas within the scopeFootnote 17of the Environmental Technology sub-program, including the following:
    • Air quality: Air pollution has been linked to a variety of adverse health outcomes,Footnote 18 is carcinogenic,Footnote 19costs the Canadian economy billions of dollars each year due to premature deaths (approximately 21,000 Canadians annually), hospital admissions, emergency room visits, and absenteeism,Footnote 20and contributes to adverse environmental effects, such as ecosystem degradation, impacts on wildlife habitat and food from pollutants such as mercury, and impacts on vegetation from air pollutants such as ground-level ozone and sulphur dioxide.Footnote 21 
    • GHG emissions and climate change: Climate change related to the release of GHGs is warming the atmosphere and oceans, having the potential to affect the natural functioning of ecosystems, as well as weather conditions, biodiversity, hydrological systems and coastal infrastructure.Footnote 22
    • Water quality: Freshwater aquatic ecosystems are degraded and under pressure due to factors such as urban development and agricultural and industrial activities,Footnote 23 eutrophication, degradation of wetlands, and lower than average water levels associated with climate change.Footnote 24
    • Chemical substances: Chemical substances negatively impact human healthFootnote 25 and the environmentFootnote 26and increase future costs associated with water treatment, clean-up of contaminated sites, and treatment of illnesses related to chemical exposure.Footnote 27
    • Waste: Risks to the environment and human health stem from a myriad sources of human generated waste, including water pollution from a range of sectors (e.g., forestry, mining and agriculture),Footnote 28,Footnote 29, Footnote 30 wastewater effluent from wastewater systems,Footnote 31 and hazardous solid wastes.Footnote 32,Footnote 33
    • Biodiversity: Aglobal decline of biodiversity (i.e., the variety of species of plants, animals and other organisms) is recognized as a serious environmental issueFootnote 34 in view of biodiversity’s importance to human health, prosperity, security and well-being,Footnote 35and its contribution to essential goods and services that flow from healthy and diverse natural systems.Footnote 36, Footnote 37
  • An expert assessment of Canada’s performance in S&T and innovation concludes that global environmental trends require that the private sector embrace innovation-focused business strategies to compete and surviveFootnote 38 and that market forces alone, such as domestic competition, cannot be relied on to consistently produce the best outcomes in environmental science innovation. Government leadership is often required to stimulate and encourage innovation to achieve the aspirational goals that firms cannot pursue on their own (e.g., fuel efficiency standards, emission standards, carbon pricing).Footnote 39
  • Key informants consistently indicated that there is a continuing need for a process to verify the performance claims of innovative environmental technologies in order to increase confidence among potential buyers and thereby promote the uptake of these technologies, driven by factors such as Canada’s commitments to reduce GHGs and minimize the environmental impacts of pollutants. In a similar vein, a recent studyFootnote 40 suggests that the main driver for companies to pursue ETV verification is the perceived benefit in terms of greater credibility or increased marketability of their product. More generally, the ETV international comparison provides evidence of need and public support for this type of program in a number of jurisdictions.Footnote 41 It is important that evidence of a need for intervention in these areas not be construed as evidence of a need for the specific ETV Program model. That is, the evaluation does not draw conclusions on the continuing need for this specific program as a means of promoting uptake of environmental technologies.
  • Although the CETACs are no longer funded by EC, key informants perceive there is a need to support and advise environmental technology companies, particularly SMEs, that will continue as a result of ongoing concerns about environmental issues and demand for environmental solutions that are often driven by regulatory requirements. Other programs, such as the NRC’s Industrial Research Assistance Program, currently exist to help address these needs.
  • Evidence from interviews and documents indicates that technology analysis and assessment is needed to provide EC’s senior management with well-reasoned, horizontal assessments of the environmental impacts of technologies. The exact nature of the need varies according to the questions and issues requiring policy and/or regulatory responses from EC.Footnote 42, Footnote 43

4.1.2    Alignment with Government Priorities

Evaluation Issue: Relevance Rating

2. Are the following areas of activity aligned with federal government priorities?

  • ETV Program
  • Provision of technology analysis and assessment
Acceptable

The Environmental Technology sub-program is aligned with Environment Canada’s strategic outcomes and priorities related to environmental protection and the development/assessment of technologies to support the Department’s scientific work and regulations. It is also compatible with federal government priorities related to technological innovation.

  • Most interviewees indicated that the work of the Environmental Technology sub-program in supporting innovative environmental technologies is aligned with EC strategic outcomes related to the conservation and restoration of Canada’s natural environment for present and future generations and the minimization of threats to Canadians and their environment from pollution. The work of the sub-program is also generally viewed as supportive of federal priorities concerning responsible resource development and reduction of the impacts of environmental contamination.
  • With respect to ETV, internal program documents Footnote 44, Footnote 45 indicate that the work is closely aligned with Science and Technology Branch priorities to strengthen ties with the global knowledge economy and improve the means to measure and report on the impacts of federal S&T improvements.
  • The 2013 Speech from the Throne makes multiple references to the federal government’s role in promoting “greater commercialization of research and development” and the Government’s “leadership in science and technology [to help] Canadian business remain competitive.”Footnote 46 
  • The activities of the Environmental Technology sub-program are also compatible with the federal S&T Strategy, which highlights “a commitment to keep science, technology, and innovation at the forefront of government policy for years to come,”Footnote 47 and with Environment Canada’s Science Strategy 2014-2019, which acknowledges that science includes the development of technologies essential to the Department’s scientific work, assessment of technologies, and development of tools to support regulations.Footnote 48  

4.1.3    Consistency with Federal Roles and Responsibilities

Evaluation Issue: Relevance Rating
3. Are the areas of activity consistent with federal roles and responsibilities? Acceptable

The Environmental Technology sub-program is consistent with federal and departmental roles and responsibilities related to protecting the environment and supporting science, technology and innovation.

  • Interviewees consistently affirmed that all activities of the Environmental Technology sub-program are aligned with federal roles and responsibilities. The Department of the Environment Act clearly establishes the federal role in achieving and maintaining a clean, safe and sustainable environment. A common theme in key informants’ comments is that, in order to function effectively as a science-based regulator focused on environmental protection, EC relies on technology assessment work to determine if regulatory and non-regulatory requirements for environmental performance can be met or exceeded by the best available technologies.
  • Budget 2014 confirms the consistency of the Environmental Technology sub-program with federal roles and responsibilities. The Budget refers to the Government’s “important role” in Canada’s science, technology and innovation system, including innovative activities in the private sector and federal support for business needs. EC’s 2012-13 Departmental Performance Report (DPR) and 2012-13 Report on Plans and Priorities (RPP) further assert the federal role and responsibility in the environmental technology field by referring to the technological innovation role of EC, with specific mention of the ETV Program and CETACs.
  • With respect to the ETV Program, several interviewees noted that federal involvement helps to ensure national consistency in the program’s work in facilitating the acceptance and adoption of technologies that offer superior environmental performance.

4.2    Performance

4.2.1   Efficiency and Economy

Evaluation Issue: Performance - Economy and Efficiency Rating

4. Is the design appropriate for achieving intended outcomes of the following areas of activity?

  • ETV Program
  • Provision of technology analysis and assessment
Acceptable

The evidence indicates that the design of the ETV Program is appropriate, but some key informants suggest that the level of EC funding for the delivery agent ($300K annually) limits the rate at which awareness of and demand for ETV verification by both buyers and suppliers can grow. With respect to technology analysis and assessment, as a result of resource constraints in recent years (e.g.,  Budget 2012 reductions), the division has shifted its approach away from assessing existing technologies to conducting analysis of the potential environmental impacts of emerging and new technologies.

  • In interviews, program managers indicated that resources allocated to the divisions have been reduced and streamlined in recent years, including reductions resulting from Budget 2012 decisions (detailed in section 2.3). As a result, the divisions’ activities and methods have been revised and focused through careful priority setting and selection of activities. Most notably, the sub-program has shifted from promoting technology development and diffusion to promoting environmental performance, given EC’s role as a regulator and not a developer or promoter of technologies for commercial use. The S&T Policy Division has also changed its approach to technology analysis and assessment away from conducting quantitative assessments of the impacts of environmental technologies to identifying emerging technology issues that may have significant environmental implications and to providing EC’s senior management with timely and thorough analysis of the potential environmental impacts of large-scale implementation of emerging and new technologies (e.g., hydraulic fracturing in shale gas development).
  • With respect to the ETV Program, a review of program documents detailing the steps ETV verification applicants must go through to obtain a license and the activities of third-party verification agents suggests that the program design is generally appropriate for achieving the intended outcomes. Most interviewees able to comment agree that the design and delivery of the program is appropriate, but that the level of EC funding for the delivery agent ($300K annually)Footnote 49 limits the agent’s ability to engage in outreach and promotions targeting prospective applicants and buyers of verified technologies. This, in turn, limits the rate at which awareness of and demand for ETV verification by both buyers and suppliers can grow. Supporting these findings, ETV applicants expressed satisfaction with the key stages of the program (i.e., first contact with delivery agent, screening application, technology verification, and renewal of verification), and many agreed that better marketing and communications about the program would help to increase its profile.Footnote 50Achievement of ETV’s intended outcomes (i.e., increased vendor credibility, buyer confidence and uptake of innovative technologies) is widely perceived as being dependent on continuing growth in awareness and demand for ETV verification in key sectors (e.g., water and wastewater management, mining), finalization and deployment of the ETV ISO standard, and recognition of the value of ETV and ISO in international markets for environmental technologies.
  • Information on the design and delivery structures for international ETV programs suggests that the Canadian approach to the verification of environmental technologies is recognized as proven and sound. 
    • Canada’s ETV verification process is one of two founding approaches to the verification of innovative environmental technologies, the other being the US EPA ETV program. Both were introduced in the 1990s and both models have been applied in the design of other jurisdictions’ ETV programs. Verification of environmental technologies is voluntary in four of the five international jurisdictions examined, as it is in Canada, with the Korean ETV program being the exception.
    • Compared to Korea, a challenge for Canada is the role of the provinces in environmental and resource regulation, which means EC is not in a position to mandate a role for ETV in areas that fall under provincial jurisdiction. EC has, however, facilitated a reciprocity agreement between Globe Performance Solutions and the Bureau de normalisation du Québec (BNQ) regarding the harmonized use of verification results for wastewater and drinking water treatment technologies.
    • International recognition of verifications conducted by national and the EU’s supra-national ETV programs is widely seen to be desirable and necessary for the long-term success of ETV among both vendors and buyers of innovative environmental technologies. In this regard, Canada’s participation in the ETV IWG and its leadership role in the development of the ETV ISO standard are certainly appropriate in terms of facilitating wider recognition and acceptance of verified technologies.
Evaluation Issue: Performance - Economy and Efficiency Rating
5. To what extent is the governance structure clear, appropriate and efficient for achieving expected results? Acceptable

The evaluation evidence indicates that governance structures for the areas of program activity examined in this evaluation are generally clear, appropriate and efficient. As described in sections 2.1 and 2.2, management of sub-program 3.2.3 is the responsibility of the Science Management Division (for the ETV Program and CETACs) and S&T Policy Division (for technology analysis).

  • General management and governance of the Science Management Division determines the priorities and level of effort for key activities related to the ETV Program. Governance of ETV delivery is defined by the contribution agreement with the delivery agent, which specifies the performance reporting requirements and the extent of EC’s oversight role. Documents indicate that there have been a number of communications to senior management (ADM, DM) and the Minister on the state of the ETV Program, areas in need of improvement, and verifications granted. There is evidence of clear lines of accountability, and senior management appears to be well informed. Based on the information provided, the governance structure for the ETV Program, within both EC and the third-party delivery agent, is clear and appropriate for achieving the program’s intended outcomes. Key informants did not identify significant issues with the functioning of the governance arrangement.
  • The three CETACs are independent, non-profit corporations governed by their respective boards of directors. Detailed information on the CETACs’ governance structure, board member selection and compensation, or client feedback on the centres’ governance was unavailable in the documents provided. Accountabilities and performance reporting requirements are clearly specified in the contribution agreements with EC.
  • Governance for the technology analysis and assessment activity is a function of the S&T Policy Division’s accountabilities to the DG and ADM and is monitored against the objectives set out in the Director’s and DG’s work plans. As such, the management of this function is quite straightforward and considered to be appropriate and efficient.
Evaluation Issue: Performance - Economy and Efficiency Rating

6. Are activities being undertaken and products being delivered at the lowest possible cost? How could the efficiency of the activities be improved? Are there alternative, more economical, ways of delivering the outputs of the following areas of activity?

  • ETV Program
  • CETACs
  • Provision of technology analysis and assessment
Unable to Assess
  • a. ETV Program

The available evidence suggests that current ETV activities are being delivered as intended and in a reasonably efficient manner. EC provides a relatively modest annual contribution of $300K to the delivery agent and, unlike ETV programs in some other jurisdictions, the Canadian approach does not subsidize testing and verification activities (the costs of which are borne by the technology vendors). However, in the absence of detailed data on EC’s expenditures and results for domestic and international work on the ETV Program, it is not possible to draw a conclusion on the efficiency of program delivery.

  • Environment Canada provides Globe Performance Solutions with annual funding of $300K to deliver the Canadian ETV Program. This funding is intended to cover the costs of coordinating performance verification activities, establishing/maintaining a consortium of testing and verification organizations, and marketing activities to increase awareness and uptake of ETV verification among buyers/users. The costs of verification, as well as the costs of independently generated data to support verification applications, are intended to be borne by the technology vendors. The ETV international comparison indicates that funding and pricing arrangements in other jurisdictions also feature combinations of public funding and user pay arrangements (e.g., fees for testing and/or verification work).
  • An analysis of the activities reported in annual reports indicates that the program has, for the most part, been delivered as intended and in accordance with the contribution agreement. Key activities and outputs are summarized in Table 3. Globe Performance Solutions has delivered and reported on the activities specified in its contribution agreement (2011-14), with two minor exceptions (i.e., details were not reported on the recovery of verification costs from applicants or on the development of environmental indicators for verified technologies, which is still in progress).
  • Most interviewees believe the current ETV activities are being delivered efficiently and at a relatively low cost. They noted that the approach to ETV in Canada is very lean, relying on voluntary participation by vendors and buyers, and does not subsidize testing and verification work, unlike ETV programs in some other jurisdictions (e.g., Korea and Japan). The Science Management Division’s international ETV activities are also considered to be very lean. The introduction of the ETV ISO standard is expected to be crucial to the success of the ETV Program because this will give the program international recognition and could facilitate the development of international markets for Canadian technologies. Interviewees feel that the ETV ISO standard will require continued support from EC until it is well established.
Table 3: ETV Activities and Outputs 2009-10 to 2013-14
Time Period (Delivery Agent) Inquiries Inquiries Moved to Pre-screening Application or Requested More Discussion and/or Review of Existing Test Plan and Data  Developed and/or Implemented Test Plan or Verification Plan Verifications Completed Renewals Completed Number of Canadian ETV Licenses Held
March 2009 to March 2012 (OCETA/BLOOM) 129 27 19 6 11 20 licenses held by 14 companies
October 2012 to April 2014 (Globe) 99 20 16 2 5 19 licenses held by 15 companies
Total 228 47 35 8 16 -

Source: ETV annual reports submitted to EC by delivery agents.

  • b. CETACs

The CETACs were seen by key informants to have provided a cost-effective and efficient way for EC to support efforts to increase the uptake of innovative environmental technologies and to leverage the funding provided. An analysis of financial information indicates that, for the five-year evaluation timeframe, EC contributed 32% of funding for the three CETACs, while 68% came from other sources (i.e., other federal departments, other levels of government and the private sector). The administrative ratio for EC’s work on the CETACs is low (0.04), which suggests efficient delivery and oversight of the contribution agreements.

  • As with the ETV Program, contribution agreements with EC specified the activities required of the three CETACs. Each contribution agreement was unique but included similar reporting requirements. An analysis of annual reports indicates that the activities of the centres had, for the most part, been delivered as intended and in accordance with their agreements. Some minor exceptions were as follows:
    • CETAC-West did not appear to undertake extensive efforts to pursue relations with international organizations or to engage Canadian SMEs in international opportunities.
    • BLOOM did not address the development and implementation of relevant indicators for measuring environmental impacts of technologies.
    • The reports for Enviro-accès were unclear with respect to the delivery of some activities. While the numbers of clients, SMEs and services were provided, the nature of the services and types of SMEs were not indicated.
  • The CETACs were seen by key informants to have provided a cost-effective and efficient way for EC to support efforts to increase the uptake of innovative environmental technologies and obtain leverage on the funding provided. This observation is confirmed by program documents which indicate that the centres’ involvement with SMEs leveraged considerable other funding that helped move many entrepreneurial ideas along the continuum from technology development to commercialization. An analysis of the funding sources for the CETACs, as specified in their annual reports to EC over the five-year evaluation timeframe, indicates that EC contributed 32% of funding for the three CETACs, while 68% of financial contributions came from other sources (i.e., other federal departments, other levels of government and the private sector). Similarly, a studyFootnote 51 focused on CETAC-West concluded that this centre demonstrated a strong leveraging effect of public funding to generate private investment in SMEs, stimulate R&D spending by SME clients, generate new employment and reduce business failures, thereby suggesting good value for public money.
  • Over the evaluation timeframe, the administrative ratio, which is calculated as the ratio of direct program operational costs (salaries and O&M) to contribution funding disbursed, is an average of 0.04, which is low relative to charitable organizationsFootnote 52and other EC grants and contributions programs.Footnote 53  This low ratio suggests efficient delivery and oversight of EC’s contribution agreements with the CETACs.  
  • Given the need for Environment Canada to contribute to the government’s reductions associated with Budget 2012 decisions, which coincided with the latter part of the evaluation timeframe, EC senior management concluded that the Department’s $1.2 million in annual funding to the centres should cease at the end of the 2013-14 fiscal year. The rationale for this decision included the facts that the three CETACs were originally designed to become self-sustaining and that the federal government already provides similar business development services through other programs, such as the NRC’s Industrial Research Assistance Program, the Business Development Bank of Canada and regional development agencies. The three CETACs continue to operate with slightly modified mandates without EC funding.
  • c. Technology Analysis and Assessment

The recently redesigned technology analysis and assessment function (which now focuses on analysis of potential environmental impacts of emerging and new technologies) is considered by the sub-program’s managers to be an efficient approach aligned to current resources. In response to Budget 2012 decisions, technology analysis/assessment and oversight of technology programs were streamlined beginning in 2013-14. As detailed data on EC’s expenditures for technology analysis and assessment are unavailable, however, it is not possible to make a conclusion on the efficiency of this area of activity.

  • The technology analysis and assessment function has undergone an extensive review and reorganization of its activities and resources in recent years. Most notably, the focus is now on conducting analysis of the potential environmental impacts of emerging and new technologies, as opposed to formal quantitative assessments of the impacts of existing technologies. This function is generally considered by managers to now be as efficient as possible given current resource levels. As noted earlier in section 2.3, as a result of Budget 2012 reductions, technology analysis and oversight of technology programs were streamlined starting in 2013-14, including a reduction of three FTEs. 
Evaluation Issue: Performance - Economy and Efficiency Rating
7. Are performance data being collected and reported? If so, is this information being used to inform senior management/decision makers? Unable to Assess

Performance data are being collected and reported in annual reports to EC from delivery agents for the ETV Program and CETACs and, in summary form, in the DPRs. Challenges have been encountered in collecting and reporting valid performance information for all outputs and outcomes of interest. This is due to methodological difficulties in collecting data and quantifying environmental benefits and to limitations on the division’s capacity to fulfil performance measurement requirements.

  • Some limited performance information for sub-program 3.2.3 has been provided in EC’s DPRs over the five-year time frame of this evaluation. All reports provide at least a basic verbal description of key program activities and accomplishments, but do not include data on specific results or environmental impacts of the program areas being assessed in this evaluation. Although some results on estimated reductions of GHGs and air pollutants are provided in summary form, they are not provided specifically for emissions reductions attributable to implementation of environmental technologies supported by the CETACs,Footnote 54 and similar data for the ETV Program are not yet available.
  • With respect to outputs, quantitative data on numbers and types of CETAC mentoring services and technology analysis conducted by the sub-program are not provided in DPRs, though a qualitative description of the latter is available in these reports. For the Canadian ETV Program, some output data are reported; DPRs indicate that in 2012-13 two new ETV protocols were finalized and in 2013-14 two verifications were completed and 10 more were underway (expected to be finalized in 2014-15).  
  • In general terms, as one EC manager observed, the sub-program could do more with respect to performance measurement but to do so would require a choice between allocating the limited resources available to program delivery or to the development and application of performance measurement requirements. Several of the interviewees also noted that it is challenging to quantify the environmental benefits to which the sub-program is intended to contribute due to methodological challenges in collecting the necessary data.
  • For the ETV Program specifically, performance reporting requirements are established by the contribution agreement with Globe Performance Solutions and transmitted in this delivery agent’s formal reports to EC. As discussed earlier for evaluation question 6, Globe Performance Solutions has for the most part reported on the activities specified in its contribution agreement (2011-14). However, information on the recovery of verification costs from applicants and the development of environmental indicatorsFootnote 55for verified technologies (still in progress) was not reported. The ability of Globe Performance Solutions to measure the environmental and economic impacts attributable to ETV-verified technologies is limited due to the fact that these impacts typically occur some time after the verification has been issued and companies may not be willing to share commercially sensitive information. In the view of key informants, currently available evidence of the achievement of ETV outcomes is largely dependent on anecdotal evidence and case studies of individual technologies that benefited from the ETV Program. 
  • The three CETACs have also reported on the activities required by their contribution agreements with EC and have provided data on the reduced emissions/environmental impacts of technologies they supported. Some details, however, such as indicators for measuring environmental impacts of technologies and, for Enviro-accès, a description of the nature of the services provided and the types of SMEs served, were missing from reports. 
  • In the view of program managers interviewed, performance measurement and reporting for the technology analysis and assessment function is difficult due to the varied and unpredictable nature of the demand for analysis and associated subject areas. There was previously a PMF indicator on the satisfaction of government decision-makers with the timeliness, credibility and relevance of technology assessments (intended to be measured by a survey), but data were not provided in 2012-13 because an assessment report planned for completion was not released.
Evaluation Issue: Performance - Economy and Efficiency Rating
8. What lessons have been learned from the areas of activity? Not Applicable

Some notable lessons identified in documentation and interviews are that the development of awareness and demand for verified technologies among both suppliers and potential buyers is key to the success of the ETV Program and that it takes longer than initially anticipated for companies to develop a market and become commercially successful in environmental technology. 

  • Some interviewees noted that it is more challenging and time-consuming to develop the market than may have been first anticipated, and while technical capacity is essential, success depends as much or more on developing awareness and demand for ETV verifications among both suppliers and potential buyers of environmental technologies, as well as public authorities that establish or influence public procurement requirements. While conferences and networks have been good methods for sharing information and raising awareness of the ETV Program, the cost and level of effort involved limits the rate of use by the delivery agent. The cost of testing was identified as a disincentive for SME participation and may limit the rate of growth in demand for ETV licenses. The need to establish international recognition and harmonization of ETV requirements and protocols was also noted.
  • The development of a commercially successful and profitable business in the field of environmental technology can be expected to take many years regardless of the strength of the technology involved.

4.2.2   Achievement of Intended Outcomes

Evaluation Issue: Performance - Effectiveness Rating

9. To what extent have intended outcomes been achieved as a result of the following areas of activity?

  • ETV Program
  • CETACs
  • Provision of technology analysis and assessment
See Below

Environmental Technology Verification (ETV) Program

There is limited evidence on the achievement of ETV Program outcomes. The consensus among the interviewees able to comment was that progress is being made but there is not yet conclusive evidence on the program’s degree of success. Data on the sales of environmental technology units in Canada (as an indicator of uptake) and the environmental impacts of ETV-verified technologies are not currently available.

  • a. Increased vendor credibility and buyer confidence in Canadian environmental technologies - Opportunity for Improvement
  • Most interviewees able to comment indicated that progress is being made on this outcome, but that it is premature to draw conclusions on the success of this program. They added that anecdotal evidence from participating vendors and the rate of ETV re-certification suggest that vendor credibility is increasing. In addition, in a recent study,Footnote 56 vendors indicated that obtaining the ETV verification has made at least some positive contribution to the success of their company. With regard to the ETV Green Mining Initiative (GMI)Footnote 57 project, it was also noted that if the current demonstration project is successful and the various players continue their involvement, there should be evidence on the achievement of the ETV outcomes in 4 to 5 years.
  • In the ETV international comparison, publicly available data indicate that there have been a total of 60 ETV-verified technologies in Canada since 1997.Footnote 58 Korea (whose mandatory program has resulted in a total of 156 verified technologies) is the only jurisdiction which has undertaken and made public an evaluation. The evaluationFootnote 59 indicated significant positive benefits in terms of commercial success and improved performance of the verified technologies (compared to alternate or replaced technologies).
  • As noted earlier, interview findings suggest that the level of EC funding for the delivery agent ($300K annually) limits the rate at which awareness of and demand for ETV verification by both buyers and suppliers can grow, and achievement of ETV’s intended outcomes is widely perceived as being dependent on continuing growth in awareness and demand for ETV verification. In addition, finalization and implementation of the ETV ISO standard is considered to be critical for the achievement of intended program outcomes given that this should help to increase vendor credibility and buyer confidence in the technologies and to facilitate access to international markets by Canadian technology vendors. Implementation of a more demand-driven approach--one that seeks to identify and match sectors and applications with the most to gain from ETV-verified technologies to prospective technology vendors--is also believed to be contributing to the rate of progress.
  • b. Increased uptake of innovative environmental technologies in Canada in areas that help EC meet its environmental and regulatory priorities - Unable to Assess
  • Data on the sales of environmental technology units in Canada, as an indicator of uptake, are not currently available, which limits the ability of the program to estimate environmental impacts (e.g., reductions in GHG emissions and environmental contaminants, increased energy efficiency). This is largely a function of the time between a vendor obtaining verification, the subsequent rate of market penetration of the technology involved, and consequent accumulation of environmental benefits compared to prior or alternate technologies. EC program managers indicated that a system to track vendors’ sales and a methodology to estimate the associated environmental impacts of ETV-verified technologies are desirable. The question of how such a system would be funded needs to be addressed.   
  • The consensus among the interviewees who commented on the performance of the ETV Program was that progress is being achieved but that there is not yet conclusive evidence on whether the program is truly successful. As noted, the sub-program faces challenges in quantitatively measuring the environmental impacts of ETV-verified technologies. The 2013-14 DPR indicated that a methodology is currently under development to measure reduced emissions from implementation of environmental technologies receiving certification under the Canadian ETV Program.

Canadian Environmental Technology Advancement Centres (CETACs)

The available evidence suggests that the CETACs have been successful in improving SMEs’ knowledge and skills and that the implementation of technologies supported by the centres has resulted in reduced emissions and other beneficial environmental impacts. However, the division’s managers expressed some concern about the validity of the data provided by the centres on environmental impacts due to the difficulties in measuring uptake of clients’ technologies and the associated estimated environmental benefits.

  • c. Improved knowledge and skills among SMEs for the uptake of environmental technologies - Acceptable
  • In the key informant interviews, representatives of the CETACs stated that their organizations have been successful in helping SMEs to strengthen their business development capabilities and increase uptake of innovative environmental technologies. The division’s managers agreed that the CETACs have been successful in achieving this outcome.
  • Program documents confirm that there has been an improvement in the knowledge and skills among SMEs regarding the commercialization of environmental technologies. A 2012 client satisfaction survey measuring the value of client interactions with the centresFootnote 60 indicated that the areas of greatest perceived value were connecting to business support networks (average rating of 5.1 on a 7-point scale), technical skills development (4.7) and strategic planning/development (4.6) for BLOOM clients; and environmental auditing (5.4), technical skills development (4.1) and access to financing (4.0) for Enviro-accès clients. A 2013 survey (which also incorporated earlier data from 2010)Footnote 61 indicated that CETAC-West clients saw the most value in coaching and mentoring (average rating of 6.0 on a 7-point scale), business skills development (5.8), connecting to business support networks (5.6) and strategy development (5.6).
  • Project activity reporting also supports the finding that the centres delivered training and mentoring services to help improve the knowledge and skills of their entrepreneur clients.  
  • d. Increased uptake of environmental technologies developed by SMEs -
    Acceptable
  • There is evidence of the uptake of environmental technologies developed by the SMEs in the centres’ client surveys, which indicate an increase in revenue for the companies involved, suggesting that they are selling their product. The 2012 client surveyFootnote 62estimated that the change in revenue from the point of starting to work with the centre to the present was a net increase of $41.4 million for BLOOM companies (n=19) and $26.3 million for Enviro-accès companies (n=21). The 2013 CETAC-West surveyFootnote 63 estimated that the change in revenue was a net increase of $380.6 million for CETAC-West companies (number of respondents not specified). Although these estimates may not be perfectly accurate due to the low participation rates by some CETAC clients in these surveys (10% of companies for BLOOM, 15% for Enviro-accès, though 50% for CETAC-West), the consistency of the findings between surveys and the size of the effect in terms of revenue generation across surveys suggest that the trend of increased revenues is generally robust.  
  • e. Reduced emissions from implementation of environmental technologies supported by CETACs - Acceptable
  • There is evidence that, when implemented, many of the technologies result in reduced emissions and other beneficial environmental impacts. The CETACs do report in some detail the total cumulative reductions from their industrial programs for a number of indicators, as summarized in Table 4 below. For example, in 2012-13, total estimated emissions reductions for the three CETACs were 16 tonnes of air pollutants and 122,792 tonnes of GHGs. In addition, for the same year, the estimated reduction in the release of toxic substances was 6,769 tonnes. However, the Science Management Division’s managers were somewhat less confident about the extent to which the intended environmental outcomes have been realized, based on the data reported by the centres. They noted that the centres’ surveys did provide evidence of some significant benefits but that data on the environmental benefits were challenging to collect and interpret and that there were inconsistencies among the centres in the methodologies used to calculate emission reductions (e.g., use of actual versus projected sales of units of technology to calculate emissions, differences in which air pollutants were included in calculations). Given these issues and the fact that these data are based on the centres’ own reporting, further work would be needed to improve the reliability and validity of these estimates.

Provision of Technology Analysis and Assessment

Analysis and assessment work to date has contributed to senior management’s understanding of environmental technology issues (e.g., issues related to biofuels, fracking and solar photovoltaic technologies). The sub-program’s managers believe that technology analysis and assessment should contribute to better informed decision-making, but due to recent changes in the S&T Policy Division’s approach, it is premature to draw conclusions on this outcome.

  • f. Increased understanding of the environmental impacts of new and emerging technologies by Environment Canada senior management -  Acceptable
  • In the view of the EC managers interviewed, work on technology analysis and assessment has advanced the Department’s understanding, as well as the partner departments’ understanding (e.g., NRCan), of issues such as hydraulic fracking in shale gas development and solar photovoltaic systems. Interviewees feel that recent changes to the division’s approach to technology analysis should position it to contribute further to increasing senior management’s understanding of the potential environmental impacts of emerging and new technologies.
  • The document review confirms that that the division has been involved in analysis/assessment work and has briefed senior management on a range of topics (e.g., the Assessment of Environmental Performance of Solar Photovoltaic Technologies (2013), Interdepartmental Shale Gas-Science Policy Integration Pilot (2012) and Wind Technology Roadmap (2009)). 
  • g. Improved decision-making by senior management, departmental policy and regulatory user groups, and interdepartmental governance bodies - Unable to Assess
  • In interviews, EC managers indicated that the division’s new approach to technology analysis may contribute to better informed decision-making due to increased understanding among senior management of the potential impacts of emerging and new technologies. However, the consensus among the sub-program’s managers is that given the recent changes to the approach to this function, it is too early to determine its overall effectiveness. Furthermore, there are no indicators in place to measure the impacts of technology analysis on management decision-making.
Table 4: Estimated Environmental Impacts of Technologies Supported by CETACs
Environmental Impact 2011-12 2012-13 2013-14
Reduced Emissions of Air Pollutants 2,142 tonnes 16 tonnes -
Reduced Emissions of GHGs 518,204 tonnes 122,792 tonnes 229,873 tonnes
Reduced Water Consumption 926,506 m3 1,043,967 m3 -
Improved Water Quality: Turbidity 400 FTU 475 FTU -
Improved Water Quality: Dissolved Solids 1,000 kg 750 kg -
Improved Water Quality: Biological Oxygen Demand - - -
Improved Water Quality:
Other
824,170 tonnes 8,964 tonnes -
Reduced Release of Toxic Substances 8,921 tonnes 6,769 tonnes 201 tonnes
Treatment of Contaminated Soil 612 tonnes 100 tonnes 54,000 tonnes
Diversion of Waste 61,539 tonnes 3,178 tonnes -
Reduced Energy Consumption: Natural Gas 2,287,527 m3 4,802,418 m3 9,908,239 m3
Reduced Energy Consumption: Electricity 52,523 MWh 11,435 MWh -
Reduced Energy Consumption: Other 452,263 L - -
Reduced Release of Ozone Depleting Substances 45 kg 45 kg -

Note: Figures are based on the data provided to EC in CETAC annual reports and represent approximations of the total reduced emissions and other environmental impacts for the three CETACs combined. For 2013-14, data were only available for two of the three CETACs (because BLOOM only provided figures for these three years combined, not separately for 2013-14). These figures should be regarded as estimates only due to inconsistencies in the methodologies used by the centres for calculating emissions reductions.

Evaluation Issue: Performance  - Effectiveness Rating
10. Have there been any unintended (positive or negative) outcomes? Acceptable

No negative unintended outcomes have resulted from the sub-program’s activities. Reported positive unintended outcomes include the use of aspects of the Canadian ETV approach in other jurisdictions.

  • The following positive unintended outcomes from the activities of the sub-program were identified by individual interviewees:
    • A number of other jurisdictions (both domestic and international) have applied the “Canadian ETV model” or elements of the Canadian approach in the design of their ETV initiatives.
    • Formation of a “Friends of CETAC” by alumni of CETAC-West to strengthen that centre’s mentoring activities and contribute to fundraising.

5.0    Conclusions

 Relevance

  • There is a continuing need to address environmental issues ultimately targeted by the areas of activity in this sub-program, such as reducing GHG emissions and minimizing the adverse environmental impacts of pollution. There is also a need to encourage and support innovation in environmental technologies and to promote the uptake of these technologies by consumers in order to help address environmental needs. In light of the lack of conclusive evidence on the achievement of intended outcomes of the Canadian ETV Program (discussed below), which has been in operation since 1997, the evaluation is unable to draw conclusions on the continuing need for this specific program as a means of promoting uptake of environmental technologies.
  • Technology analysis and assessment is needed to provide EC’s senior management with analysis of the potential environmental impacts of emerging and new technologies. Although there is a continued need for the services provided by the CETACs in supporting and advising SMEs for the commercial development and application of innovative environmental technologies, there are other government programs that can help to address that need (e.g., NRC’s Industrial Research Assistance Program).
  • The Environmental Technology sub-program is aligned with Environment Canada’s strategic outcomes and priorities related to environmental protection and the development/assessment of technologies to support the Department’s scientific work and regulations. It is also compatible with federal government priorities related to technological innovation.
  • The Environmental Technology sub-program is consistent with federal and departmental roles and responsibilities related to protecting the environment and supporting science, technology and innovation.

Efficiency and Economy

  • The design of the ETV Program is appropriate, but some key informants suggest that the level of EC funding for the delivery agent ($300K annually) limits the rate at which awareness of and demand for ETV verification by both buyers and suppliers can grow.
  • S&T Policy Division has changed its approach to technology analysis and assessment. Rather than assessing existing technologies, it now focuses on analysis of the potential environmental impacts of emerging and new technologies.
  • Management of sub-program 3.2.3 is the responsibility of the Science Management Division (for the ETV Program and CETACs) and S&T Policy Division (for technology analysis). EC’s oversight role and the responsibilities of the three CETACs and ETV delivery agent are specified in the contribution agreements. The governance structures for the three areas of program activity are generally clear, appropriate and efficient.
  • The available evidence suggests that ETV activities are being delivered as intended and in an efficient manner. For example, EC provides a relatively modest annual contribution of $300K to the delivery agent and, unlike ETV programs in some other jurisdictions, the Canadian approach does not subsidize testing and verification components of the program. It is not possible to draw conclusions on the efficiency of this program area, however, because detailed data on EC’s expenditures for domestic and international work on the ETV Program are unavailable.
  • The CETACs were seen by key informants to have provided a cost-effective and efficient way for EC to support efforts to increase the uptake of innovative environmental technologies and to leverage the funding provided. For the five-year evaluation timeframe, EC contributed 32% of funding for the three CETACs, while 68% came from other sources. The administrative ratio for EC’s work on the CETACs is low (0.04), which suggests efficient delivery and oversight of the contribution agreements.
  • The recently redesigned technology analysis and assessment function is considered by the sub-program’s managers to be an efficient approach compatible with current resources. In response to Budget 2012 decisions, technology analysis/assessment and oversight of technology programs were streamlined beginning in 2013-14. However, as detailed data on EC’s expenditures for technology analysis and assessment are unavailable, it is not possible to draw a conclusion on the efficiency of this area of activity.
  • Performance data are being collected and reported in annual reports to EC from delivery agents for the ETV Program and the CETACs and, in summary form, in the DPRs. There have been some challenges, however, in collecting and reporting valid performance information for all outputs and outcomes due to methodological difficulties in collecting data and quantifying environmental benefits, which require longer timeframes before being realized, and to limitations on the division’s capacity to fulfil performance measurement requirements.

Achievement of Intended Outcomes

  • There is limited evidence on the achievement of ETV Program outcomes. The consensus among the interviewees able to comment was that progress is being made but that there is not yet conclusive evidence on the program’s degree of success. Data on the sales of environmental technology units in Canada (as an indicator of uptake) and the associated environmental impacts of ETV-verified technologies are not currently available. Qualitative evidence suggests that the current level of resourcing for the ETV Program may limit the rate at which intended outcomes can be realized. This is supported by the fact that only 60 technologies have been ETV-verified in Canada since the program’s inception in 1997 and only 21 technologies are currently verified.
  • The CETACs were successful in improving SMEs’ knowledge and skills, and the implementation of technologies supported by the centres resulted in reduced emissions and other beneficial environmental impacts. However, there are concerns about the reliability of the data provided by the centres on environmental impacts.
  • Analysis and assessment work to date has contributed to senior management’s understanding of environmental technology issues (e.g., biofuels, fracking and solar photovoltaic technologies). In the view of the sub-program’s managers, the technology analysis function should contribute to better informed decision-making. However, due to recent changes in the S&T Policy Division’s approach, it is premature to draw conclusions on this outcome.

6.0   Recommendation and Management Response

Given the dearth of performance information on the Canadian ETV Program, the evaluation evidence indicates a need for the development and implementation of a performance measurement strategy for this program, including the estimation of environmental impacts attributable to ETV-verified technologies. However, given a recent decision by EC senior management to no longer provide contribution funding to the ETV delivery agent beyond March 2015, such a recommendation is no longer relevant for EC.

The following recommendation is directed to the Assistant Deputy Minister, Science and Technology Branch, as the senior departmental official responsible for the management of the Environmental Technology sub-program:

Recommendation 1: Develop and implement strategies to improve awareness and uptake of ETV-verified technologies. The evaluation evidence suggests that the success of the ETV process is dependent on continuing growth in awareness and demand for ETV verification in key sectors such as water/wastewater management and mining, as well as finalization and deployment of the ETV ISO standard and recognition of the value of ETV and ISO in international markets for environmental technologies. In addition, efforts to foster inclusion and/or recognition of ETV verification in public procurement policies could facilitate uptake. Development of strategies should take into account factors such as: which sectors offer the best potential for emission reductions and other environmental benefits; which activities have the best potential to generate increases in awareness and interest (e.g., pilot and demonstration projects); the level of funding and human resources necessary to undertake such activities; the rate at which the verification infrastructure should be expanded to respond to a growth in demand; and the extent to which partners and additional resources could be engaged to facilitate the expanded level of ETV activity. 

Statement of Agreement/Disagreement with the Recommendation

The ADM, Science and Technology Branch, agrees with the recommendation.

Management Action

It is understood that the success of the ETV process is closely linked with awareness and related benefits. That said, although our efforts regarding the management of the Canadian ETV Program ended as of March 31, 2015, our efforts in developing and promoting the ISO-ETV standard (ISO 14034) are expanding. It is expected that once ISO 14034 is developed and established in Canada in 2016-17, the responsibility of implementing the ETV process will be market driven.  

  1. The development of the ISO 14034 is well underway.  A Draft International Standard (DIS) was submitted to the ISO Secretariat in Geneva in March 2015. The DIS will now be translated and reviewed by all ISO countries (~ 140 countries potentially). Uptake and knowledge of ISO 14034 by stakeholders in Canada is essential for the longer term success of ETV. We will collaborate with CSA Group (formerly the Canadian Standards Association) as well as the Standards Council of Canada (SCC) to engage Canadian stakeholders to ensure awareness, usage and/or implementation of the ETV process.
  2. Continued collaboration with important programs and organizations, such as Sustainable Development Technology Canada (SDTC) and the Green Municipal Fund (GMF) to implement ETV in their evaluation processes. This action will help foster the inclusion and recognition of ETV verification in public policies.
  3. Building on a recent reciprocity agreement for verification of wastewater and drinking water technologies between the delivery agent and the Bureau de normalisation du Québec (BNQ), a pilot project will be undertaken to increase awareness and interest of ETV in this key sector.
Timeline Deliverable(s) Responsible Party
July 2015 Second year of MOU with the Standards Council of Canada. Main tasks of SCC will be to distribute the DIS to all key Canadian stakeholders and inform them about the upcoming ISO-ETV standard. SCC will also work with technical stakeholders to develop an accreditation framework for ETV. ADM S&T Branch
June 2015 Contribution agreement with CSA Group that will focus on ETV outreach ensuring awareness and knowledge of ETV in Canada. ADM S&T Branch
September 2015 Pilot project in place with BNQ to verify a wastewater or drinking water technology that will test/evaluate the process in place and increase the interest for further verifications in the sector. ADM S&T Branch

Annex 1

Summary of Findings Footnote 64
Evaluation
Question
Acceptable Opportunity for Improvement Attention Required Not Applicable Unable to Assess
Relevance:  
1. Is there a continued need for the areas of activity as part of this sub-program? Yes No No No No
2. Are the areas of activity aligned with federal government priorities? Yes No No No No
3. Are the areas of activity consistent with federal roles and responsibilities? Yes No No No No
Performance:  
4. Is the design appropriate for achieving intended outcomes of the areas of activity? Yes No No No No
5. To what extent is the governance structure clear, appropriate and efficient for achieving expected results? Yes No No No No
6. Are activities being undertaken and products being delivered at the lowest possible cost?  No No No No Yes
7. Are performance data being collected and reported? No No Yes  No No
8. What lessons have been learned from the areas of activity? No No No Yes No
9. To what extent have intended outcomes been achieved as a result of the areas of activity?
ETV Program
Increased vendor credibility and buyer confidence in Canadian environmental technologies No Yes No No No
Increased uptake of innovative environmental technologies in Canada in areas that help EC meet its environmental and regulatory priorities No No No No Yes
CETACs
Improved knowledge and skills among SMEs for the uptake of environmental technologies Yes No No No No
Increased uptake of environmental technologies developed by SMEs Yes No No No No
Reduced emissions from implementation of environmental technologies supported by the CETACs Yes No No No No
Provision of Technology Analysis and Assessment
Increased understanding of the environmental impacts of new and emerging technologies by EC senior management Yes No No No No
Improved decision-making by senior management, departmental policy and regulatory user groups, and interdepartmental governance bodies No No No No Yes
10. Have there been any unintended (positive or negative) outcomes? Yes No No No No
Report a problem or mistake on this page
Please select all that apply:

Thank you for your help!

You will not receive a reply. For enquiries, contact us.

Date modified: