Strategic performance measurement: Evaluating the effectiveness of risk management actions on toxic substances in protecting Canadians and their environment
The Government of Canada actively manages toxic substances through the Chemicals Management Plan. When a substance is found toxic to human health or the environment under the Canadian Environmental Protection Act, 1999 (CEPA), the Government takes action to manage the risk identified. While this work has been ongoing for many years, no systematic evaluation of these actions has been undertaken. The fall 2018 report of the Commissioner of the Environment and Sustainable Development (CESD) to the Parliament of Canada found that Health Canada and Environment and Climate Change Canada should do more to “assess whether they were meeting their overall objectives to reduce the risks of toxic substances to human health and the environment”.
In response to the CESD recommendation, the following Performance Measurement Evaluation Strategy (the Strategy) was developed. The Strategy sets out the approach for evaluating the effectiveness of actions taken to address risks posed by substances found toxic under CEPA. It also outlines why, how and when performance measurement evaluations for the risk management of these substances will be undertaken.
Each evaluation will compare the current state of a toxic substance with its state before risk management tools were implemented. Findings will be based on the best available information. However, evaluating how risk management strategies and tools impact human health and the environment is a lengthy process. For example, it can take decades to see a decrease in the presence of certain environmental contaminants in humans, such as lead.
Performance measurement evaluation reports will determine whether: the objectives have been met; progress can be measured but additional action is required; or, the risk management strategy or action has not achieved the intended results.
Follow-up actions will be recommended where objectives are not being met, or where linkages between actions and outcomes are not clear. Future evaluations may be recommended for some substances, while others may simply require routine monitoring.
For substances requiring performance re-evaluation, plans to address gaps and track key indicators will be developed. Follow-up actions may include undertaking new research, adjusting the objectives of risk management approaches or instruments, and/or recommending substances for additional actions, such as monitoring, compliance promotion or enforcement. The readiness and prioritization exercises will be repeated to ensure that new information and emerging concerns are considered. Information will be available on the Performance Measurement Evaluation for Toxic Substances webpage, including performance measurement evaluation reports, work plans, and details on how to provide information.
The performance measurement evaluation process will help ensure that risk management measures are applied where needed to protect Canadians and the environment from toxic substances. Through the evaluation of risk management actions, Canadians can have confidence that their health and the environment are being protected from harmful substances.
Taking action on toxic substances: How it works in Canada
Chemicals are part of everyday life, essential to our economy, our communities and our homes. While chemicals provide benefits, they may also have harmful effects if not properly managed. The Canadian Environmental Protection Act, 1999 (CEPA) is one of the most important environmental laws in Canada to assess and manage substancesFootnote 1. A guiding principle of CEPA is to prevent pollution and manage the risks from potentially harmful substances to which humans and the environment might be exposed. CEPA describes the conditions under which a substance is "toxic" as those that enter or may enter the environment at levels or conditions that:
- have or may have a harmful effect on the environment and its biological diversity;
- are or could be dangerous to the environment on which life depends; or
- are or could be dangerous to human life or health.
The Government of Canada, through the Chemicals Management Plan, assesses substances to determine if they pose a risk to human health or the environment through the risk assessment process. When a risk is identified for a substance, the risk management process begins. Risk management strategies and tools are identified and implemented to mitigate the risks. By taking actions, including the development and implementation of risk management instruments, carrying out compliance promotion activities and enforcing regulations, the Government of Canada can help reduce the exposure of Canadians and their environment to toxic substances. The performance measurement evaluation process helps to evaluate the effectiveness of risk management. Figure 1 illustrates the role of performance measurement evaluation in the chemicals management cycle.
Figure 1: Chemicals management cycle indicating how performance measurement can contribute to the ongoing protection of human health and the environment from the risks of harmful substances.
The figure depicts the chemicals management cycle and how performance measurement can ensure ongoing protection from risks to help protect human health and the environment with four integrated components linked in a circle: identifying risks, developing strategies and tools to manage the risks from toxic substances, a hub for implementation, compliance promotion and enforcement, and a hub for performance measurement evaluation. These link back to the other three components based on the outcome of the evaluation.
Risk management actions may involve using regulatory and non-regulatory tools under CEPA, the Canada Consumer Product Safety Act, the Food and Drugs Act, the Pest Control Products Act, the Fisheries Act, or other legislation. The objectives of risk management strategies or tools vary from decreasing the presence of a substance in consumer and commercial products; to minimizing the release of substances to the environment; to the complete prohibition and virtual elimination of a substance entering or being used in Canada. Tools can include, but are not limited to, regulations (e.g., regulations administered by Environment and Climate Change Canada and by Health Canada), Significant New Activity (SNAc) provisions, Pollution Prevention Planning Notices, codes of practice, release guidelines, environmental performance agreements, and outreach and engagement.
Engaging stakeholders and the public is central to managing toxic substances in Canada. The public and stakeholders are engaged when a substance is assessed and proposed to be concluded as toxic to human health or the environment. Engagement continues throughout the risk management process, including identifying risk management options and approaches, selecting tools to mitigate risks, and developing or modifying those tools. Engaging relevant stakeholders during the performance measurement evaluation process is important to ensure all relevant data has been received and considered. Providing results to Canadians on how well risk management strategies and tools have performed will enable stakeholders and the public to better understand how effectively their health and the environment are protected from toxic substances.
This Performance Measurement Evaluation Strategy outlines when and why performance measurement evaluations on the risk management of toxic substances will be undertaken and how the public and interested stakeholders can stay informed and provide information.
1. Why measure performance?
Performance measurement evaluation is an essential part of the risk management cycle because it validates whether or not risk management strategies and tools are helping to protect Canadians and their environment. A performance measurement evaluation will demonstrate the progress these risk management strategies and tools have made toward meeting their individual objectives and if changes are needed over time. Under this Strategy, work plans and timelines will be developed to measure performance. Performance measurement will be considered early in the risk management process (e.g., at the risk management scope or approach stage) to better integrate objectives or targets, and a data collection plan will be developed to enable future performance measurement evaluation. Potential environmental and human health factors impacted by climate change and emerging use patterns of substances will also be considered, when appropriate, in the context of the performance of risk management tools.
The Commissioner of the Environment and Sustainable Development (CESD) provided a report to the Parliament of Canada in fall 2018 on the management of toxic substances. It found that Health Canada and Environment and Climate Change Canada should do more to “assess whether they were meeting their overall objectives to reduce the risks of toxic substances to human health and the environment”. The report recommended that both departments “establish a long-term, systematic approach to evaluate how effective their actions are in controlling toxic substances”. In response to the CESD’s recommendation, this Performance Measurement Evaluation Strategy was developed. The Strategy sets out the approach to evaluate the effectiveness of actions taken on substances found toxic under CEPA.
Objective: To measure the performance of risk management strategies and tools on chemical substances. The results of performance measurement evaluation will help the government to apply risk management where needed for the ongoing protection to human health and the environment from the risks posed by toxic substances.
Scope: This strategy will be applied to substances that have been concluded to be harmful to human health and/or the environment and were determined through the risk assessment process to meet the CEPA definition of “toxic” and where risk management measures have been implemented.
Results from performance measurement evaluations will help demonstrate to Canadians how effectively risks from substances are being managed in Canada, where improvements can be made to risk management strategies and tools, and how the efforts to manage risks from substances help protect Canadians and their environment in the long term. It should be noted that there may be minor differences between how Health Canada and Environment and Climate Change Canada implement the Strategy, based on their specific priorities, activities, resources and processes.
2. Selecting substances for performance measurement evaluation
The performance measurement evaluation approach will focus on the substances that are deemed harmful to human health or the environment and meet the CEPA definition of a toxic substance. The selection process consists of two distinct exercises – readiness and prioritization. A substance is “ready” for performance measurement if the necessary criteria to measure performance are met. For substances that are not yet ready, the missing elements will be identified and where appropriate, plans detailing what is needed to fill the missing elements will be developed. Substances that are ready will be identified for “prioritization”. The readiness and prioritization exercises will be repeated on a cyclical basis to ensure that new information and emerging concerns are considered.
Certain toxic substances are subject to ongoing performance measurement and reporting on progress under specific programs. Additional performance measurement activities for these substances beyond those already underway may not be necessary.Footnote 2
Some specific and minimum information must be available on a substance (identified below), to evaluate the performance of the corresponding risk management strategies and tools. The readiness step helps to identify which substances can proceed to prioritization for performance measurement, which substances are not ready, and the specific information required to make those substances ready to proceed. The following criteria will help determine which substances are ready for performance measurement evaluation:
- an environmental or human health objective or target has been identified;
- risk management tools have been implemented;
- sufficient time has passed since risk management tools were implemented for changes to be observed;
- key performance indicators have been identified and data are available; and
- baseline or trend information is available to assess key performance indicators.
An environmental or human health objective states what should be achieved to address environmental or human health concerns identified during the risk assessment of a toxic substance. Setting objectives is the first step for risk management. These objectives will help decision makers determine the most appropriate ways to manage the risks from the substance, and serve as a benchmark for performance measurement evaluation.
For example, the proposed environmental objective for triclosan is to reduce concentrations of triclosan in the aquatic environment to levels below the Federal Environmental Quality Guidelines value of 0.47 µg/L.
Key performance indicator data are used to evaluate progress in meeting environmental and/or human health objectives. Available quantitative and/or qualitative performance indicators are considered for each substance. These can include, but are not limited to:
- concentrations and/or quantities in environmental media (e.g., water, sediment, air, soil) and food;
- concentrations in the human body, known as biomonitoring (e.g., in blood, urine, hair);
- concentrations in wildlife;
- concentrations in products available to consumers;
- quantities or concentrations released from sources (e.g., industrial facilities);
- number of products containing a substance currently available to consumers;
- quantity of a substance present in Canada, including in products;
- epidemiological data/disease clusters/incidence of disease;
- compliance with enforceable risk management tools; and/or
- industry participation in non-regulatory risk management tools (e.g., Codes of Practice).
2.1.1 Outcome of the readiness exercise
Substances that meet all the readiness criteria are then prioritized to determine the order in which those substances will be selected for performance measurement evaluation. Figure 2 below summarizes the readiness exercise. Readiness plans will be developed for substances that do not meet all the readiness criteria. Those plans will explain why a substance is not ready, and provide steps and timelines for achieving readiness. Readiness will be achieved easily in some cases, for example by establishing missing risk management objectives. In other cases, many years will be required to achieve readiness, such as in cases where analytical methodologies need to be developed, validated and implemented to obtain the key performance indicator data.
Figure 2: Steps needed for certain substances to move to the prioritization step.
The figure is a flow chart for certain substances to move to the prioritization step. Toxic substances undergo a readiness exercise, and, if ready, move on to prioritization. If not ready, information or data needs are addressed, including setting objectives, addressing baseline information needs, obtaining key indicator data and addressing the implementation of risk management tools. Once needs are addressed, substances are now ready for prioritization. Finally, prioritized substances undergo performance measurement evaluation and reporting.
All substances that are identified as “ready” will be prioritized to establish the order in which they will be evaluated. Prioritization identifies which ready substances should be evaluated first so that any changes to risk management efforts can be directed where they are most needed.
Prioritization is based on certain criteria including, but not limited to, and in no particular order:
- time passed since commitment to manage risks, where the longer a risk management strategy or tool has been in place, the more of a priority it may be;
- alignment with departmental activities that are planned or underway, that would inform or complement performance measurement evaluations;
- alignment with international priorities and activities (e.g., the introduction of new or modified regulations in other jurisdictions);
- completeness of datasets to enable a comprehensive performance measurement evaluation; and
- emerging concerns identified by indicator trends or findings (e.g., data indicating low compliance with a risk management tool, or increased uses or releases of a substance, may be considered a higher priority for evaluation).
Prioritizing a substance is based on the number of criteria met, which criteria are met and the extent to which they are satisfied. For example:
- a substance meeting several criteria or satisfying more fully many criteria may be prioritized over other ready substances for earlier performance measurement evaluation; or
- a substance with a very complete data set across all key performance indicators and over the applicable timeline for a risk management strategy or tool may be rated more highly than a substance with a data set covering only a few years.
Some criteria may be given more weight when prioritizing a substance for evaluation, i.e., to reflect what is important for measuring performance and what is important for protecting human health and the environment. For example, a substance that has been risk managed for a long time may not be as high a priority as a substance that has been managed for less time but where there are emerging concerns identified through indicator trends or other sources of information. In that case, a substance risk managed for less time may qualify for earlier performance measurement than one that has been risk managed for a longer period, in order to evaluate the on-going effectiveness of the risk management strategies and tools and determine if adjustments are needed.
2.2.3 Outcome of the prioritization exercise
Prioritization determines the order that substances will be evaluated, with a rationale for their assigned level of priority. A work plan will be developed and periodically updated with associated timelines for prioritized substances. Timelines will be made public and accessible through the Performance Measurement Evaluation of Toxic Substances webpage. While the number of evaluations conducted may vary year to year, one to two evaluations are expected to be conducted at a minimum each year.
3. Measuring progress in protecting Canadians and the environment: Evidence-based evaluations
The performance of risk management strategies and tools will be evaluated by assessing all relevant available information, and comparing the current state with conditions before risk management tools were implemented (known as baseline data). Findings will be based on the best available information for key performance indicators. Where baseline data do not exist, other approaches will be used, including, but not limited to, guidelines and targets. Conclusions will be based on whether risk management strategies and tools met the environmental or human health objectives. While efforts will be made to identify links between the findings and the implementation of risk management strategies and tools, these linkages are challenging considering multiple external variables. The aim will be to identify links based on information available while acknowledging uncertainties. In addition, new information (e.g., emerging concerns or new sources of exposure) will be taken into consideration to ensure long-term effectiveness of actions in place. The evaluations will be comprehensive and evidence-based, and will determine if the environmental and/or human health and risk management objectives have been met. Performance measurement evaluation reports may include recommendations for follow-up actions.
3.1 Data Sources
Key performance indicator data will be collected and analyzed to evaluate the effectiveness of risk management tools that have been in place. Where possible, data will be sourced from programs and activities already in place. In other cases, data may need to be generated, as identified in the readiness or prioritization steps. Sources of data may include, but are not limited to:
- environmental monitoring: Chemicals Management Plan Environmental Monitoring and Surveillance Program, which leverages existing departmental programs such as the National Air Pollution Surveillance Program, Freshwater Quality Monitoring and Surveillance, Air Quality program science;
- biomonitoring: Canadian Health Measures Survey, Maternal-Infant Research on Environmental Chemicals Study;
- releases: National Pollutant Release Inventory, Air Pollutant Emission Inventory;
- food monitoring and surveillance: Canadian Total Diet Study and targeted studies;
- non-confidential information submitted by industry and other stakeholders: information provided through a voluntary or mandatory initiative, cosmetic notifications, Significant New Activity notifications;
- compliance monitoring and enforcement data: product surveillance, laboratory testing, threat risk assessments, results of departmental enforcement activities;
- information submitted in response to risk management instruments: Pollution Prevention Planning Notices, codes of practice, regulations, and environmental performance agreements; and
- other existing sources of relevant information:
3.1.1 Stakeholder engagement
Evaluations will consider all relevant information that is readily available. However, the public and stakeholders may have information that is not available to the government. To address information needs, Health Canada and Environment and Climate Change Canada may contact specific stakeholders for additional details that would inform certain evaluations, including providing baseline information or obtaining more current data. The types of information that stakeholders may possess that could inform performance measurement evaluations could include, but are not limited to:
- information outside of reporting requirements such as the National Pollutant Release Inventory (NPRI), mandatory surveys or risk management tools;
- information on the concentration of a substance in a product of interest or a release to the environment;
- research activities related to a substance, including monitoring; and
- quantity of a substance throughout the supply chain.
Stakeholders who may have relevant information could include industry, non-governmental organizations, and other levels of government, such as provinces, territories or municipalities.
Stakeholders can submit data to inform upcoming performance measurement activities through the online reporting system on Environment and Climate Change Canada's Single Window. If there are questions regarding submissions, they can be addressed by Environment and Climate Change Canada’s Substances Management Information Line. Stakeholders who provide information may submit a request that it be treated as confidential.
Progress in achieving the human health and/or environmental objectives for a substance will be determined by analyzing the best available data. Analyses of specific data sets contribute to the overall evaluation findings for a substance. As part of the overall analysis, the evaluations will also look at what influence a risk management strategy or tool has had on any observed trends or changes. For example, if a reduction in emissions of a substance is measured following the implementation of risk management strategies or tools, these reductions could be attributed to the actions taken. This analysis can help determine the ongoing relevance and effectiveness of a strategy or tool in meeting their objectives.
The analysis may determine if:
- the overall human health and/or environmental objectives have been achieved;
- Why or why not?
- If achieved, can the objectives be linked to the risk management strategies and tools?
- there are opportunities for risk management efficiencies;
- new sources of exposure are likely to be addressed by existing risk management strategies and tools;
- there are uncertainties and data gaps;
- potential follow-up actions need to be taken; and/or
- the indicators were appropriate to measure objectives and whether they are anticipated to remain so moving forward.
Guidelines such as the Federal Environmental Quality Guidelines can be used to measure performance by providing a benchmark against which current environmental or human exposure levels can be compared. In addition, specific targets (e.g., a percentage reduction) and timelines for achievement could be set to measure progress against.
While the analysis will be conducted with the best available information, it should be noted that it can take a significant amount of time to see how risk management strategies and tools impact human health and the environment. For example, it can take decades to see a decrease in the presence of certain environmental contaminants in humans, such as lead. As a result, the timeline for conducting performance measurement evaluation can vary between substances.
4. Reporting: Sharing the progress in managing risks
Performance measurement evaluation reports will summarize relevant indicator data, analysis, and other findings to demonstrate how effectively risk management measures have met objectives. Performance measurement evaluations will make one or more statements on the overall progress achieved, such as:
- the environmental and/or human health objectives have been met or there is sufficient progress toward achieving objectives;
- progress has been made to meet objectives but would benefit from additional information to determine whether additional actions may be required for ongoing effectiveness;
- the risk management strategy or tool is not achieving the intended results and follow-up actions will be recommended; or
- the evaluation is not conclusive and additional time, actions and/or data are required to assess the overall risk management of a substance.
Follow-up actions will be recommended where the performance measurement evaluation shows objectives are not adequately being met, or where linkages between actions and outcomes are not clear. Future performance measurement evaluation may be recommended for some substances, while others may only require ongoing monitoring. For those substances requiring future re-evaluation, data collection or evaluation plans will be developed to address gaps and to track key performance indicators based on the current findings.
Follow-up actions could include, but are not limited to:
- recommending changes to the risk management instrument or development of a new instrument (e.g., regulations);
- working with partners to address data gaps;
- undertaking studies or research;
- recommending substances for other activities, such as Identification of Risk Assessment Priorities monitoring (e.g., new sources of exposure, emerging science), compliance promotion or enforcement.
Where the performance measurement evaluation shows objectives are adequately being met, risk management strategies and tools will be maintained, along with the monitoring of any new information as required. Lessons learned from the outcomes of performance measurement evaluation will be applied in the development of future risk management strategies and tools.
5. Conclusion: Building confidence with systematic evaluations
Performance measurement evaluation will be considered when developing a risk management strategy or approach. Baseline data important for future performance evaluations will be gathered and ongoing data collection plans will be developed so that the best possible information is available for future analysis.
Performance measurement findings may be used in risk management and risk assessment prioritization. Information related to risk management performance measurement will be available on the Performance Measurement Evaluation for Toxic Substances webpage, including performance measurement evaluation reports, work plans and details on how to provide information.
Through performance measurement evaluation of risk management strategies and tools, Canadians can have confidence that their health and environment are being protected from harmful substances. The findings of performance measurement evaluations will be available through the Performance Measurement Evaluation for Toxic Substances webpage.
Condition that existed before a risk management tool has been implemented.
A statement that describes what should be done to address environmental concerns related to a chemical substance.
An environmental objective can be qualitative (e.g., reduce exposures to the environment) or quantitative (e.g., achieve a level of 5 micrograms/L in surface water).
Once an environmental objective is decided upon, a risk management objective(s) is (are) established.
Human health objective
A statement that describes what should be done to address human health concerns related to a chemical substance.
A human health objective can be qualitative (e.g., reduce exposures to the general population) or quantitative (e.g., a maximum long-term exposure to naphthalene below 0.010 mg/m3 in residential indoor air).
Once a human health objective is decided upon, a risk management objective(s) is (are) established.
|Key performance indicators
Qualitative or quantitative and measures an output or outcome at a given time (e.g., percentage reduction in releases of a toxic substance from a baseline year). Indicators should be:
|Performance measurement evaluation
Qualitative or quantitative means of measuring an output or outcome with the intention of gauging performance. In the management of chemical substances context, specific information collected before and after risk management tools have been implemented is evaluated to determine the effectiveness of the tools in achieving the environmental or human health objective (in the case of a strategy) or risk management objective (in the case of an individual tool or instrument).
|Risk management objective
A goal or target that describes the expected result. The result would be achieved through the implementation of a risk management instrument.
A risk management objective may be described in quantitative terms such as a performance standard, release limit, product content limit, percent reduction or more qualitative terms such as "lowest achievable levels".
Risk management objectives are developed to contribute to the achievement of, or progress towards an environmental or human health objective.
|Risk management strategy or approach
An outline of the proposed approach to manage the risks, or potential risks, to the environment and human health from a substance or group of substances.
The Canadian Environmental Protection Act, 1999 (CEPA) describes a substance as toxic "if it is entering or may enter the environment in a quantity or concentration or under conditions that:
These are the chemical substances for which risk management strategies and tools generally are implemented, to help prevent and control risks and protect human health and the environment. Before the government can take certain actions under CEPA on these substances, they have to be added to the List of Toxic Substances. In special cases, some substances, which have been assessed and concluded to meet one or more of the criteria under section 64, have not been added to the List of Toxic Substances. These substances are included on the Non-Statutory List.
- Date modified: